Oct 30 12:51:00.651897 ip-10-0-137-6 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 12:51:01.031512 ip-10-0-137-6 kubenswrapper[2569]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 12:51:01.031512 ip-10-0-137-6 kubenswrapper[2569]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 12:51:01.031512 ip-10-0-137-6 kubenswrapper[2569]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 12:51:01.031512 ip-10-0-137-6 kubenswrapper[2569]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 12:51:01.034873 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.034792    2569 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:51:01.034873 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.034872    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:51:01.034873 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.034876    2569 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:51:01.034966 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.034879    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:51:01.034966 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.034883    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:51:01.034966 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.034886    2569 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:51:01.034966 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.034888    2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:51:01.034966 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.034891    2569 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:51:01.034966 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.034894    2569 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:51:01.034966 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.034896    2569 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:51:01.034966 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.034899    2569 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:51:01.034966 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.034902    2569 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:51:01.034966 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.034904    2569 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:51:01.034966 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.034907    2569 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:51:01.034966 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.034917    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:51:01.034966 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.034920    2569 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:51:01.034966 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.034923    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:51:01.034966 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.034925    2569 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:51:01.034966 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.034928    2569 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:51:01.034966 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.034931    2569 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:51:01.034966 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.034933    2569 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:51:01.034966 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.034936    2569 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:51:01.035406 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.034938    2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:51:01.035406 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.034941    2569 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:51:01.035406 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.034943    2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:51:01.035406 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.034946    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:51:01.035406 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.034949    2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:51:01.035406 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.034951    2569 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:51:01.035406 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.034954    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:51:01.035406 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.034956    2569 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:51:01.035406 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.034959    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:51:01.035406 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.034961    2569 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:51:01.035406 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.034965    2569 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:51:01.035406 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.034967    2569 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:51:01.035406 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.034970    2569 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:51:01.035406 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.034973    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:51:01.035406 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.034976    2569 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:51:01.035406 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.034978    2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:51:01.035406 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.034981    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:51:01.035406 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.034984    2569 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:51:01.035406 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.034987    2569 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:51:01.035900 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.034989    2569 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:51:01.035900 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.034992    2569 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:51:01.035900 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.034994    2569 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:51:01.035900 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.034997    2569 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:51:01.035900 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.034999    2569 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:51:01.035900 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.035002    2569 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:51:01.035900 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.035004    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:51:01.035900 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.035006    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:51:01.035900 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.035009    2569 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:51:01.035900 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.035011    2569 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:51:01.035900 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.035014    2569 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:51:01.035900 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.035016    2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:51:01.035900 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.035019    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:51:01.035900 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.035021    2569 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:51:01.035900 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.035025    2569 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:51:01.035900 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.035030    2569 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:51:01.035900 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.035035    2569 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:51:01.035900 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.035038    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:51:01.035900 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.035041    2569 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:51:01.035900 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.035044    2569 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:51:01.036358 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.035047    2569 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:51:01.036358 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.035050    2569 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:51:01.036358 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.035052    2569 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:51:01.036358 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.035055    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:51:01.036358 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.035058    2569 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:51:01.036358 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.035060    2569 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:51:01.036358 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.035063    2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:51:01.036358 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.035065    2569 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:51:01.036358 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.035068    2569 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:51:01.036358 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.035070    2569 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:51:01.036358 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.035074    2569 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:51:01.036358 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.035077    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:51:01.036358 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.035079    2569 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:51:01.036358 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.035082    2569 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:51:01.036358 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.035085    2569 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:51:01.036358 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.035088    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:51:01.036358 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.035091    2569 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:51:01.036358 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.035093    2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:51:01.036358 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.035096    2569 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:51:01.036358 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.035098    2569 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:51:01.036879 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.035101    2569 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:51:01.036879 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.035104    2569 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:51:01.036879 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.035106    2569 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:51:01.036879 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.035109    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:51:01.036879 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.035111    2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:51:01.036879 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.035115    2569 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:51:01.036879 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.035118    2569 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:51:01.036879 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.035121    2569 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:51:01.036879 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.035124    2569 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:51:01.036879 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.035126    2569 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:51:01.036879 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.035129    2569 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:51:01.036879 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.035132    2569 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:51:01.036879 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.035134    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:51:01.036879 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.035137    2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:51:01.036879 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.035140    2569 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:51:01.036879 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.035142    2569 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:51:01.036879 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.035144    2569 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:51:01.036879 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.035147    2569 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:51:01.036879 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.035150    2569 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:51:01.037323 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.035154    2569 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:51:01.037323 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.035158    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:51:01.037323 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.035161    2569 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:51:01.037323 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.035165    2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:51:01.037323 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.035168    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:51:01.037323 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.035172    2569 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:51:01.037323 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.035964    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:51:01.037323 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.035972    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:51:01.037323 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.035976    2569 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:51:01.037323 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.035978    2569 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:51:01.037323 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.035981    2569 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:51:01.037323 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.035984    2569 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:51:01.037323 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.035987    2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:51:01.037323 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.035989    2569 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:51:01.037323 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.035992    2569 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:51:01.037323 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.035995    2569 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:51:01.037323 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.035997    2569 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:51:01.037323 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036000    2569 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:51:01.037323 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036002    2569 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:51:01.037323 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036005    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:51:01.037809 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036008    2569 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:51:01.037809 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036011    2569 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:51:01.037809 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036013    2569 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:51:01.037809 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036016    2569 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:51:01.037809 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036019    2569 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:51:01.037809 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036022    2569 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:51:01.037809 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036024    2569 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:51:01.037809 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036027    2569 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:51:01.037809 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036029    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:51:01.037809 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036032    2569 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:51:01.037809 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036035    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:51:01.037809 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036037    2569 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:51:01.037809 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036039    2569 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:51:01.037809 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036042    2569 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:51:01.037809 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036044    2569 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:51:01.037809 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036047    2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:51:01.037809 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036050    2569 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:51:01.037809 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036052    2569 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:51:01.037809 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036055    2569 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:51:01.037809 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036059    2569 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:51:01.038275 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036061    2569 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:51:01.038275 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036064    2569 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:51:01.038275 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036066    2569 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:51:01.038275 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036069    2569 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:51:01.038275 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036072    2569 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:51:01.038275 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036075    2569 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:51:01.038275 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036077    2569 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:51:01.038275 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036080    2569 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:51:01.038275 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036082    2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:51:01.038275 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036085    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:51:01.038275 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036087    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:51:01.038275 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036091    2569 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:51:01.038275 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036094    2569 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:51:01.038275 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036097    2569 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:51:01.038275 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036100    2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:51:01.038275 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036103    2569 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:51:01.038275 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036105    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:51:01.038275 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036108    2569 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:51:01.038275 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036110    2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:51:01.038732 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036113    2569 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:51:01.038732 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036116    2569 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:51:01.038732 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036118    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:51:01.038732 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036121    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:51:01.038732 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036124    2569 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:51:01.038732 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036126    2569 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:51:01.038732 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036129    2569 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:51:01.038732 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036131    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:51:01.038732 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036134    2569 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:51:01.038732 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036138    2569 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:51:01.038732 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036142    2569 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:51:01.038732 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036145    2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:51:01.038732 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036148    2569 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:51:01.038732 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036151    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:51:01.038732 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036154    2569 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:51:01.038732 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036156    2569 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:51:01.038732 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036159    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:51:01.038732 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036162    2569 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:51:01.038732 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036165    2569 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:51:01.039178 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036168    2569 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:51:01.039178 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036170    2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:51:01.039178 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036174    2569 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:51:01.039178 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036178    2569 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:51:01.039178 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036181    2569 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:51:01.039178 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036184    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:51:01.039178 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036187    2569 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:51:01.039178 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036189    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:51:01.039178 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036192    2569 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:51:01.039178 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036195    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:51:01.039178 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036197    2569 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:51:01.039178 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036200    2569 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:51:01.039178 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036203    2569 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:51:01.039178 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036205    2569 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:51:01.039178 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036208    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:51:01.039178 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036210    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:51:01.039178 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036213    2569 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:51:01.039178 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036216    2569 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:51:01.039178 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036219    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:51:01.039668 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036222    2569 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:51:01.039668 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036224    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:51:01.039668 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036227    2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:51:01.039668 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036229    2569 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:51:01.039668 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036231    2569 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:51:01.039668 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036234    2569 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:51:01.039668 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036236    2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:51:01.039668 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036240    2569 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:51:01.039668 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036242    2569 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:51:01.039668 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036264    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:51:01.039668 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036269    2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:51:01.039668 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036272    2569 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:51:01.039668 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036274    2569 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:51:01.039668 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036278    2569 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:51:01.039668 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036281    2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:51:01.039668 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036782    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:51:01.039668 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036788    2569 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:51:01.039668 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036791    2569 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:51:01.039668 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036795    2569 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:51:01.040107 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036798    2569 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:51:01.040107 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036800    2569 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:51:01.040107 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036803    2569 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:51:01.040107 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036806    2569 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:51:01.040107 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036808    2569 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:51:01.040107 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036811    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:51:01.040107 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036813    2569 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:51:01.040107 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036816    2569 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:51:01.040107 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036818    2569 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:51:01.040107 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036821    2569 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:51:01.040107 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036824    2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:51:01.040107 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036827    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:51:01.040107 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036830    2569 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:51:01.040107 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036832    2569 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:51:01.040107 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036835    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:51:01.040107 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036838    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:51:01.040107 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036840    2569 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:51:01.040107 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036843    2569 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:51:01.040107 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036845    2569 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:51:01.040107 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036848    2569 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:51:01.040107 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036851    2569 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:51:01.040619 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036855    2569 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:51:01.040619 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036859    2569 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:51:01.040619 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036862    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:51:01.040619 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036864    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:51:01.040619 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036867    2569 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:51:01.040619 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036869    2569 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:51:01.040619 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036872    2569 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:51:01.040619 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036874    2569 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:51:01.040619 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036877    2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:51:01.040619 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036879    2569 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:51:01.040619 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036882    2569 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:51:01.040619 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036884    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:51:01.040619 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036887    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:51:01.040619 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036890    2569 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:51:01.040619 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036893    2569 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:51:01.040619 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036895    2569 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:51:01.040619 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036898    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:51:01.040619 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036901    2569 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:51:01.040619 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036903    2569 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:51:01.040619 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036906    2569 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:51:01.041084 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036908    2569 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:51:01.041084 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036911    2569 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:51:01.041084 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036913    2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:51:01.041084 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036916    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:51:01.041084 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036919    2569 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:51:01.041084 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036921    2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:51:01.041084 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036924    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:51:01.041084 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036926    2569 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:51:01.041084 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036928    2569 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:51:01.041084 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036931    2569 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:51:01.041084 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036934    2569 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:51:01.041084 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036937    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:51:01.041084 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036940    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:51:01.041084 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036943    2569 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:51:01.041084 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036945    2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:51:01.041084 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036948    2569 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:51:01.041084 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036950    2569 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:51:01.041084 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036953    2569 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:51:01.041084 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036957    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:51:01.041531 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036959    2569 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:51:01.041531 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036962    2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:51:01.041531 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036964    2569 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:51:01.041531 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036967    2569 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:51:01.041531 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036970    2569 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:51:01.041531 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036972    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:51:01.041531 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036974    2569 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:51:01.041531 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036977    2569 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:51:01.041531 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036979    2569 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:51:01.041531 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036983    2569 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:51:01.041531 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036988    2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:51:01.041531 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036992    2569 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:51:01.041531 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036994    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:51:01.041531 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.036997    2569 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:51:01.041531 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.037000    2569 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:51:01.041531 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.037003    2569 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:51:01.041531 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.037005    2569 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:51:01.041531 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.037007    2569 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:51:01.041531 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.037010    2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:51:01.041983 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.037012    2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:51:01.041983 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.037015    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:51:01.041983 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.037017    2569 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:51:01.041983 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.037019    2569 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:51:01.041983 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.037022    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:51:01.041983 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.037025    2569 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:51:01.041983 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.037027    2569 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:51:01.041983 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.037030    2569 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:51:01.041983 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.037033    2569 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:51:01.041983 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.037035    2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:51:01.041983 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.037038    2569 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:51:01.041983 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.037040    2569 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:51:01.041983 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.037043    2569 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:51:01.041983 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.037045    2569 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:51:01.041983 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.037048    2569 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:51:01.041983 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.037050    2569 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:51:01.041983 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.037052    2569 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:51:01.041983 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.037055    2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:51:01.041983 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.037057    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:51:01.042415 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.037060    2569 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:51:01.042415 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.037063    2569 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:51:01.042415 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.037066    2569 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:51:01.042415 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.037068    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:51:01.042415 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.037984    2569 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 12:51:01.042415 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.037997    2569 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 12:51:01.042415 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038003    2569 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 12:51:01.042415 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038007    2569 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 12:51:01.042415 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038012    2569 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 12:51:01.042415 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038016    2569 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 12:51:01.042415 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038020    2569 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 12:51:01.042415 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038024    2569 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 12:51:01.042415 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038028    2569 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 12:51:01.042415 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038031    2569 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 12:51:01.042415 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038034    2569 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 12:51:01.042415 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038040    2569 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 12:51:01.042415 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038043    2569 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 12:51:01.042415 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038046    2569 flags.go:64] FLAG: --cgroup-root=""
Oct 30 12:51:01.042415 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038049    2569 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 12:51:01.042415 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038052    2569 flags.go:64] FLAG: --client-ca-file=""
Oct 30 12:51:01.042415 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038055    2569 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 12:51:01.042415 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038059    2569 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 12:51:01.042415 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038064    2569 flags.go:64] FLAG: --cluster-domain=""
Oct 30 12:51:01.042946 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038067    2569 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 12:51:01.042946 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038070    2569 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 12:51:01.042946 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038074    2569 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 12:51:01.042946 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038077    2569 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 12:51:01.042946 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038082    2569 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 12:51:01.042946 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038085    2569 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 12:51:01.042946 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038088    2569 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 12:51:01.042946 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038092    2569 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 12:51:01.042946 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038095    2569 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 12:51:01.042946 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038098    2569 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 12:51:01.042946 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038101    2569 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 12:51:01.042946 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038105    2569 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 12:51:01.042946 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038108    2569 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 12:51:01.042946 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038112    2569 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 12:51:01.042946 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038115    2569 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 12:51:01.042946 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038118    2569 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 12:51:01.042946 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038121    2569 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 12:51:01.042946 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038126    2569 flags.go:64] FLAG: --enable-server="true"
Oct 30 12:51:01.042946 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038129    2569 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 12:51:01.042946 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038133    2569 flags.go:64] FLAG: --event-burst="100"
Oct 30 12:51:01.042946 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038136    2569 flags.go:64] FLAG: --event-qps="50"
Oct 30 12:51:01.042946 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038139    2569 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 12:51:01.042946 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038143    2569 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 12:51:01.042946 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038146    2569 flags.go:64] FLAG: --eviction-hard=""
Oct 30 12:51:01.043553 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038151    2569 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 12:51:01.043553 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038154    2569 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 12:51:01.043553 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038157    2569 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 12:51:01.043553 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038160    2569 flags.go:64] FLAG: --eviction-soft=""
Oct 30 12:51:01.043553 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038163    2569 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 12:51:01.043553 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038166    2569 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 12:51:01.043553 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038169    2569 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 12:51:01.043553 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038175    2569 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 12:51:01.043553 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038179    2569 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 12:51:01.043553 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038182    2569 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 12:51:01.043553 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038185    2569 flags.go:64] FLAG: --feature-gates=""
Oct 30 12:51:01.043553 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038189    2569 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 12:51:01.043553 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038192    2569 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 12:51:01.043553 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038195    2569 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 12:51:01.043553 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038199    2569 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 12:51:01.043553 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038202    2569 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 12:51:01.043553 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038205    2569 flags.go:64] FLAG: --help="false"
Oct 30 12:51:01.043553 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038208    2569 flags.go:64] FLAG: --hostname-override="ip-10-0-137-6.ec2.internal"
Oct 30 12:51:01.043553 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038211    2569 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 12:51:01.043553 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038214    2569 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 12:51:01.043553 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038217    2569 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 12:51:01.043553 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038221    2569 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 12:51:01.043553 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038224    2569 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 12:51:01.044120 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038227    2569 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 12:51:01.044120 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038230    2569 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 12:51:01.044120 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038234    2569 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 12:51:01.044120 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038237    2569 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 12:51:01.044120 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038240    2569 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 12:51:01.044120 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038243    2569 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 12:51:01.044120 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038246    2569 flags.go:64] FLAG: --kube-reserved=""
Oct 30 12:51:01.044120 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038249    2569 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 12:51:01.044120 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038252    2569 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 12:51:01.044120 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038257    2569 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 12:51:01.044120 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038260    2569 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 12:51:01.044120 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038263    2569 flags.go:64] FLAG: --lock-file=""
Oct 30 12:51:01.044120 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038266    2569 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 12:51:01.044120 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038269    2569 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 12:51:01.044120 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038272    2569 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 12:51:01.044120 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038278    2569 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 12:51:01.044120 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038281    2569 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 12:51:01.044120 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038284    2569 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 12:51:01.044120 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038288    2569 flags.go:64] FLAG: --logging-format="text"
Oct 30 12:51:01.044120 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038291    2569 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 12:51:01.044120 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038294    2569 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 12:51:01.044120 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038297    2569 flags.go:64] FLAG: --manifest-url=""
Oct 30 12:51:01.044120 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038300    2569 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 12:51:01.044120 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038305    2569 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 12:51:01.044120 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038308    2569 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 12:51:01.044838 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038311    2569 flags.go:64] FLAG: --max-pods="110"
Oct 30 12:51:01.044838 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038314    2569 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 12:51:01.044838 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038318    2569 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 12:51:01.044838 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038321    2569 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 12:51:01.044838 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038324    2569 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 12:51:01.044838 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038327    2569 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 12:51:01.044838 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038330    2569 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 12:51:01.044838 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038333    2569 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 12:51:01.044838 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038340    2569 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 12:51:01.044838 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038344    2569 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 12:51:01.044838 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038347    2569 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 12:51:01.044838 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038350    2569 flags.go:64] FLAG: --pod-cidr=""
Oct 30 12:51:01.044838 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038353    2569 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 12:51:01.044838 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038356    2569 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 12:51:01.044838 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038359    2569 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 12:51:01.044838 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038362    2569 flags.go:64] FLAG: --port="10250"
Oct 30 12:51:01.044838 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038367    2569 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 12:51:01.044838 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038370    2569 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0265679eade489590"
Oct 30 12:51:01.044838 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038373    2569 flags.go:64] FLAG: --qos-reserved=""
Oct 30 12:51:01.044838 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038376    2569 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 12:51:01.044838 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038379    2569 flags.go:64] FLAG: --register-node="true"
Oct 30 12:51:01.044838 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038382    2569 flags.go:64] FLAG: --register-with-taints=""
Oct 30 12:51:01.044838 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038386    2569 flags.go:64] FLAG: --registry-burst="10"
Oct 30 12:51:01.044838 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038389    2569 flags.go:64] FLAG: --registry-qps="5"
Oct 30 12:51:01.044838 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038392    2569 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 12:51:01.044838 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038395    2569 flags.go:64] FLAG: --reserved-memory=""
Oct 30 12:51:01.045451 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038400    2569 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 12:51:01.045451 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038403    2569 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 12:51:01.045451 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038406    2569 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 12:51:01.045451 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038409    2569 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 12:51:01.045451 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038412    2569 flags.go:64] FLAG: --runonce="false"
Oct 30 12:51:01.045451 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038415    2569 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 12:51:01.045451 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038418    2569 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 12:51:01.045451 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038421    2569 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 12:51:01.045451 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038424    2569 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 12:51:01.045451 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038427    2569 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 12:51:01.045451 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038430    2569 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 12:51:01.045451 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038433    2569 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 12:51:01.045451 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038436    2569 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 12:51:01.045451 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038439    2569 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 12:51:01.045451 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038442    2569 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 12:51:01.045451 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038446    2569 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 12:51:01.045451 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038449    2569 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 12:51:01.045451 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038452    2569 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 12:51:01.045451 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038456    2569 flags.go:64] FLAG: --system-cgroups=""
Oct 30 12:51:01.045451 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038459    2569 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 12:51:01.045451 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038464    2569 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 12:51:01.045451 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038467    2569 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 12:51:01.045451 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038471    2569 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 12:51:01.045451 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038476    2569 flags.go:64] FLAG: --tls-min-version=""
Oct 30 12:51:01.045451 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038479    2569 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 12:51:01.046055 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038481    2569 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 12:51:01.046055 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038484    2569 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 12:51:01.046055 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038487    2569 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 12:51:01.046055 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038490    2569 flags.go:64] FLAG: --v="2"
Oct 30 12:51:01.046055 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038495    2569 flags.go:64] FLAG: --version="false"
Oct 30 12:51:01.046055 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038499    2569 flags.go:64] FLAG: --vmodule=""
Oct 30 12:51:01.046055 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038503    2569 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 12:51:01.046055 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038507    2569 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 12:51:01.046055 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038655    2569 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:51:01.046055 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038659    2569 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:51:01.046055 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038663    2569 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:51:01.046055 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038666    2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:51:01.046055 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038670    2569 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:51:01.046055 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038673    2569 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:51:01.046055 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038676    2569 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:51:01.046055 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038679    2569 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:51:01.046055 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038682    2569 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:51:01.046055 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038684    2569 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:51:01.046055 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038687    2569 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:51:01.046055 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038690    2569 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:51:01.046055 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038693    2569 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:51:01.046055 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038695    2569 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:51:01.046571 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038700    2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:51:01.046571 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038702    2569 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:51:01.046571 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038705    2569 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:51:01.046571 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038708    2569 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:51:01.046571 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038710    2569 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:51:01.046571 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038713    2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:51:01.046571 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038715    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:51:01.046571 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038719    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:51:01.046571 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038722    2569 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:51:01.046571 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038725    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:51:01.046571 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038727    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:51:01.046571 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038730    2569 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:51:01.046571 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038733    2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:51:01.046571 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038736    2569 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:51:01.046571 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038738    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:51:01.046571 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038741    2569 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:51:01.046571 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038744    2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:51:01.046571 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038746    2569 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:51:01.046571 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038750    2569 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:51:01.046571 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038752    2569 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:51:01.047068 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038755    2569 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:51:01.047068 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038757    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:51:01.047068 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038759    2569 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:51:01.047068 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038762    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:51:01.047068 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038764    2569 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:51:01.047068 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038767    2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:51:01.047068 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038770    2569 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:51:01.047068 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038772    2569 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:51:01.047068 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038775    2569 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:51:01.047068 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038777    2569 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:51:01.047068 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038780    2569 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:51:01.047068 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038783    2569 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:51:01.047068 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038786    2569 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:51:01.047068 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038788    2569 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:51:01.047068 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038791    2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:51:01.047068 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038794    2569 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:51:01.047068 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038796    2569 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:51:01.047068 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038799    2569 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:51:01.047068 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038801    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:51:01.047068 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038805    2569 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:51:01.047539 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038808    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:51:01.047539 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038811    2569 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:51:01.047539 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038813    2569 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:51:01.047539 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038816    2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:51:01.047539 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038819    2569 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:51:01.047539 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038821    2569 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:51:01.047539 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038824    2569 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:51:01.047539 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038827    2569 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:51:01.047539 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038829    2569 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:51:01.047539 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038832    2569 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:51:01.047539 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038835    2569 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:51:01.047539 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038838    2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:51:01.047539 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038840    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:51:01.047539 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038843    2569 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:51:01.047539 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038845    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:51:01.047539 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038849    2569 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:51:01.047539 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038851    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:51:01.047539 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038854    2569 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:51:01.047539 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038856    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:51:01.047539 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038859    2569 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:51:01.048062 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038861    2569 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:51:01.048062 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038864    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:51:01.048062 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038866    2569 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:51:01.048062 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038869    2569 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:51:01.048062 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038871    2569 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:51:01.048062 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038874    2569 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:51:01.048062 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038876    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:51:01.048062 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038879    2569 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:51:01.048062 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038881    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:51:01.048062 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038884    2569 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:51:01.048062 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038886    2569 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:51:01.048062 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038891    2569 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:51:01.048062 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038893    2569 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:51:01.048062 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038896    2569 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:51:01.048062 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038898    2569 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:51:01.048062 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038901    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:51:01.048062 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038903    2569 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:51:01.048062 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038906    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:51:01.048062 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038908    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:51:01.048512 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038911    2569 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:51:01.048512 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038915    2569 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:51:01.048512 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038918    2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:51:01.048512 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038921    2569 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:51:01.048512 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038924    2569 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:51:01.048512 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038927    2569 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:51:01.048512 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038930    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:51:01.048512 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038933    2569 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:51:01.048512 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038935    2569 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:51:01.048512 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038938    2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:51:01.048512 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038940    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:51:01.048512 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038943    2569 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:51:01.048512 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.038945    2569 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:51:01.048512 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.038971    2569 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 12:51:01.048512 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039006    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:51:01.048872 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039010    2569 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:51:01.048872 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039012    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:51:01.048872 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039016    2569 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:51:01.048872 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039019    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:51:01.048872 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039021    2569 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:51:01.048872 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039024    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:51:01.048872 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039026    2569 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:51:01.048872 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039029    2569 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:51:01.048872 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039032    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:51:01.048872 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039035    2569 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:51:01.048872 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039039    2569 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:51:01.048872 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039042    2569 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:51:01.048872 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039045    2569 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:51:01.048872 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039048    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:51:01.048872 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039050    2569 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:51:01.048872 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039053    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:51:01.048872 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039055    2569 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:51:01.048872 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039058    2569 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:51:01.048872 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039061    2569 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:51:01.049322 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039063    2569 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:51:01.049322 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039066    2569 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:51:01.049322 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039068    2569 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:51:01.049322 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039071    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:51:01.049322 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039074    2569 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:51:01.049322 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039076    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:51:01.049322 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039079    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:51:01.049322 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039081    2569 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:51:01.049322 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039085    2569 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:51:01.049322 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039088    2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:51:01.049322 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039092    2569 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:51:01.049322 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039095    2569 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:51:01.049322 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039097    2569 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:51:01.049322 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039100    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:51:01.049322 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039103    2569 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:51:01.049322 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039106    2569 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:51:01.049322 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039108    2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:51:01.049322 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039111    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:51:01.049322 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039113    2569 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:51:01.049796 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039116    2569 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:51:01.049796 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039118    2569 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:51:01.049796 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039121    2569 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:51:01.049796 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039123    2569 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:51:01.049796 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039127    2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:51:01.049796 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039131    2569 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:51:01.049796 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039135    2569 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:51:01.049796 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039141    2569 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:51:01.049796 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039144    2569 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:51:01.049796 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039148    2569 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:51:01.049796 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039150    2569 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:51:01.049796 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039153    2569 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:51:01.049796 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039156    2569 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:51:01.049796 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039158    2569 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:51:01.049796 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039161    2569 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:51:01.049796 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039163    2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:51:01.049796 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039166    2569 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:51:01.049796 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039168    2569 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:51:01.049796 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039171    2569 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:51:01.049796 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039174    2569 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:51:01.050260 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039177    2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:51:01.050260 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039180    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:51:01.050260 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039182    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:51:01.050260 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039185    2569 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:51:01.050260 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039187    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:51:01.050260 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039190    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:51:01.050260 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039193    2569 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:51:01.050260 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039195    2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:51:01.050260 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039198    2569 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:51:01.050260 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039200    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:51:01.050260 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039203    2569 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:51:01.050260 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039205    2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:51:01.050260 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039208    2569 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:51:01.050260 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039210    2569 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:51:01.050260 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039213    2569 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:51:01.050260 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039215    2569 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:51:01.050260 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039218    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:51:01.050260 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039221    2569 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:51:01.050260 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039223    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:51:01.050260 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039227    2569 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:51:01.050759 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039230    2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:51:01.050759 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039233    2569 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:51:01.050759 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039235    2569 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:51:01.050759 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039238    2569 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:51:01.050759 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039240    2569 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:51:01.050759 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039243    2569 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:51:01.050759 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039246    2569 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:51:01.050759 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039248    2569 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:51:01.050759 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039251    2569 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:51:01.050759 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039253    2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:51:01.050759 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039256    2569 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:51:01.050759 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039259    2569 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:51:01.050759 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039261    2569 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:51:01.050759 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039264    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:51:01.050759 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039267    2569 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:51:01.050759 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039269    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:51:01.050759 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039271    2569 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:51:01.050759 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039274    2569 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:51:01.050759 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039276    2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:51:01.050759 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039279    2569 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:51:01.051213 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039281    2569 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:51:01.051213 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039284    2569 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:51:01.051213 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039286    2569 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:51:01.051213 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039289    2569 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:51:01.051213 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039291    2569 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:51:01.051213 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039294    2569 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:51:01.051213 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.039296    2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:51:01.051213 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.046038    2569 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 12:51:01.051213 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.046053    2569 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 12:51:01.051213 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.046070    2569 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 12:51:01.051213 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.046075    2569 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 12:51:01.051213 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046132    2569 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:51:01.051213 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046138    2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:51:01.051213 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046141    2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:51:01.051213 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046144    2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:51:01.051213 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046147    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:51:01.051213 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046149    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:51:01.051213 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046152    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:51:01.051213 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046155    2569 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:51:01.051663 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046158    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:51:01.051663 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046160    2569 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:51:01.051663 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046163    2569 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:51:01.051663 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046166    2569 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:51:01.051663 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046173    2569 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:51:01.051663 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046176    2569 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:51:01.051663 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046179    2569 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:51:01.051663 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046182    2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:51:01.051663 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046185    2569 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:51:01.051663 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046187    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:51:01.051663 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046190    2569 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:51:01.051663 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046193    2569 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:51:01.051663 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046196    2569 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:51:01.051663 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046198    2569 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:51:01.051663 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046201    2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:51:01.051663 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046204    2569 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:51:01.051663 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046206    2569 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:51:01.051663 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046209    2569 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:51:01.051663 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046212    2569 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:51:01.051663 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046215    2569 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:51:01.052132 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046218    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:51:01.052132 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046221    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:51:01.052132 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046224    2569 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:51:01.052132 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046227    2569 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:51:01.052132 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046230    2569 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:51:01.052132 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046234    2569 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:51:01.052132 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046238    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:51:01.052132 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046241    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:51:01.052132 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046244    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:51:01.052132 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046247    2569 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:51:01.052132 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046250    2569 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:51:01.052132 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046252    2569 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:51:01.052132 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046255    2569 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:51:01.052132 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046257    2569 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:51:01.052132 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046260    2569 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:51:01.052132 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046263    2569 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:51:01.052132 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046265    2569 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:51:01.052132 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046268    2569 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:51:01.052132 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046271    2569 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:51:01.052613 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046274    2569 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:51:01.052613 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046277    2569 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:51:01.052613 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046279    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:51:01.052613 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046282    2569 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:51:01.052613 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046284    2569 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:51:01.052613 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046287    2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:51:01.052613 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046289    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:51:01.052613 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046292    2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:51:01.052613 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046294    2569 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:51:01.052613 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046297    2569 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:51:01.052613 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046299    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:51:01.052613 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046302    2569 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:51:01.052613 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046305    2569 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:51:01.052613 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046308    2569 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:51:01.052613 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046311    2569 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:51:01.052613 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046313    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:51:01.052613 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046316    2569 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:51:01.052613 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046319    2569 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:51:01.052613 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046321    2569 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:51:01.053898 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046324    2569 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:51:01.053898 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046326    2569 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:51:01.053898 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046329    2569 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:51:01.053898 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046332    2569 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:51:01.053898 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046334    2569 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:51:01.053898 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046337    2569 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:51:01.053898 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046339    2569 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:51:01.053898 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046342    2569 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:51:01.053898 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046345    2569 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:51:01.053898 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046348    2569 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:51:01.053898 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046350    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:51:01.053898 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046353    2569 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:51:01.053898 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046356    2569 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:51:01.053898 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046360    2569 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:51:01.053898 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046362    2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:51:01.053898 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046365    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:51:01.053898 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046367    2569 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:51:01.053898 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046370    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:51:01.053898 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046373    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:51:01.053898 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046375    2569 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:51:01.054415 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046377    2569 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:51:01.054415 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046380    2569 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:51:01.054415 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046382    2569 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:51:01.054415 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046385    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:51:01.054415 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046387    2569 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:51:01.054415 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046390    2569 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:51:01.054415 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046393    2569 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:51:01.054415 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046396    2569 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:51:01.054415 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046400    2569 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:51:01.054415 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046402    2569 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:51:01.054415 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046405    2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:51:01.054415 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046408    2569 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:51:01.054415 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046410    2569 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:51:01.054415 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046413    2569 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:51:01.054415 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046415    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:51:01.054415 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046419    2569 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:51:01.054415 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046423    2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:51:01.054415 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046426    2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:51:01.054415 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046428    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:51:01.054888 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046431    2569 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:51:01.054888 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.046458    2569 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 12:51:01.054888 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046608    2569 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:51:01.054888 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046616    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:51:01.054888 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046620    2569 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:51:01.054888 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046623    2569 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:51:01.054888 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046626    2569 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:51:01.054888 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046629    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:51:01.054888 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046632    2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:51:01.054888 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046634    2569 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:51:01.054888 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046637    2569 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:51:01.054888 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046640    2569 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:51:01.054888 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046642    2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:51:01.054888 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046644    2569 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:51:01.054888 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046647    2569 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:51:01.055236 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046650    2569 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:51:01.055236 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046652    2569 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:51:01.055236 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046655    2569 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:51:01.055236 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046657    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:51:01.055236 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046660    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:51:01.055236 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046663    2569 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:51:01.055236 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046666    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:51:01.055236 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046668    2569 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:51:01.055236 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046671    2569 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:51:01.055236 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046673    2569 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:51:01.055236 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046676    2569 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:51:01.055236 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046678    2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:51:01.055236 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046681    2569 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:51:01.055236 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046683    2569 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:51:01.055236 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046686    2569 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:51:01.055236 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046688    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:51:01.055236 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046691    2569 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:51:01.055236 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046693    2569 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:51:01.055236 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046696    2569 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:51:01.055236 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046698    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:51:01.055708 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046701    2569 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:51:01.055708 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046703    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:51:01.055708 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046706    2569 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:51:01.055708 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046709    2569 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:51:01.055708 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046712    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:51:01.055708 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046714    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:51:01.055708 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046717    2569 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:51:01.055708 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046719    2569 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:51:01.055708 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046723    2569 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:51:01.055708 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046728    2569 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:51:01.055708 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046731    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:51:01.055708 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046733    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:51:01.055708 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046736    2569 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:51:01.055708 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046739    2569 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:51:01.055708 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046741    2569 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:51:01.055708 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046744    2569 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:51:01.055708 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046746    2569 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:51:01.055708 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046749    2569 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:51:01.055708 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046751    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:51:01.056146 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046754    2569 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:51:01.056146 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046757    2569 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:51:01.056146 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046759    2569 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:51:01.056146 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046762    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:51:01.056146 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046764    2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:51:01.056146 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046766    2569 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:51:01.056146 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046769    2569 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:51:01.056146 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046771    2569 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:51:01.056146 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046774    2569 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:51:01.056146 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046776    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:51:01.056146 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046779    2569 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:51:01.056146 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046781    2569 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:51:01.056146 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046784    2569 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:51:01.056146 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046786    2569 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:51:01.056146 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046789    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:51:01.056146 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046791    2569 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:51:01.056146 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046794    2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:51:01.056146 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046796    2569 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:51:01.056146 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046799    2569 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:51:01.056146 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046802    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:51:01.056636 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046804    2569 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:51:01.056636 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046807    2569 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:51:01.056636 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046809    2569 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:51:01.056636 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046812    2569 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:51:01.056636 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046814    2569 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:51:01.056636 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046817    2569 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:51:01.056636 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046819    2569 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:51:01.056636 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046822    2569 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:51:01.056636 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046824    2569 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:51:01.056636 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046827    2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:51:01.056636 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046830    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:51:01.056636 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046832    2569 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:51:01.056636 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046835    2569 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:51:01.056636 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046837    2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:51:01.056636 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046840    2569 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:51:01.056636 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046842    2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:51:01.056636 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046845    2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:51:01.056636 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046847    2569 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:51:01.056636 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046850    2569 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:51:01.056636 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046852    2569 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:51:01.057142 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046855    2569 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:51:01.057142 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046857    2569 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:51:01.057142 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046860    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:51:01.057142 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046862    2569 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:51:01.057142 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046865    2569 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:51:01.057142 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046868    2569 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:51:01.057142 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046870    2569 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:51:01.057142 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046873    2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:51:01.057142 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046875    2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:51:01.057142 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046878    2569 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:51:01.057142 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046881    2569 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:51:01.057142 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046884    2569 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:51:01.057142 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046887    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:51:01.057142 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046889    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:51:01.057142 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.046912    2569 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 12:51:01.057142 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046945    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:51:01.057518 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046949    2569 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:51:01.057518 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046952    2569 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:51:01.057518 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046955    2569 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:51:01.057518 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046957    2569 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:51:01.057518 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046960    2569 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:51:01.057518 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046963    2569 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:51:01.057518 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046966    2569 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:51:01.057518 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046969    2569 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:51:01.057518 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046971    2569 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:51:01.057518 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046974    2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:51:01.057518 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046977    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:51:01.057518 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046979    2569 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:51:01.057518 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046982    2569 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:51:01.057518 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046984    2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:51:01.057518 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046987    2569 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:51:01.057518 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046989    2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:51:01.057518 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046992    2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:51:01.057518 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046994    2569 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:51:01.057518 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046997    2569 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:51:01.057518 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.046999    2569 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:51:01.058125 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047002    2569 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:51:01.058125 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047004    2569 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:51:01.058125 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047007    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:51:01.058125 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047009    2569 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:51:01.058125 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047012    2569 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:51:01.058125 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047018    2569 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:51:01.058125 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047021    2569 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:51:01.058125 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047024    2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:51:01.058125 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047027    2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:51:01.058125 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047030    2569 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:51:01.058125 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047033    2569 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:51:01.058125 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047035    2569 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:51:01.058125 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047038    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:51:01.058125 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047040    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:51:01.058125 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047045    2569 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:51:01.058125 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047048    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:51:01.058125 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047051    2569 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:51:01.058125 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047054    2569 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:51:01.058125 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047057    2569 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:51:01.058569 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047060    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:51:01.058569 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047062    2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:51:01.058569 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047065    2569 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:51:01.058569 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047068    2569 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:51:01.058569 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047070    2569 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:51:01.058569 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047073    2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:51:01.058569 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047076    2569 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:51:01.058569 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047079    2569 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:51:01.058569 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047082    2569 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:51:01.058569 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047084    2569 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:51:01.058569 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047087    2569 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:51:01.058569 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047089    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:51:01.058569 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047092    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:51:01.058569 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047094    2569 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:51:01.058569 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047097    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:51:01.058569 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047100    2569 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:51:01.058569 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047102    2569 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:51:01.058569 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047105    2569 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:51:01.058569 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047109    2569 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:51:01.059063 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047111    2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:51:01.059063 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047114    2569 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:51:01.059063 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047117    2569 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:51:01.059063 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047120    2569 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:51:01.059063 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047122    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:51:01.059063 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047125    2569 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:51:01.059063 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047127    2569 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:51:01.059063 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047130    2569 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:51:01.059063 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047133    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:51:01.059063 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047135    2569 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:51:01.059063 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047139    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:51:01.059063 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047141    2569 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:51:01.059063 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047145    2569 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:51:01.059063 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047148    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:51:01.059063 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047151    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:51:01.059063 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047154    2569 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:51:01.059063 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047156    2569 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:51:01.059063 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047160    2569 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:51:01.059063 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047163    2569 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:51:01.059502 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047166    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:51:01.059502 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047169    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:51:01.059502 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047171    2569 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:51:01.059502 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047174    2569 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:51:01.059502 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047176    2569 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:51:01.059502 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047198    2569 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:51:01.059502 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047202    2569 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:51:01.059502 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047205    2569 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:51:01.059502 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047209    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:51:01.059502 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047212    2569 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:51:01.059502 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047215    2569 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:51:01.059502 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047217    2569 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:51:01.059502 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047221    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:51:01.059502 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047224    2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:51:01.059502 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047226    2569 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:51:01.059502 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047229    2569 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:51:01.059502 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047231    2569 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:51:01.059502 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047234    2569 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:51:01.059502 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047237    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:51:01.059502 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047240    2569 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:51:01.059979 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047242    2569 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:51:01.059979 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047245    2569 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:51:01.059979 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047247    2569 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:51:01.059979 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047250    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:51:01.059979 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047252    2569 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:51:01.059979 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047258    2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:51:01.059979 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047260    2569 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:51:01.059979 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:01.047263    2569 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:51:01.059979 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.047312    2569 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 12:51:01.059979 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.049513    2569 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 12:51:01.059979 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.050557    2569 server.go:1014] "Starting client certificate rotation"
Oct 30 12:51:01.059979 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.050653    2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 12:51:01.059979 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.051503    2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 12:51:01.059979 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.054173    2569 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 12:51:01.059979 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.058849    2569 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 12:51:01.072611 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.072585    2569 log.go:25] "Validated CRI v1 runtime API"
Oct 30 12:51:01.077619 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.077589    2569 log.go:25] "Validated CRI v1 image API"
Oct 30 12:51:01.078696 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.078672    2569 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 12:51:01.081795 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.081775    2569 fs.go:135] Filesystem UUIDs: map[79892a95-1ec4-412c-98aa-4788f9cef552:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 cc1aad2c-dca2-41db-bf03-471d34c82d3d:/dev/nvme0n1p3]
Oct 30 12:51:01.081871 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.081794    2569 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 12:51:01.086038 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.085936    2569 manager.go:217] Machine: {Timestamp:2026-10-30 12:51:01.085466314 +0000 UTC m=+0.324844519 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099399 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2025572fc46201b88d8a24bd390f6e SystemUUID:ec202557-2fc4-6201-b88d-8a24bd390f6e BootID:6b0b2603-de76-426d-a575-92fc5ddc5aed Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f6:ed:f9:5b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f6:ed:f9:5b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fa:63:eb:6b:b2:87 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 12:51:01.086038 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.086033    2569 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 12:51:01.086135 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.086107    2569 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 12:51:01.087371 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.087345    2569 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 12:51:01.087504 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.087373    2569 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-6.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 12:51:01.087548 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.087506    2569 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 12:51:01.087548 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.087514    2569 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 12:51:01.087615 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.087556    2569 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 12:51:01.087615 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.087575    2569 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 12:51:01.088592 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.088581    2569 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 12:51:01.093908 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.093896    2569 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 12:51:01.094047 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.094037    2569 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 12:51:01.094708 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.094696    2569 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 12:51:01.094756 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.094711    2569 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 12:51:01.094756 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.094725    2569 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 12:51:01.094756 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.094737    2569 kubelet.go:405] "Adding apiserver pod source"
Oct 30 12:51:01.094756 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.094749    2569 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 12:51:01.095088 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.095077    2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 12:51:01.097509 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.097481    2569 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 12:51:01.106356 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.106337    2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 12:51:01.107261 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.107248    2569 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 12:51:01.107324 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.107278    2569 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 12:51:01.107425 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.107415    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 12:51:01.107459 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.107431    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 12:51:01.107459 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.107437    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 12:51:01.107459 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.107443    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 12:51:01.107459 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.107448    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 12:51:01.107459 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.107455    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 12:51:01.108225 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.108216    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 12:51:01.108261 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.108225    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 12:51:01.108261 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.108234    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 12:51:01.108261 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.108240    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 12:51:01.108261 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.108248    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 12:51:01.108261 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.108257    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 12:51:01.108973 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.108964    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 12:51:01.108973 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.108973    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 12:51:01.112675 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.112661    2569 server.go:1263] "Started kubelet"
Oct 30 12:51:01.112793 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.112738    2569 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 12:51:01.112793 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.112759    2569 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 12:51:01.112938 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.112828    2569 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 12:51:01.113528 ip-10-0-137-6 systemd[1]: Started Kubernetes Kubelet.
Oct 30 12:51:01.114169 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.114153    2569 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 12:51:01.115084 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.115070    2569 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 12:51:01.118676 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:01.118660    2569 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 12:51:01.123352 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.123332    2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 12:51:01.124686 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.124672    2569 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 12:51:01.124832 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:01.124800    2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-6.ec2.internal\" not found"
Oct 30 12:51:01.125492 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.125475    2569 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 12:51:01.125587 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.125514    2569 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 12:51:01.125587 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.125529    2569 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 12:51:01.125587 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.125548    2569 factory.go:55] Registering systemd factory
Oct 30 12:51:01.125587 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.125562    2569 factory.go:223] Registration of the systemd container factory successfully
Oct 30 12:51:01.125754 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.125664    2569 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 12:51:01.125754 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.125671    2569 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 12:51:01.125827 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.125796    2569 factory.go:153] Registering CRI-O factory
Oct 30 12:51:01.125827 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.125807    2569 factory.go:223] Registration of the crio container factory successfully
Oct 30 12:51:01.125884 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.125870    2569 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 12:51:01.125913 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.125890    2569 factory.go:103] Registering Raw factory
Oct 30 12:51:01.125913 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.125903    2569 manager.go:1199] Started watching for new ooms in manager
Oct 30 12:51:01.128009 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.127312    2569 manager.go:319] Starting recovery of all containers
Oct 30 12:51:01.141638 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.141617    2569 manager.go:324] Recovery completed
Oct 30 12:51:01.146561 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.146537    2569 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 12:51:01.147652 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.147638    2569 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 12:51:01.149361 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.149344    2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-6.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 12:51:01.149442 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.149373    2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-6.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 12:51:01.149442 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.149382    2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-6.ec2.internal" event="NodeHasSufficientPID"
Oct 30 12:51:01.149931 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.149892    2569 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-6.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 12:51:01.149995 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.149954    2569 cpu_manager.go:226] "Starting" policy="none"
Oct 30 12:51:01.149995 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.149959    2569 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 12:51:01.149995 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.149964    2569 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 12:51:01.149995 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.149984    2569 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 12:51:01.150705 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.150689    2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-6.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 12:51:01.150769 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.150716    2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-6.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 12:51:01.150769 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.150726    2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-6.ec2.internal" event="NodeHasSufficientPID"
Oct 30 12:51:01.151222 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:01.149901    2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-6.ec2.internal.18e34fa85a5b83ad  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-6.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-6.ec2.internal,},FirstTimestamp:2026-10-30 12:51:01.112636333 +0000 UTC m=+0.352014538,LastTimestamp:2026-10-30 12:51:01.112636333 +0000 UTC m=+0.352014538,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-6.ec2.internal,}"
Oct 30 12:51:01.152211 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.152198    2569 policy_none.go:50] "Start"
Oct 30 12:51:01.152259 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.152214    2569 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 12:51:01.152259 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.152224    2569 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 12:51:01.154230 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.154219    2569 policy_none.go:44] "Start"
Oct 30 12:51:01.159383 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:01.159360    2569 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-6.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 12:51:01.177041 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.177022    2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6v4cx"
Oct 30 12:51:01.188327 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:01.188302    2569 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-6.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 12:51:01.188327 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:01.188313    2569 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 12:51:01.188457 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:01.188358    2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-6.ec2.internal.18e34fa85c8bdfcf  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-6.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-6.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-6.ec2.internal,},FirstTimestamp:2026-10-30 12:51:01.149360079 +0000 UTC m=+0.388738287,LastTimestamp:2026-10-30 12:51:01.149360079 +0000 UTC m=+0.388738287,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-6.ec2.internal,}"
Oct 30 12:51:01.188536 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:01.188501    2569 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 12:51:01.191172 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.191147    2569 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 12:51:01.193545 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.193532    2569 manager.go:341] "Starting Device Plugin manager"
Oct 30 12:51:01.193617 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:01.193563    2569 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 12:51:01.193617 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.193573    2569 server.go:87] "Starting device plugin registration server"
Oct 30 12:51:01.193821 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.193808    2569 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 12:51:01.193885 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.193824    2569 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 12:51:01.193931 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.193916    2569 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 12:51:01.194141 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.194128    2569 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 12:51:01.194141 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.194139    2569 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 12:51:01.194474 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:01.194455    2569 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 12:51:01.194554 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:01.194503    2569 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-6.ec2.internal\" not found"
Oct 30 12:51:01.199391 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.199374    2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6v4cx"
Oct 30 12:51:01.202659 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:01.202565    2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-6.ec2.internal.18e34fa85c8c2293  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-6.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-137-6.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-137-6.ec2.internal,},FirstTimestamp:2026-10-30 12:51:01.149377171 +0000 UTC m=+0.388755376,LastTimestamp:2026-10-30 12:51:01.149377171 +0000 UTC m=+0.388755376,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-6.ec2.internal,}"
Oct 30 12:51:01.257916 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.257794    2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 12:51:01.259899 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.259124    2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 12:51:01.259899 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.259143    2569 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 12:51:01.259899 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.259159    2569 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 12:51:01.259899 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:01.259187    2569 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 12:51:01.280378 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.280364    2569 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 12:51:01.294842 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.294801    2569 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 12:51:01.295730 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.295716    2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-6.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 12:51:01.295799 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.295740    2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-6.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 12:51:01.295799 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.295749    2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-6.ec2.internal" event="NodeHasSufficientPID"
Oct 30 12:51:01.295799 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.295770    2569 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-6.ec2.internal"
Oct 30 12:51:01.319226 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.319210    2569 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-137-6.ec2.internal"
Oct 30 12:51:01.319271 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:01.319229    2569 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-6.ec2.internal\": node \"ip-10-0-137-6.ec2.internal\" not found"
Oct 30 12:51:01.359948 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.359917    2569 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-6.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-6.ec2.internal"]
Oct 30 12:51:01.360038 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.359975    2569 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 12:51:01.360686 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.360672    2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-6.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 12:51:01.360766 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.360708    2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-6.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 12:51:01.360766 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.360722    2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-6.ec2.internal" event="NodeHasSufficientPID"
Oct 30 12:51:01.362840 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.362825    2569 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 12:51:01.362998 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.362984    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-6.ec2.internal"
Oct 30 12:51:01.363047 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.363016    2569 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 12:51:01.363518 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.363494    2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-6.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 12:51:01.363518 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.363515    2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-6.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 12:51:01.363655 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.363527    2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-6.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 12:51:01.363655 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.363535    2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-6.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 12:51:01.363655 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.363544    2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-6.ec2.internal" event="NodeHasSufficientPID"
Oct 30 12:51:01.363655 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.363546    2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-6.ec2.internal" event="NodeHasSufficientPID"
Oct 30 12:51:01.365530 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.365517    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-6.ec2.internal"
Oct 30 12:51:01.365622 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.365541    2569 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 12:51:01.366197 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.366180    2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-6.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 12:51:01.366261 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.366211    2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-6.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 12:51:01.366261 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.366224    2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-6.ec2.internal" event="NodeHasSufficientPID"
Oct 30 12:51:01.376854 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:01.376838    2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-6.ec2.internal\" not found"
Oct 30 12:51:01.394800 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:01.394782    2569 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-6.ec2.internal\" not found" node="ip-10-0-137-6.ec2.internal"
Oct 30 12:51:01.396818 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:01.396804    2569 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-6.ec2.internal\" not found" node="ip-10-0-137-6.ec2.internal"
Oct 30 12:51:01.427348 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.427325    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a60e17e4a3a53aed12f9737e4f90e05b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-6.ec2.internal\" (UID: \"a60e17e4a3a53aed12f9737e4f90e05b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-6.ec2.internal"
Oct 30 12:51:01.427397 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.427353    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a60e17e4a3a53aed12f9737e4f90e05b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-6.ec2.internal\" (UID: \"a60e17e4a3a53aed12f9737e4f90e05b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-6.ec2.internal"
Oct 30 12:51:01.427397 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.427368    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c8cfc7a7881c11ddfae618f888b1edd8-config\") pod \"kube-apiserver-proxy-ip-10-0-137-6.ec2.internal\" (UID: \"c8cfc7a7881c11ddfae618f888b1edd8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-6.ec2.internal"
Oct 30 12:51:01.477103 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:01.477086    2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-6.ec2.internal\" not found"
Oct 30 12:51:01.528281 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.528260    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a60e17e4a3a53aed12f9737e4f90e05b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-6.ec2.internal\" (UID: \"a60e17e4a3a53aed12f9737e4f90e05b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-6.ec2.internal"
Oct 30 12:51:01.528369 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.528283    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a60e17e4a3a53aed12f9737e4f90e05b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-6.ec2.internal\" (UID: \"a60e17e4a3a53aed12f9737e4f90e05b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-6.ec2.internal"
Oct 30 12:51:01.528369 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.528299    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c8cfc7a7881c11ddfae618f888b1edd8-config\") pod \"kube-apiserver-proxy-ip-10-0-137-6.ec2.internal\" (UID: \"c8cfc7a7881c11ddfae618f888b1edd8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-6.ec2.internal"
Oct 30 12:51:01.528369 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.528328    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c8cfc7a7881c11ddfae618f888b1edd8-config\") pod \"kube-apiserver-proxy-ip-10-0-137-6.ec2.internal\" (UID: \"c8cfc7a7881c11ddfae618f888b1edd8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-6.ec2.internal"
Oct 30 12:51:01.528369 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.528358    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a60e17e4a3a53aed12f9737e4f90e05b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-6.ec2.internal\" (UID: \"a60e17e4a3a53aed12f9737e4f90e05b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-6.ec2.internal"
Oct 30 12:51:01.528520 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.528358    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a60e17e4a3a53aed12f9737e4f90e05b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-6.ec2.internal\" (UID: \"a60e17e4a3a53aed12f9737e4f90e05b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-6.ec2.internal"
Oct 30 12:51:01.577170 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:01.577122    2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-6.ec2.internal\" not found"
Oct 30 12:51:01.677952 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:01.677932    2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-6.ec2.internal\" not found"
Oct 30 12:51:01.698225 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.698208    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-6.ec2.internal"
Oct 30 12:51:01.700364 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:01.700348    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-6.ec2.internal"
Oct 30 12:51:01.778247 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:01.778229    2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-6.ec2.internal\" not found"
Oct 30 12:51:01.878891 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:01.878851    2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-6.ec2.internal\" not found"
Oct 30 12:51:01.979365 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:01.979347    2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-6.ec2.internal\" not found"
Oct 30 12:51:02.050783 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:02.050766    2569 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 12:51:02.051119 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:02.050854    2569 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 12:51:02.080065 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:02.080044    2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-6.ec2.internal\" not found"
Oct 30 12:51:02.180458 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:02.180433    2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-6.ec2.internal\" not found"
Oct 30 12:51:02.205276 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:02.205244    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 12:46:01 +0000 UTC" deadline="2028-07-08 14:08:13.510778718 +0000 UTC"
Oct 30 12:51:02.205276 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:02.205275    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14809h17m11.305507338s"
Oct 30 12:51:02.281459 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:02.281435    2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-6.ec2.internal\" not found"
Oct 30 12:51:02.286241 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:02.286212    2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda60e17e4a3a53aed12f9737e4f90e05b.slice/crio-9fb14d5547656f78b7318c2bab087962d514662e903b402003c3830713792fe5 WatchSource:0}: Error finding container 9fb14d5547656f78b7318c2bab087962d514662e903b402003c3830713792fe5: Status 404 returned error can't find the container with id 9fb14d5547656f78b7318c2bab087962d514662e903b402003c3830713792fe5
Oct 30 12:51:02.292644 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:02.292624    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 12:51:02.296334 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:02.296314    2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8cfc7a7881c11ddfae618f888b1edd8.slice/crio-febe2ba051c1b55bf7ad7e46d5651b755cb0e91501337c63e0d74aea981ca5dd WatchSource:0}: Error finding container febe2ba051c1b55bf7ad7e46d5651b755cb0e91501337c63e0d74aea981ca5dd: Status 404 returned error can't find the container with id febe2ba051c1b55bf7ad7e46d5651b755cb0e91501337c63e0d74aea981ca5dd
Oct 30 12:51:02.298353 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:02.298335    2569 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 12:51:02.383131 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:02.383105    2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-6.ec2.internal\" not found"
Oct 30 12:51:02.383131 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:02.383126    2569 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 12:51:02.384250 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:02.384229    2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-6.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 12:51:02.384340 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:02.384256    2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-6.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 12:51:02.384340 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:02.384266    2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-6.ec2.internal" event="NodeHasSufficientPID"
Oct 30 12:51:02.485743 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:02.485695    2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-6.ec2.internal\" not found"
Oct 30 12:51:02.544208 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:02.544187    2569 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 12:51:02.585907 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:02.585882    2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-6.ec2.internal\" not found"
Oct 30 12:51:02.593414 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:02.593394    2569 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 12:51:02.625774 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:02.625745    2569 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-6.ec2.internal"
Oct 30 12:51:02.655846 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:02.655827    2569 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 12:51:02.656955 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:02.656925    2569 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-6.ec2.internal"
Oct 30 12:51:02.695013 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:02.694991    2569 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 12:51:03.095902 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.095875    2569 apiserver.go:52] "Watching apiserver"
Oct 30 12:51:03.125280 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.125234    2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 12:51:03.131037 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.131016    2569 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 12:51:03.131440 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.131416    2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-zvkh9","openshift-multus/multus-mv5cv","openshift-multus/network-metrics-daemon-79wgd","openshift-network-diagnostics/network-check-target-dlxch","kube-system/global-pull-secret-syncer-rxg7f","kube-system/konnectivity-agent-vlfkx","kube-system/kube-apiserver-proxy-ip-10-0-137-6.ec2.internal","openshift-cluster-node-tuning-operator/tuned-7r6q9","openshift-image-registry/node-ca-7pxn6","openshift-network-operator/iptables-alerter-9p6w9","openshift-ovn-kubernetes/ovnkube-node-wpkrj","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5mdl","openshift-dns/node-resolver-7mvnz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-6.ec2.internal"]
Oct 30 12:51:03.134324 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.134298    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9p6w9"
Oct 30 12:51:03.136643 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.136622    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7mvnz"
Oct 30 12:51:03.139239 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.139218    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-79wgd"
Oct 30 12:51:03.139338 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:03.139290    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-79wgd" podUID="69eab030-f488-433c-869a-52d628824391"
Oct 30 12:51:03.141596 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.141571    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dlxch"
Oct 30 12:51:03.141720 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:03.141671    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dlxch" podUID="c1927296-d775-43c4-a874-8842ebfeb3c8"
Oct 30 12:51:03.145049 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.145029    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rxg7f"
Oct 30 12:51:03.145141 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:03.145106    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rxg7f" podUID="815986e1-07b4-43fb-95f5-745316bd5105"
Oct 30 12:51:03.147432 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.147411    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vlfkx"
Oct 30 12:51:03.151404 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.151306    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7pxn6"
Oct 30 12:51:03.153795 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.153779    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 12:51:03.156373 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.156350    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:03.156487 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.156426    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:03.156712 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.156691    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-g6csn\""
Oct 30 12:51:03.156823 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.156798    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 12:51:03.156968 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.156949    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 12:51:03.157248 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.157232    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9fz8p\""
Oct 30 12:51:03.157315 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.157306    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 12:51:03.158812 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.158415    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zvkh9"
Oct 30 12:51:03.159425 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.159390    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 12:51:03.161408 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.161385    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:03.167261 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.167234    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5mdl"
Oct 30 12:51:03.190568 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.190541    2569 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 12:51:03.190702 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.190610    2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-d9s77"
Oct 30 12:51:03.226619 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.226584    2569 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 12:51:03.238549 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.238525    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7d4b7525-1c1b-4e65-b263-349ab1a0f32f-konnectivity-ca\") pod \"konnectivity-agent-vlfkx\" (UID: \"7d4b7525-1c1b-4e65-b263-349ab1a0f32f\") " pod="kube-system/konnectivity-agent-vlfkx"
Oct 30 12:51:03.238678 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.238556    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-log-socket\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:03.238678 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.238579    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f8f7d922-7364-444f-a932-278628e26b83-socket-dir\") pod \"aws-ebs-csi-driver-node-p5mdl\" (UID: \"f8f7d922-7364-444f-a932-278628e26b83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5mdl"
Oct 30 12:51:03.238678 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.238604    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/19115051-3b70-47cd-a7a3-313b1b3c26c7-iptables-alerter-script\") pod \"iptables-alerter-9p6w9\" (UID: \"19115051-3b70-47cd-a7a3-313b1b3c26c7\") " pod="openshift-network-operator/iptables-alerter-9p6w9"
Oct 30 12:51:03.238872 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.238707    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/85602130-14de-443d-b237-b213c0ce8b82-etc-sysctl-d\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:03.238872 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.238841    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-multus-cni-dir\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:03.238980 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.238870    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a1ca584f-61b1-4398-a0f6-5788e476aa20-multus-daemon-config\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:03.238980 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.238898    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ff0a920b-f15f-4909-906b-61dc41ef1263-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zvkh9\" (UID: \"ff0a920b-f15f-4909-906b-61dc41ef1263\") " pod="openshift-multus/multus-additional-cni-plugins-zvkh9"
Oct 30 12:51:03.238980 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.238924    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f8f7d922-7364-444f-a932-278628e26b83-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p5mdl\" (UID: \"f8f7d922-7364-444f-a932-278628e26b83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5mdl"
Oct 30 12:51:03.238980 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.238965    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c83bcf7b-a1d6-457b-9d7c-bb03c2cbd449-hosts-file\") pod \"node-resolver-7mvnz\" (UID: \"c83bcf7b-a1d6-457b-9d7c-bb03c2cbd449\") " pod="openshift-dns/node-resolver-7mvnz"
Oct 30 12:51:03.239187 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.239002    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/31d09b91-c973-4cf9-a7d8-d602ff9994f6-host\") pod \"node-ca-7pxn6\" (UID: \"31d09b91-c973-4cf9-a7d8-d602ff9994f6\") " pod="openshift-image-registry/node-ca-7pxn6"
Oct 30 12:51:03.239187 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.239039    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/85602130-14de-443d-b237-b213c0ce8b82-etc-sysctl-conf\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:03.239187 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.239077    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/85602130-14de-443d-b237-b213c0ce8b82-sys\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:03.239187 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.239094    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f8f7d922-7364-444f-a932-278628e26b83-registration-dir\") pod \"aws-ebs-csi-driver-node-p5mdl\" (UID: \"f8f7d922-7364-444f-a932-278628e26b83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5mdl"
Oct 30 12:51:03.239187 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.239117    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f8f7d922-7364-444f-a932-278628e26b83-sys-fs\") pod \"aws-ebs-csi-driver-node-p5mdl\" (UID: \"f8f7d922-7364-444f-a932-278628e26b83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5mdl"
Oct 30 12:51:03.239187 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.239137    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85602130-14de-443d-b237-b213c0ce8b82-tmp\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:03.239187 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.239152    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/815986e1-07b4-43fb-95f5-745316bd5105-dbus\") pod \"global-pull-secret-syncer-rxg7f\" (UID: \"815986e1-07b4-43fb-95f5-745316bd5105\") " pod="kube-system/global-pull-secret-syncer-rxg7f"
Oct 30 12:51:03.239187 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.239167    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-host-run-ovn-kubernetes\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:03.239583 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.239220    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-host-cni-bin\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:03.239583 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.239242    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2d5efe23-0e0d-4a4e-9d03-42829f371c09-env-overrides\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:03.239583 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.239268    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78v2l\" (UniqueName: \"kubernetes.io/projected/19115051-3b70-47cd-a7a3-313b1b3c26c7-kube-api-access-78v2l\") pod \"iptables-alerter-9p6w9\" (UID: \"19115051-3b70-47cd-a7a3-313b1b3c26c7\") " pod="openshift-network-operator/iptables-alerter-9p6w9"
Oct 30 12:51:03.239583 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.239295    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-multus-socket-dir-parent\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:03.239583 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.239318    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-var-lib-openvswitch\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:03.239583 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.239337    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c83bcf7b-a1d6-457b-9d7c-bb03c2cbd449-tmp-dir\") pod \"node-resolver-7mvnz\" (UID: \"c83bcf7b-a1d6-457b-9d7c-bb03c2cbd449\") " pod="openshift-dns/node-resolver-7mvnz"
Oct 30 12:51:03.239583 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.239356    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfwjf\" (UniqueName: \"kubernetes.io/projected/c1927296-d775-43c4-a874-8842ebfeb3c8-kube-api-access-jfwjf\") pod \"network-check-target-dlxch\" (UID: \"c1927296-d775-43c4-a874-8842ebfeb3c8\") " pod="openshift-network-diagnostics/network-check-target-dlxch"
Oct 30 12:51:03.239583 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.239373    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85602130-14de-443d-b237-b213c0ce8b82-etc-kubernetes\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:03.239583 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.239388    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-cnibin\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:03.239583 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.239412    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-host-slash\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:03.239583 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.239561    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdplx\" (UniqueName: \"kubernetes.io/projected/31d09b91-c973-4cf9-a7d8-d602ff9994f6-kube-api-access-zdplx\") pod \"node-ca-7pxn6\" (UID: \"31d09b91-c973-4cf9-a7d8-d602ff9994f6\") " pod="openshift-image-registry/node-ca-7pxn6"
Oct 30 12:51:03.240145 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.239588    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/85602130-14de-443d-b237-b213c0ce8b82-etc-systemd\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:03.240145 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.239621    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-host-run-k8s-cni-cncf-io\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:03.240145 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.239660    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-etc-kubernetes\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:03.240145 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.239676    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-run-openvswitch\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:03.240145 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.239690    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/31d09b91-c973-4cf9-a7d8-d602ff9994f6-serviceca\") pod \"node-ca-7pxn6\" (UID: \"31d09b91-c973-4cf9-a7d8-d602ff9994f6\") " pod="openshift-image-registry/node-ca-7pxn6"
Oct 30 12:51:03.240145 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.239710    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-host-var-lib-kubelet\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:03.240145 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.239752    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pppc\" (UniqueName: \"kubernetes.io/projected/a1ca584f-61b1-4398-a0f6-5788e476aa20-kube-api-access-7pppc\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:03.240145 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.239772    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/85602130-14de-443d-b237-b213c0ce8b82-etc-sysconfig\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:03.240145 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.239789    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-host-var-lib-cni-bin\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:03.240145 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.239808    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-multus-conf-dir\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:03.240145 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.239829    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-host-run-netns\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:03.240145 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.239845    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-run-systemd\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:03.240145 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.239859    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-node-log\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:03.240145 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.239879    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2d5efe23-0e0d-4a4e-9d03-42829f371c09-ovnkube-config\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:03.240145 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.239921    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtppn\" (UniqueName: \"kubernetes.io/projected/f8f7d922-7364-444f-a932-278628e26b83-kube-api-access-wtppn\") pod \"aws-ebs-csi-driver-node-p5mdl\" (UID: \"f8f7d922-7364-444f-a932-278628e26b83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5mdl"
Oct 30 12:51:03.240145 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.239937    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-systemd-units\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:03.240145 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.239955    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxq79\" (UniqueName: \"kubernetes.io/projected/69eab030-f488-433c-869a-52d628824391-kube-api-access-hxq79\") pod \"network-metrics-daemon-79wgd\" (UID: \"69eab030-f488-433c-869a-52d628824391\") " pod="openshift-multus/network-metrics-daemon-79wgd"
Oct 30 12:51:03.240995 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.239971    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/815986e1-07b4-43fb-95f5-745316bd5105-original-pull-secret\") pod \"global-pull-secret-syncer-rxg7f\" (UID: \"815986e1-07b4-43fb-95f5-745316bd5105\") " pod="kube-system/global-pull-secret-syncer-rxg7f"
Oct 30 12:51:03.240995 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.239993    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-host-cni-netd\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:03.240995 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.240017    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2d5efe23-0e0d-4a4e-9d03-42829f371c09-ovnkube-script-lib\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:03.240995 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.240042    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-host-var-lib-cni-multus\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:03.240995 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.240066    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ff0a920b-f15f-4909-906b-61dc41ef1263-cni-binary-copy\") pod \"multus-additional-cni-plugins-zvkh9\" (UID: \"ff0a920b-f15f-4909-906b-61dc41ef1263\") " pod="openshift-multus/multus-additional-cni-plugins-zvkh9"
Oct 30 12:51:03.240995 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.240088    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:03.240995 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.240105    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls6hw\" (UniqueName: \"kubernetes.io/projected/c83bcf7b-a1d6-457b-9d7c-bb03c2cbd449-kube-api-access-ls6hw\") pod \"node-resolver-7mvnz\" (UID: \"c83bcf7b-a1d6-457b-9d7c-bb03c2cbd449\") " pod="openshift-dns/node-resolver-7mvnz"
Oct 30 12:51:03.240995 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.240121    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-749nf\" (UniqueName: \"kubernetes.io/projected/85602130-14de-443d-b237-b213c0ce8b82-kube-api-access-749nf\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:03.240995 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.240136    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ff0a920b-f15f-4909-906b-61dc41ef1263-cnibin\") pod \"multus-additional-cni-plugins-zvkh9\" (UID: \"ff0a920b-f15f-4909-906b-61dc41ef1263\") " pod="openshift-multus/multus-additional-cni-plugins-zvkh9"
Oct 30 12:51:03.240995 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.240150    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ff0a920b-f15f-4909-906b-61dc41ef1263-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zvkh9\" (UID: \"ff0a920b-f15f-4909-906b-61dc41ef1263\") " pod="openshift-multus/multus-additional-cni-plugins-zvkh9"
Oct 30 12:51:03.240995 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.240166    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ff0a920b-f15f-4909-906b-61dc41ef1263-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zvkh9\" (UID: \"ff0a920b-f15f-4909-906b-61dc41ef1263\") " pod="openshift-multus/multus-additional-cni-plugins-zvkh9"
Oct 30 12:51:03.240995 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.240182    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f8f7d922-7364-444f-a932-278628e26b83-device-dir\") pod \"aws-ebs-csi-driver-node-p5mdl\" (UID: \"f8f7d922-7364-444f-a932-278628e26b83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5mdl"
Oct 30 12:51:03.240995 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.240255    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/85602130-14de-443d-b237-b213c0ce8b82-etc-modprobe-d\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:03.240995 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.240275    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/85602130-14de-443d-b237-b213c0ce8b82-var-lib-kubelet\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:03.240995 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.240293    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69eab030-f488-433c-869a-52d628824391-metrics-certs\") pod \"network-metrics-daemon-79wgd\" (UID: \"69eab030-f488-433c-869a-52d628824391\") " pod="openshift-multus/network-metrics-daemon-79wgd"
Oct 30 12:51:03.240995 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.240315    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f8f7d922-7364-444f-a932-278628e26b83-etc-selinux\") pod \"aws-ebs-csi-driver-node-p5mdl\" (UID: \"f8f7d922-7364-444f-a932-278628e26b83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5mdl"
Oct 30 12:51:03.241784 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.240335    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/85602130-14de-443d-b237-b213c0ce8b82-etc-tuned\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:03.241784 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.240350    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-system-cni-dir\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:03.241784 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.240367    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/815986e1-07b4-43fb-95f5-745316bd5105-kubelet-config\") pod \"global-pull-secret-syncer-rxg7f\" (UID: \"815986e1-07b4-43fb-95f5-745316bd5105\") " pod="kube-system/global-pull-secret-syncer-rxg7f"
Oct 30 12:51:03.241784 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.240388    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7d4b7525-1c1b-4e65-b263-349ab1a0f32f-agent-certs\") pod \"konnectivity-agent-vlfkx\" (UID: \"7d4b7525-1c1b-4e65-b263-349ab1a0f32f\") " pod="kube-system/konnectivity-agent-vlfkx"
Oct 30 12:51:03.241784 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.240412    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-host-kubelet\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:03.241784 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.240429    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-etc-openvswitch\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:03.241784 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.240444    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/85602130-14de-443d-b237-b213c0ce8b82-host\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:03.241784 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.240459    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-host-run-netns\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:03.241784 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.240484    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-hostroot\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:03.241784 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.240506    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6s7g\" (UniqueName: \"kubernetes.io/projected/ff0a920b-f15f-4909-906b-61dc41ef1263-kube-api-access-q6s7g\") pod \"multus-additional-cni-plugins-zvkh9\" (UID: \"ff0a920b-f15f-4909-906b-61dc41ef1263\") " pod="openshift-multus/multus-additional-cni-plugins-zvkh9"
Oct 30 12:51:03.241784 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.240527    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2d5efe23-0e0d-4a4e-9d03-42829f371c09-ovn-node-metrics-cert\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:03.241784 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.240549    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5s5m\" (UniqueName: \"kubernetes.io/projected/2d5efe23-0e0d-4a4e-9d03-42829f371c09-kube-api-access-x5s5m\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:03.241784 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.240567    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/19115051-3b70-47cd-a7a3-313b1b3c26c7-host-slash\") pod \"iptables-alerter-9p6w9\" (UID: \"19115051-3b70-47cd-a7a3-313b1b3c26c7\") " pod="openshift-network-operator/iptables-alerter-9p6w9"
Oct 30 12:51:03.241784 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.240594    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a1ca584f-61b1-4398-a0f6-5788e476aa20-cni-binary-copy\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:03.241784 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.240612    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-host-run-multus-certs\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:03.241784 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.240628    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ff0a920b-f15f-4909-906b-61dc41ef1263-system-cni-dir\") pod \"multus-additional-cni-plugins-zvkh9\" (UID: \"ff0a920b-f15f-4909-906b-61dc41ef1263\") " pod="openshift-multus/multus-additional-cni-plugins-zvkh9"
Oct 30 12:51:03.241784 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.240643    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-run-ovn\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:03.242693 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.240661    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/85602130-14de-443d-b237-b213c0ce8b82-run\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:03.242693 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.240700    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/85602130-14de-443d-b237-b213c0ce8b82-lib-modules\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:03.262032 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.261995    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 12:51:03.265339 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.264531    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 12:51:03.265339 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.264620    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-78z4l\""
Oct 30 12:51:03.265339 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.264718    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 12:51:03.265339 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.264867    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-588rd\""
Oct 30 12:51:03.265339 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.264874    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 12:51:03.265339 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.264904    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 12:51:03.265339 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.264939    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 12:51:03.265339 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.264953    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 12:51:03.265339 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.264984    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 12:51:03.265339 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.264996    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 12:51:03.265339 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.265022    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 12:51:03.265339 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.265032    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 12:51:03.265339 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.265052    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 12:51:03.265339 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.265062    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 12:51:03.265339 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.265098    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 12:51:03.265339 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.265111    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 12:51:03.265339 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.265162    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 12:51:03.265339 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.265193    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-fv4k6\""
Oct 30 12:51:03.265339 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.265201    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pdl9w\""
Oct 30 12:51:03.265339 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.265219    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 12:51:03.265339 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.265069    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 12:51:03.265339 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.265237    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 12:51:03.265339 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.264997    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 12:51:03.265339 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.265288    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 12:51:03.265339 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.265327    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 12:51:03.265339 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.265339    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-d8ng4\""
Oct 30 12:51:03.266656 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.265437    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-nklv7\""
Oct 30 12:51:03.266656 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.265948    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-6.ec2.internal" event={"ID":"c8cfc7a7881c11ddfae618f888b1edd8","Type":"ContainerStarted","Data":"febe2ba051c1b55bf7ad7e46d5651b755cb0e91501337c63e0d74aea981ca5dd"}
Oct 30 12:51:03.267290 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.267267    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-6.ec2.internal" event={"ID":"a60e17e4a3a53aed12f9737e4f90e05b","Type":"ContainerStarted","Data":"9fb14d5547656f78b7318c2bab087962d514662e903b402003c3830713792fe5"}
Oct 30 12:51:03.268540 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.268520    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-hfgh7\""
Oct 30 12:51:03.308402 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.308379    2569 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 12:51:03.341181 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.341153    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-host-run-netns\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:03.341263 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.341190    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-run-systemd\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:03.341263 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.341216    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-node-log\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:03.341263 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.341240    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2d5efe23-0e0d-4a4e-9d03-42829f371c09-ovnkube-config\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:03.341412 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.341266    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtppn\" (UniqueName: \"kubernetes.io/projected/f8f7d922-7364-444f-a932-278628e26b83-kube-api-access-wtppn\") pod \"aws-ebs-csi-driver-node-p5mdl\" (UID: \"f8f7d922-7364-444f-a932-278628e26b83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5mdl"
Oct 30 12:51:03.341412 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.341290    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-systemd-units\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:03.341412 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.341314    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxq79\" (UniqueName: \"kubernetes.io/projected/69eab030-f488-433c-869a-52d628824391-kube-api-access-hxq79\") pod \"network-metrics-daemon-79wgd\" (UID: \"69eab030-f488-433c-869a-52d628824391\") " pod="openshift-multus/network-metrics-daemon-79wgd"
Oct 30 12:51:03.341412 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.341341    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/815986e1-07b4-43fb-95f5-745316bd5105-original-pull-secret\") pod \"global-pull-secret-syncer-rxg7f\" (UID: \"815986e1-07b4-43fb-95f5-745316bd5105\") " pod="kube-system/global-pull-secret-syncer-rxg7f"
Oct 30 12:51:03.341412 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.341364    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-host-cni-netd\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:03.341412 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.341387    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2d5efe23-0e0d-4a4e-9d03-42829f371c09-ovnkube-script-lib\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:03.341412 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.341411    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-host-var-lib-cni-multus\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:03.341710 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.341439    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ff0a920b-f15f-4909-906b-61dc41ef1263-cni-binary-copy\") pod \"multus-additional-cni-plugins-zvkh9\" (UID: \"ff0a920b-f15f-4909-906b-61dc41ef1263\") " pod="openshift-multus/multus-additional-cni-plugins-zvkh9"
Oct 30 12:51:03.341710 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.341465    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:03.341710 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.341490    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls6hw\" (UniqueName: \"kubernetes.io/projected/c83bcf7b-a1d6-457b-9d7c-bb03c2cbd449-kube-api-access-ls6hw\") pod \"node-resolver-7mvnz\" (UID: \"c83bcf7b-a1d6-457b-9d7c-bb03c2cbd449\") " pod="openshift-dns/node-resolver-7mvnz"
Oct 30 12:51:03.341710 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.341522    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-749nf\" (UniqueName: \"kubernetes.io/projected/85602130-14de-443d-b237-b213c0ce8b82-kube-api-access-749nf\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:03.341710 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.341533    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-systemd-units\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:03.341710 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.341549    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ff0a920b-f15f-4909-906b-61dc41ef1263-cnibin\") pod \"multus-additional-cni-plugins-zvkh9\" (UID: \"ff0a920b-f15f-4909-906b-61dc41ef1263\") " pod="openshift-multus/multus-additional-cni-plugins-zvkh9"
Oct 30 12:51:03.341710 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.341573    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ff0a920b-f15f-4909-906b-61dc41ef1263-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zvkh9\" (UID: \"ff0a920b-f15f-4909-906b-61dc41ef1263\") " pod="openshift-multus/multus-additional-cni-plugins-zvkh9"
Oct 30 12:51:03.341710 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.341591    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-host-run-netns\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:03.341710 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.341602    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ff0a920b-f15f-4909-906b-61dc41ef1263-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zvkh9\" (UID: \"ff0a920b-f15f-4909-906b-61dc41ef1263\") " pod="openshift-multus/multus-additional-cni-plugins-zvkh9"
Oct 30 12:51:03.341710 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.341628    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f8f7d922-7364-444f-a932-278628e26b83-device-dir\") pod \"aws-ebs-csi-driver-node-p5mdl\" (UID: \"f8f7d922-7364-444f-a932-278628e26b83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5mdl"
Oct 30 12:51:03.341710 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.341637    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-run-systemd\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:03.341710 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.341655    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/85602130-14de-443d-b237-b213c0ce8b82-etc-modprobe-d\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:03.341710 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.341695    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-node-log\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:03.341710 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.341703    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/85602130-14de-443d-b237-b213c0ce8b82-var-lib-kubelet\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:03.342232 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.341730    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69eab030-f488-433c-869a-52d628824391-metrics-certs\") pod \"network-metrics-daemon-79wgd\" (UID: \"69eab030-f488-433c-869a-52d628824391\") " pod="openshift-multus/network-metrics-daemon-79wgd"
Oct 30 12:51:03.342232 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.341755    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f8f7d922-7364-444f-a932-278628e26b83-etc-selinux\") pod \"aws-ebs-csi-driver-node-p5mdl\" (UID: \"f8f7d922-7364-444f-a932-278628e26b83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5mdl"
Oct 30 12:51:03.342232 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.341781    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/85602130-14de-443d-b237-b213c0ce8b82-etc-tuned\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:03.342232 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.341804    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-system-cni-dir\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:03.342232 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.341830    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/815986e1-07b4-43fb-95f5-745316bd5105-kubelet-config\") pod \"global-pull-secret-syncer-rxg7f\" (UID: \"815986e1-07b4-43fb-95f5-745316bd5105\") " pod="kube-system/global-pull-secret-syncer-rxg7f"
Oct 30 12:51:03.342232 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.341853    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7d4b7525-1c1b-4e65-b263-349ab1a0f32f-agent-certs\") pod \"konnectivity-agent-vlfkx\" (UID: \"7d4b7525-1c1b-4e65-b263-349ab1a0f32f\") " pod="kube-system/konnectivity-agent-vlfkx"
Oct 30 12:51:03.342232 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.341876    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-host-kubelet\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:03.342232 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.341901    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-etc-openvswitch\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:03.342232 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.341927    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/85602130-14de-443d-b237-b213c0ce8b82-host\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:03.342232 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.341952    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-host-run-netns\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:03.342232 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.341979    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-hostroot\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:03.342232 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.342007    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6s7g\" (UniqueName: \"kubernetes.io/projected/ff0a920b-f15f-4909-906b-61dc41ef1263-kube-api-access-q6s7g\") pod \"multus-additional-cni-plugins-zvkh9\" (UID: \"ff0a920b-f15f-4909-906b-61dc41ef1263\") " pod="openshift-multus/multus-additional-cni-plugins-zvkh9"
Oct 30 12:51:03.342232 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.342034    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2d5efe23-0e0d-4a4e-9d03-42829f371c09-ovn-node-metrics-cert\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:03.342232 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.342059    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5s5m\" (UniqueName: \"kubernetes.io/projected/2d5efe23-0e0d-4a4e-9d03-42829f371c09-kube-api-access-x5s5m\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:03.342232 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.342084    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/19115051-3b70-47cd-a7a3-313b1b3c26c7-host-slash\") pod \"iptables-alerter-9p6w9\" (UID: \"19115051-3b70-47cd-a7a3-313b1b3c26c7\") " pod="openshift-network-operator/iptables-alerter-9p6w9"
Oct 30 12:51:03.342232 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.342109    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a1ca584f-61b1-4398-a0f6-5788e476aa20-cni-binary-copy\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:03.342232 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.342135    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-host-run-multus-certs\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:03.342988 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.342165    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ff0a920b-f15f-4909-906b-61dc41ef1263-system-cni-dir\") pod \"multus-additional-cni-plugins-zvkh9\" (UID: \"ff0a920b-f15f-4909-906b-61dc41ef1263\") " pod="openshift-multus/multus-additional-cni-plugins-zvkh9"
Oct 30 12:51:03.342988 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.342194    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-run-ovn\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:03.342988 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.342221    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/85602130-14de-443d-b237-b213c0ce8b82-run\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:03.342988 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.342244    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/85602130-14de-443d-b237-b213c0ce8b82-lib-modules\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:03.342988 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.342271    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7d4b7525-1c1b-4e65-b263-349ab1a0f32f-konnectivity-ca\") pod \"konnectivity-agent-vlfkx\" (UID: \"7d4b7525-1c1b-4e65-b263-349ab1a0f32f\") " pod="kube-system/konnectivity-agent-vlfkx"
Oct 30 12:51:03.342988 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.342295    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-log-socket\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:03.342988 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.342320    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f8f7d922-7364-444f-a932-278628e26b83-socket-dir\") pod \"aws-ebs-csi-driver-node-p5mdl\" (UID: \"f8f7d922-7364-444f-a932-278628e26b83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5mdl"
Oct 30 12:51:03.342988 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.342342    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/19115051-3b70-47cd-a7a3-313b1b3c26c7-iptables-alerter-script\") pod \"iptables-alerter-9p6w9\" (UID: \"19115051-3b70-47cd-a7a3-313b1b3c26c7\") " pod="openshift-network-operator/iptables-alerter-9p6w9"
Oct 30 12:51:03.342988 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.342360    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/85602130-14de-443d-b237-b213c0ce8b82-etc-sysctl-d\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:03.342988 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.342381    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-multus-cni-dir\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:03.342988 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.342397    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a1ca584f-61b1-4398-a0f6-5788e476aa20-multus-daemon-config\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:03.342988 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.342421    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ff0a920b-f15f-4909-906b-61dc41ef1263-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zvkh9\" (UID: \"ff0a920b-f15f-4909-906b-61dc41ef1263\") " pod="openshift-multus/multus-additional-cni-plugins-zvkh9"
Oct 30 12:51:03.342988 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.342449    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f8f7d922-7364-444f-a932-278628e26b83-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p5mdl\" (UID: \"f8f7d922-7364-444f-a932-278628e26b83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5mdl"
Oct 30 12:51:03.342988 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.342477    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c83bcf7b-a1d6-457b-9d7c-bb03c2cbd449-hosts-file\") pod \"node-resolver-7mvnz\" (UID: \"c83bcf7b-a1d6-457b-9d7c-bb03c2cbd449\") " pod="openshift-dns/node-resolver-7mvnz"
Oct 30 12:51:03.342988 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.342502    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/31d09b91-c973-4cf9-a7d8-d602ff9994f6-host\") pod \"node-ca-7pxn6\" (UID: \"31d09b91-c973-4cf9-a7d8-d602ff9994f6\") " pod="openshift-image-registry/node-ca-7pxn6"
Oct 30 12:51:03.342988 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.342529    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/85602130-14de-443d-b237-b213c0ce8b82-etc-sysctl-conf\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:03.342988 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.342550    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/85602130-14de-443d-b237-b213c0ce8b82-sys\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:03.343694 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.342599    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f8f7d922-7364-444f-a932-278628e26b83-registration-dir\") pod \"aws-ebs-csi-driver-node-p5mdl\" (UID: \"f8f7d922-7364-444f-a932-278628e26b83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5mdl"
Oct 30 12:51:03.343694 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.342627    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f8f7d922-7364-444f-a932-278628e26b83-sys-fs\") pod \"aws-ebs-csi-driver-node-p5mdl\" (UID: \"f8f7d922-7364-444f-a932-278628e26b83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5mdl"
Oct 30 12:51:03.343694 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.342652    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85602130-14de-443d-b237-b213c0ce8b82-tmp\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:03.343694 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.342661    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-host-run-netns\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:03.343694 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.342667    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/815986e1-07b4-43fb-95f5-745316bd5105-dbus\") pod \"global-pull-secret-syncer-rxg7f\" (UID: \"815986e1-07b4-43fb-95f5-745316bd5105\") " pod="kube-system/global-pull-secret-syncer-rxg7f"
Oct 30 12:51:03.343694 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.342705    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-host-run-ovn-kubernetes\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:03.343694 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.342740    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-host-cni-bin\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:03.343694 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.342763    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2d5efe23-0e0d-4a4e-9d03-42829f371c09-env-overrides\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:03.343694 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.342788    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78v2l\" (UniqueName: \"kubernetes.io/projected/19115051-3b70-47cd-a7a3-313b1b3c26c7-kube-api-access-78v2l\") pod \"iptables-alerter-9p6w9\" (UID: \"19115051-3b70-47cd-a7a3-313b1b3c26c7\") " pod="openshift-network-operator/iptables-alerter-9p6w9"
Oct 30 12:51:03.343694 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.342816    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-multus-socket-dir-parent\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:03.343694 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.342843    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-var-lib-openvswitch\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:03.343694 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.342870    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c83bcf7b-a1d6-457b-9d7c-bb03c2cbd449-tmp-dir\") pod \"node-resolver-7mvnz\" (UID: \"c83bcf7b-a1d6-457b-9d7c-bb03c2cbd449\") " pod="openshift-dns/node-resolver-7mvnz"
Oct 30 12:51:03.343694 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.342898    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfwjf\" (UniqueName: \"kubernetes.io/projected/c1927296-d775-43c4-a874-8842ebfeb3c8-kube-api-access-jfwjf\") pod \"network-check-target-dlxch\" (UID: \"c1927296-d775-43c4-a874-8842ebfeb3c8\") " pod="openshift-network-diagnostics/network-check-target-dlxch"
Oct 30 12:51:03.343694 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.342927    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85602130-14de-443d-b237-b213c0ce8b82-etc-kubernetes\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:03.343694 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.342951    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-cnibin\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:03.343694 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.342967    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/85602130-14de-443d-b237-b213c0ce8b82-var-lib-kubelet\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:03.343694 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.342972    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-host-slash\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:03.344481 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.343025    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-host-slash\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:03.344481 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:03.343039    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:51:03.344481 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.343075    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-hostroot\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:03.344481 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:03.343104    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69eab030-f488-433c-869a-52d628824391-metrics-certs podName:69eab030-f488-433c-869a-52d628824391 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:03.843084155 +0000 UTC m=+3.082462352 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/69eab030-f488-433c-869a-52d628824391-metrics-certs") pod "network-metrics-daemon-79wgd" (UID: "69eab030-f488-433c-869a-52d628824391") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:51:03.344481 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.343377    2569 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 12:51:03.344481 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.343530    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2d5efe23-0e0d-4a4e-9d03-42829f371c09-ovnkube-script-lib\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:03.344481 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.343602    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-host-var-lib-cni-multus\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:03.344481 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.343809    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-etc-openvswitch\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:03.344481 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.343876    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/19115051-3b70-47cd-a7a3-313b1b3c26c7-host-slash\") pod \"iptables-alerter-9p6w9\" (UID: \"19115051-3b70-47cd-a7a3-313b1b3c26c7\") " pod="openshift-network-operator/iptables-alerter-9p6w9"
Oct 30 12:51:03.344481 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.343883    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/815986e1-07b4-43fb-95f5-745316bd5105-kubelet-config\") pod \"global-pull-secret-syncer-rxg7f\" (UID: \"815986e1-07b4-43fb-95f5-745316bd5105\") " pod="kube-system/global-pull-secret-syncer-rxg7f"
Oct 30 12:51:03.359588 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.350767    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/85602130-14de-443d-b237-b213c0ce8b82-host\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:03.359588 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.350838    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdplx\" (UniqueName: \"kubernetes.io/projected/31d09b91-c973-4cf9-a7d8-d602ff9994f6-kube-api-access-zdplx\") pod \"node-ca-7pxn6\" (UID: \"31d09b91-c973-4cf9-a7d8-d602ff9994f6\") " pod="openshift-image-registry/node-ca-7pxn6"
Oct 30 12:51:03.359588 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.350900    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/85602130-14de-443d-b237-b213c0ce8b82-etc-systemd\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:03.359588 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.350943    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-host-run-k8s-cni-cncf-io\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:03.359588 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.350979    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-etc-kubernetes\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:03.359588 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.351007    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-run-openvswitch\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:03.359588 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.351041    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/31d09b91-c973-4cf9-a7d8-d602ff9994f6-serviceca\") pod \"node-ca-7pxn6\" (UID: \"31d09b91-c973-4cf9-a7d8-d602ff9994f6\") " pod="openshift-image-registry/node-ca-7pxn6"
Oct 30 12:51:03.359588 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.351077    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-host-var-lib-kubelet\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:03.359588 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.351112    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pppc\" (UniqueName: \"kubernetes.io/projected/a1ca584f-61b1-4398-a0f6-5788e476aa20-kube-api-access-7pppc\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:03.359588 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.351145    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/85602130-14de-443d-b237-b213c0ce8b82-etc-sysconfig\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:03.359588 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.351169    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-host-var-lib-cni-bin\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:03.359588 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.351200    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-multus-conf-dir\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:03.359588 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.351284    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-multus-conf-dir\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:03.359588 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.343025    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/85602130-14de-443d-b237-b213c0ce8b82-etc-modprobe-d\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:03.359588 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.351767    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/85602130-14de-443d-b237-b213c0ce8b82-etc-systemd\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:03.359588 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.351827    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-host-run-k8s-cni-cncf-io\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:03.359588 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.351878    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-etc-kubernetes\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:03.359588 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.351932    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-run-openvswitch\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:03.360434 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.352408    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/31d09b91-c973-4cf9-a7d8-d602ff9994f6-serviceca\") pod \"node-ca-7pxn6\" (UID: \"31d09b91-c973-4cf9-a7d8-d602ff9994f6\") " pod="openshift-image-registry/node-ca-7pxn6"
Oct 30 12:51:03.360434 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.352478    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-host-var-lib-kubelet\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:03.360434 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.352765    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ff0a920b-f15f-4909-906b-61dc41ef1263-cni-binary-copy\") pod \"multus-additional-cni-plugins-zvkh9\" (UID: \"ff0a920b-f15f-4909-906b-61dc41ef1263\") " pod="openshift-multus/multus-additional-cni-plugins-zvkh9"
Oct 30 12:51:03.360434 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.353058    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/85602130-14de-443d-b237-b213c0ce8b82-etc-sysconfig\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:03.360434 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.353119    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-host-var-lib-cni-bin\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:03.360434 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.353171    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-system-cni-dir\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:03.360434 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.353386    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/85602130-14de-443d-b237-b213c0ce8b82-etc-tuned\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:03.360434 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.353755    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2d5efe23-0e0d-4a4e-9d03-42829f371c09-ovn-node-metrics-cert\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:03.360434 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.353847    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ff0a920b-f15f-4909-906b-61dc41ef1263-cnibin\") pod \"multus-additional-cni-plugins-zvkh9\" (UID: \"ff0a920b-f15f-4909-906b-61dc41ef1263\") " pod="openshift-multus/multus-additional-cni-plugins-zvkh9"
Oct 30 12:51:03.360434 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.353908    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:03.360434 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.354376    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/85602130-14de-443d-b237-b213c0ce8b82-sys\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:03.360434 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.359528    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f8f7d922-7364-444f-a932-278628e26b83-registration-dir\") pod \"aws-ebs-csi-driver-node-p5mdl\" (UID: \"f8f7d922-7364-444f-a932-278628e26b83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5mdl"
Oct 30 12:51:03.360434 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.359553    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f8f7d922-7364-444f-a932-278628e26b83-device-dir\") pod \"aws-ebs-csi-driver-node-p5mdl\" (UID: \"f8f7d922-7364-444f-a932-278628e26b83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5mdl"
Oct 30 12:51:03.360434 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.359630    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f8f7d922-7364-444f-a932-278628e26b83-sys-fs\") pod \"aws-ebs-csi-driver-node-p5mdl\" (UID: \"f8f7d922-7364-444f-a932-278628e26b83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5mdl"
Oct 30 12:51:03.360434 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.360264    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/19115051-3b70-47cd-a7a3-313b1b3c26c7-iptables-alerter-script\") pod \"iptables-alerter-9p6w9\" (UID: \"19115051-3b70-47cd-a7a3-313b1b3c26c7\") " pod="openshift-network-operator/iptables-alerter-9p6w9"
Oct 30 12:51:03.360434 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.360342    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-host-run-multus-certs\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:03.360434 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.360395    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ff0a920b-f15f-4909-906b-61dc41ef1263-system-cni-dir\") pod \"multus-additional-cni-plugins-zvkh9\" (UID: \"ff0a920b-f15f-4909-906b-61dc41ef1263\") " pod="openshift-multus/multus-additional-cni-plugins-zvkh9"
Oct 30 12:51:03.361236 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.360444    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-run-ovn\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:03.361236 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.360546    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/85602130-14de-443d-b237-b213c0ce8b82-run\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:03.361236 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.360643    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/85602130-14de-443d-b237-b213c0ce8b82-lib-modules\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:03.361236 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.361169    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7d4b7525-1c1b-4e65-b263-349ab1a0f32f-konnectivity-ca\") pod \"konnectivity-agent-vlfkx\" (UID: \"7d4b7525-1c1b-4e65-b263-349ab1a0f32f\") " pod="kube-system/konnectivity-agent-vlfkx"
Oct 30 12:51:03.361429 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.361245    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-log-socket\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:03.361429 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.361348    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f8f7d922-7364-444f-a932-278628e26b83-socket-dir\") pod \"aws-ebs-csi-driver-node-p5mdl\" (UID: \"f8f7d922-7364-444f-a932-278628e26b83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5mdl"
Oct 30 12:51:03.361981 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.361954    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a1ca584f-61b1-4398-a0f6-5788e476aa20-cni-binary-copy\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:03.362069 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.362056    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f8f7d922-7364-444f-a932-278628e26b83-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p5mdl\" (UID: \"f8f7d922-7364-444f-a932-278628e26b83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5mdl"
Oct 30 12:51:03.362137 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.362126    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c83bcf7b-a1d6-457b-9d7c-bb03c2cbd449-hosts-file\") pod \"node-resolver-7mvnz\" (UID: \"c83bcf7b-a1d6-457b-9d7c-bb03c2cbd449\") " pod="openshift-dns/node-resolver-7mvnz"
Oct 30 12:51:03.362193 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.362182    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/31d09b91-c973-4cf9-a7d8-d602ff9994f6-host\") pod \"node-ca-7pxn6\" (UID: \"31d09b91-c973-4cf9-a7d8-d602ff9994f6\") " pod="openshift-image-registry/node-ca-7pxn6"
Oct 30 12:51:03.362302 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.362285    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/85602130-14de-443d-b237-b213c0ce8b82-etc-sysctl-conf\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:03.362368 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.362353    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-host-kubelet\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:03.362634 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.362608    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c83bcf7b-a1d6-457b-9d7c-bb03c2cbd449-tmp-dir\") pod \"node-resolver-7mvnz\" (UID: \"c83bcf7b-a1d6-457b-9d7c-bb03c2cbd449\") " pod="openshift-dns/node-resolver-7mvnz"
Oct 30 12:51:03.362921 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.362879    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2d5efe23-0e0d-4a4e-9d03-42829f371c09-env-overrides\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:03.363050 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.363031    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/815986e1-07b4-43fb-95f5-745316bd5105-dbus\") pod \"global-pull-secret-syncer-rxg7f\" (UID: \"815986e1-07b4-43fb-95f5-745316bd5105\") " pod="kube-system/global-pull-secret-syncer-rxg7f"
Oct 30 12:51:03.363134 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.363104    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-host-run-ovn-kubernetes\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:03.363206 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.363181    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-host-cni-bin\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:03.363286 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.363258    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-multus-socket-dir-parent\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:03.363917 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.363366    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-var-lib-openvswitch\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:03.363917 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.363453    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85602130-14de-443d-b237-b213c0ce8b82-etc-kubernetes\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:03.363917 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:03.363515    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 12:51:03.363917 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:03.363578    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/815986e1-07b4-43fb-95f5-745316bd5105-original-pull-secret podName:815986e1-07b4-43fb-95f5-745316bd5105 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:03.863560836 +0000 UTC m=+3.102939049 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/815986e1-07b4-43fb-95f5-745316bd5105-original-pull-secret") pod "global-pull-secret-syncer-rxg7f" (UID: "815986e1-07b4-43fb-95f5-745316bd5105") : object "kube-system"/"original-pull-secret" not registered
Oct 30 12:51:03.363917 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.342899    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2d5efe23-0e0d-4a4e-9d03-42829f371c09-ovnkube-config\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:03.363917 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.363694    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-multus-cni-dir\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:03.363917 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.363798    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/85602130-14de-443d-b237-b213c0ce8b82-etc-sysctl-d\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:03.363917 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.342927    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-host-cni-netd\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:03.363917 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.363830    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-cnibin\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:03.365096 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.364566    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ff0a920b-f15f-4909-906b-61dc41ef1263-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zvkh9\" (UID: \"ff0a920b-f15f-4909-906b-61dc41ef1263\") " pod="openshift-multus/multus-additional-cni-plugins-zvkh9"
Oct 30 12:51:03.365096 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.364688    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a1ca584f-61b1-4398-a0f6-5788e476aa20-multus-daemon-config\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:03.365096 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.364790    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f8f7d922-7364-444f-a932-278628e26b83-etc-selinux\") pod \"aws-ebs-csi-driver-node-p5mdl\" (UID: \"f8f7d922-7364-444f-a932-278628e26b83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5mdl"
Oct 30 12:51:03.365096 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.364896    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ff0a920b-f15f-4909-906b-61dc41ef1263-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zvkh9\" (UID: \"ff0a920b-f15f-4909-906b-61dc41ef1263\") " pod="openshift-multus/multus-additional-cni-plugins-zvkh9"
Oct 30 12:51:03.365096 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.365002    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ff0a920b-f15f-4909-906b-61dc41ef1263-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zvkh9\" (UID: \"ff0a920b-f15f-4909-906b-61dc41ef1263\") " pod="openshift-multus/multus-additional-cni-plugins-zvkh9"
Oct 30 12:51:03.367823 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.367800    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85602130-14de-443d-b237-b213c0ce8b82-tmp\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:03.370845 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.370546    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7d4b7525-1c1b-4e65-b263-349ab1a0f32f-agent-certs\") pod \"konnectivity-agent-vlfkx\" (UID: \"7d4b7525-1c1b-4e65-b263-349ab1a0f32f\") " pod="kube-system/konnectivity-agent-vlfkx"
Oct 30 12:51:03.373748 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.373723    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtppn\" (UniqueName: \"kubernetes.io/projected/f8f7d922-7364-444f-a932-278628e26b83-kube-api-access-wtppn\") pod \"aws-ebs-csi-driver-node-p5mdl\" (UID: \"f8f7d922-7364-444f-a932-278628e26b83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5mdl"
Oct 30 12:51:03.374352 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.374330    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxq79\" (UniqueName: \"kubernetes.io/projected/69eab030-f488-433c-869a-52d628824391-kube-api-access-hxq79\") pod \"network-metrics-daemon-79wgd\" (UID: \"69eab030-f488-433c-869a-52d628824391\") " pod="openshift-multus/network-metrics-daemon-79wgd"
Oct 30 12:51:03.383148 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.383123    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6s7g\" (UniqueName: \"kubernetes.io/projected/ff0a920b-f15f-4909-906b-61dc41ef1263-kube-api-access-q6s7g\") pod \"multus-additional-cni-plugins-zvkh9\" (UID: \"ff0a920b-f15f-4909-906b-61dc41ef1263\") " pod="openshift-multus/multus-additional-cni-plugins-zvkh9"
Oct 30 12:51:03.383646 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.383621    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5s5m\" (UniqueName: \"kubernetes.io/projected/2d5efe23-0e0d-4a4e-9d03-42829f371c09-kube-api-access-x5s5m\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:03.384017 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.383991    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pppc\" (UniqueName: \"kubernetes.io/projected/a1ca584f-61b1-4398-a0f6-5788e476aa20-kube-api-access-7pppc\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:03.385349 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.385325    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls6hw\" (UniqueName: \"kubernetes.io/projected/c83bcf7b-a1d6-457b-9d7c-bb03c2cbd449-kube-api-access-ls6hw\") pod \"node-resolver-7mvnz\" (UID: \"c83bcf7b-a1d6-457b-9d7c-bb03c2cbd449\") " pod="openshift-dns/node-resolver-7mvnz"
Oct 30 12:51:03.385986 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.385965    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdplx\" (UniqueName: \"kubernetes.io/projected/31d09b91-c973-4cf9-a7d8-d602ff9994f6-kube-api-access-zdplx\") pod \"node-ca-7pxn6\" (UID: \"31d09b91-c973-4cf9-a7d8-d602ff9994f6\") " pod="openshift-image-registry/node-ca-7pxn6"
Oct 30 12:51:03.387349 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.387324    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-749nf\" (UniqueName: \"kubernetes.io/projected/85602130-14de-443d-b237-b213c0ce8b82-kube-api-access-749nf\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:03.393929 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.393905    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78v2l\" (UniqueName: \"kubernetes.io/projected/19115051-3b70-47cd-a7a3-313b1b3c26c7-kube-api-access-78v2l\") pod \"iptables-alerter-9p6w9\" (UID: \"19115051-3b70-47cd-a7a3-313b1b3c26c7\") " pod="openshift-network-operator/iptables-alerter-9p6w9"
Oct 30 12:51:03.395175 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:03.394902    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 12:51:03.395175 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:03.394933    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 12:51:03.395175 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:03.394948    2569 projected.go:196] Error preparing data for projected volume kube-api-access-jfwjf for pod openshift-network-diagnostics/network-check-target-dlxch: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:51:03.395175 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:03.395017    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c1927296-d775-43c4-a874-8842ebfeb3c8-kube-api-access-jfwjf podName:c1927296-d775-43c4-a874-8842ebfeb3c8 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:03.895000538 +0000 UTC m=+3.134378745 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jfwjf" (UniqueName: "kubernetes.io/projected/c1927296-d775-43c4-a874-8842ebfeb3c8-kube-api-access-jfwjf") pod "network-check-target-dlxch" (UID: "c1927296-d775-43c4-a874-8842ebfeb3c8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:51:03.450852 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.450827    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9p6w9"
Oct 30 12:51:03.459920 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.459900    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7mvnz"
Oct 30 12:51:03.463345 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:03.463320    2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19115051_3b70_47cd_a7a3_313b1b3c26c7.slice/crio-85ca31ad3a144e45ff7a5c955c9f421b21596c0a6dbf03d3496fab6c5afeb18a WatchSource:0}: Error finding container 85ca31ad3a144e45ff7a5c955c9f421b21596c0a6dbf03d3496fab6c5afeb18a: Status 404 returned error can't find the container with id 85ca31ad3a144e45ff7a5c955c9f421b21596c0a6dbf03d3496fab6c5afeb18a
Oct 30 12:51:03.469408 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:03.469377    2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc83bcf7b_a1d6_457b_9d7c_bb03c2cbd449.slice/crio-5db4412bd71a6d83812f0301b973336ae78cdeadaac587d366503762741963b7 WatchSource:0}: Error finding container 5db4412bd71a6d83812f0301b973336ae78cdeadaac587d366503762741963b7: Status 404 returned error can't find the container with id 5db4412bd71a6d83812f0301b973336ae78cdeadaac587d366503762741963b7
Oct 30 12:51:03.472153 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.471997    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vlfkx"
Oct 30 12:51:03.480416 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.480387    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7pxn6"
Oct 30 12:51:03.482590 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:03.482568    2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d4b7525_1c1b_4e65_b263_349ab1a0f32f.slice/crio-841c7315696222e35d4b041031582f10b1b24393ed6535120d67e8c0f88c6f4a WatchSource:0}: Error finding container 841c7315696222e35d4b041031582f10b1b24393ed6535120d67e8c0f88c6f4a: Status 404 returned error can't find the container with id 841c7315696222e35d4b041031582f10b1b24393ed6535120d67e8c0f88c6f4a
Oct 30 12:51:03.487775 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.487756    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:03.495001 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:03.494972    2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31d09b91_c973_4cf9_a7d8_d602ff9994f6.slice/crio-209dbc8895cd9800cf2706ae099c9d409f916b6439290d9964aab249e7d3dd51 WatchSource:0}: Error finding container 209dbc8895cd9800cf2706ae099c9d409f916b6439290d9964aab249e7d3dd51: Status 404 returned error can't find the container with id 209dbc8895cd9800cf2706ae099c9d409f916b6439290d9964aab249e7d3dd51
Oct 30 12:51:03.495736 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.495719    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:03.498271 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:03.498244    2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85602130_14de_443d_b237_b213c0ce8b82.slice/crio-d2455509aa4088564bd1e727852dfde1c72c68aa9a4a4b2a319b39d1e3f094bc WatchSource:0}: Error finding container d2455509aa4088564bd1e727852dfde1c72c68aa9a4a4b2a319b39d1e3f094bc: Status 404 returned error can't find the container with id d2455509aa4088564bd1e727852dfde1c72c68aa9a4a4b2a319b39d1e3f094bc
Oct 30 12:51:03.506468 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:03.506445    2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1ca584f_61b1_4398_a0f6_5788e476aa20.slice/crio-c883f63b50ff78fea1cbad9c51a48d375c9dab138bf49c1e08b0a71211be2a8e WatchSource:0}: Error finding container c883f63b50ff78fea1cbad9c51a48d375c9dab138bf49c1e08b0a71211be2a8e: Status 404 returned error can't find the container with id c883f63b50ff78fea1cbad9c51a48d375c9dab138bf49c1e08b0a71211be2a8e
Oct 30 12:51:03.521968 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.521947    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zvkh9"
Oct 30 12:51:03.532495 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:03.532468    2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff0a920b_f15f_4909_906b_61dc41ef1263.slice/crio-75919339ef0b9c35042d1e42d475595815f0bec1ff4bfee178334d65fe6c9e99 WatchSource:0}: Error finding container 75919339ef0b9c35042d1e42d475595815f0bec1ff4bfee178334d65fe6c9e99: Status 404 returned error can't find the container with id 75919339ef0b9c35042d1e42d475595815f0bec1ff4bfee178334d65fe6c9e99
Oct 30 12:51:03.533122 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.533102    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5mdl"
Oct 30 12:51:03.541731 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.541710    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:03.544295 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:03.544269    2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8f7d922_7364_444f_a932_278628e26b83.slice/crio-ba1b5a455d610b60c5d2e4e14e0168f6a3555dfacf0ec593869f1489f8c588d1 WatchSource:0}: Error finding container ba1b5a455d610b60c5d2e4e14e0168f6a3555dfacf0ec593869f1489f8c588d1: Status 404 returned error can't find the container with id ba1b5a455d610b60c5d2e4e14e0168f6a3555dfacf0ec593869f1489f8c588d1
Oct 30 12:51:03.554161 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:03.554137    2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d5efe23_0e0d_4a4e_9d03_42829f371c09.slice/crio-a103dec3bf04fc7374a948e56580a14754926df6a11fdc6fa78b06ab10452ed4 WatchSource:0}: Error finding container a103dec3bf04fc7374a948e56580a14754926df6a11fdc6fa78b06ab10452ed4: Status 404 returned error can't find the container with id a103dec3bf04fc7374a948e56580a14754926df6a11fdc6fa78b06ab10452ed4
Oct 30 12:51:03.855355 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.854783    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69eab030-f488-433c-869a-52d628824391-metrics-certs\") pod \"network-metrics-daemon-79wgd\" (UID: \"69eab030-f488-433c-869a-52d628824391\") " pod="openshift-multus/network-metrics-daemon-79wgd"
Oct 30 12:51:03.855355 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:03.854975    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:51:03.855355 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:03.855043    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69eab030-f488-433c-869a-52d628824391-metrics-certs podName:69eab030-f488-433c-869a-52d628824391 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:04.855024855 +0000 UTC m=+4.094403079 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/69eab030-f488-433c-869a-52d628824391-metrics-certs") pod "network-metrics-daemon-79wgd" (UID: "69eab030-f488-433c-869a-52d628824391") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:51:03.957696 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.955873    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfwjf\" (UniqueName: \"kubernetes.io/projected/c1927296-d775-43c4-a874-8842ebfeb3c8-kube-api-access-jfwjf\") pod \"network-check-target-dlxch\" (UID: \"c1927296-d775-43c4-a874-8842ebfeb3c8\") " pod="openshift-network-diagnostics/network-check-target-dlxch"
Oct 30 12:51:03.957696 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:03.956151    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/815986e1-07b4-43fb-95f5-745316bd5105-original-pull-secret\") pod \"global-pull-secret-syncer-rxg7f\" (UID: \"815986e1-07b4-43fb-95f5-745316bd5105\") " pod="kube-system/global-pull-secret-syncer-rxg7f"
Oct 30 12:51:03.957696 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:03.956324    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 12:51:03.957696 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:03.956400    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/815986e1-07b4-43fb-95f5-745316bd5105-original-pull-secret podName:815986e1-07b4-43fb-95f5-745316bd5105 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:04.956380462 +0000 UTC m=+4.195758681 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/815986e1-07b4-43fb-95f5-745316bd5105-original-pull-secret") pod "global-pull-secret-syncer-rxg7f" (UID: "815986e1-07b4-43fb-95f5-745316bd5105") : object "kube-system"/"original-pull-secret" not registered
Oct 30 12:51:03.957696 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:03.956822    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 12:51:03.957696 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:03.956856    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 12:51:03.957696 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:03.956869    2569 projected.go:196] Error preparing data for projected volume kube-api-access-jfwjf for pod openshift-network-diagnostics/network-check-target-dlxch: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:51:03.957696 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:03.956916    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c1927296-d775-43c4-a874-8842ebfeb3c8-kube-api-access-jfwjf podName:c1927296-d775-43c4-a874-8842ebfeb3c8 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:04.956902074 +0000 UTC m=+4.196280282 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-jfwjf" (UniqueName: "kubernetes.io/projected/c1927296-d775-43c4-a874-8842ebfeb3c8-kube-api-access-jfwjf") pod "network-check-target-dlxch" (UID: "c1927296-d775-43c4-a874-8842ebfeb3c8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:51:04.194049 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:04.193839    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 12:46:03 +0000 UTC" deadline="2028-08-10 14:06:00.162554342 +0000 UTC"
Oct 30 12:51:04.194049 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:04.193909    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15601h14m55.968650105s"
Oct 30 12:51:04.259789 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:04.259760    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rxg7f"
Oct 30 12:51:04.260002 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:04.259959    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rxg7f" podUID="815986e1-07b4-43fb-95f5-745316bd5105"
Oct 30 12:51:04.285208 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:04.284488    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-6.ec2.internal" event={"ID":"c8cfc7a7881c11ddfae618f888b1edd8","Type":"ContainerStarted","Data":"b5ddeb7249724e94c8420e2cd96674ca97900fa8dfcc1c7608aafe9fbf0dcfde"}
Oct 30 12:51:04.288282 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:04.288227    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj" event={"ID":"2d5efe23-0e0d-4a4e-9d03-42829f371c09","Type":"ContainerStarted","Data":"a103dec3bf04fc7374a948e56580a14754926df6a11fdc6fa78b06ab10452ed4"}
Oct 30 12:51:04.292306 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:04.292210    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5mdl" event={"ID":"f8f7d922-7364-444f-a932-278628e26b83","Type":"ContainerStarted","Data":"ba1b5a455d610b60c5d2e4e14e0168f6a3555dfacf0ec593869f1489f8c588d1"}
Oct 30 12:51:04.299104 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:04.299076    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7r6q9" event={"ID":"85602130-14de-443d-b237-b213c0ce8b82","Type":"ContainerStarted","Data":"d2455509aa4088564bd1e727852dfde1c72c68aa9a4a4b2a319b39d1e3f094bc"}
Oct 30 12:51:04.311872 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:04.311808    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7pxn6" event={"ID":"31d09b91-c973-4cf9-a7d8-d602ff9994f6","Type":"ContainerStarted","Data":"209dbc8895cd9800cf2706ae099c9d409f916b6439290d9964aab249e7d3dd51"}
Oct 30 12:51:04.315101 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:04.315037    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7mvnz" event={"ID":"c83bcf7b-a1d6-457b-9d7c-bb03c2cbd449","Type":"ContainerStarted","Data":"5db4412bd71a6d83812f0301b973336ae78cdeadaac587d366503762741963b7"}
Oct 30 12:51:04.329208 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:04.328625    2569 generic.go:356] "Generic (PLEG): container finished" podID="a60e17e4a3a53aed12f9737e4f90e05b" containerID="7dfc08fd708e7f6ca6f640380cce379aee75b71617a39e48cd64aa11a308153d" exitCode=0
Oct 30 12:51:04.329208 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:04.328725    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-6.ec2.internal" event={"ID":"a60e17e4a3a53aed12f9737e4f90e05b","Type":"ContainerDied","Data":"7dfc08fd708e7f6ca6f640380cce379aee75b71617a39e48cd64aa11a308153d"}
Oct 30 12:51:04.332508 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:04.332381    2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-6.ec2.internal" podStartSLOduration=1.3323653069999999 podStartE2EDuration="1.332365307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:51:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:04.331426178 +0000 UTC m=+3.570804427" watchObservedRunningTime="2026-10-30 12:51:04.332365307 +0000 UTC m=+3.571743525"
Oct 30 12:51:04.336645 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:04.336560    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zvkh9" event={"ID":"ff0a920b-f15f-4909-906b-61dc41ef1263","Type":"ContainerStarted","Data":"75919339ef0b9c35042d1e42d475595815f0bec1ff4bfee178334d65fe6c9e99"}
Oct 30 12:51:04.347905 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:04.347874    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mv5cv" event={"ID":"a1ca584f-61b1-4398-a0f6-5788e476aa20","Type":"ContainerStarted","Data":"c883f63b50ff78fea1cbad9c51a48d375c9dab138bf49c1e08b0a71211be2a8e"}
Oct 30 12:51:04.354308 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:04.354281    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vlfkx" event={"ID":"7d4b7525-1c1b-4e65-b263-349ab1a0f32f","Type":"ContainerStarted","Data":"841c7315696222e35d4b041031582f10b1b24393ed6535120d67e8c0f88c6f4a"}
Oct 30 12:51:04.361235 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:04.361206    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9p6w9" event={"ID":"19115051-3b70-47cd-a7a3-313b1b3c26c7","Type":"ContainerStarted","Data":"85ca31ad3a144e45ff7a5c955c9f421b21596c0a6dbf03d3496fab6c5afeb18a"}
Oct 30 12:51:04.864791 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:04.864645    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69eab030-f488-433c-869a-52d628824391-metrics-certs\") pod \"network-metrics-daemon-79wgd\" (UID: \"69eab030-f488-433c-869a-52d628824391\") " pod="openshift-multus/network-metrics-daemon-79wgd"
Oct 30 12:51:04.864961 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:04.864863    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:51:04.864961 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:04.864922    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69eab030-f488-433c-869a-52d628824391-metrics-certs podName:69eab030-f488-433c-869a-52d628824391 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:06.864904678 +0000 UTC m=+6.104282874 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/69eab030-f488-433c-869a-52d628824391-metrics-certs") pod "network-metrics-daemon-79wgd" (UID: "69eab030-f488-433c-869a-52d628824391") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:51:04.966332 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:04.965762    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfwjf\" (UniqueName: \"kubernetes.io/projected/c1927296-d775-43c4-a874-8842ebfeb3c8-kube-api-access-jfwjf\") pod \"network-check-target-dlxch\" (UID: \"c1927296-d775-43c4-a874-8842ebfeb3c8\") " pod="openshift-network-diagnostics/network-check-target-dlxch"
Oct 30 12:51:04.966332 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:04.965820    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/815986e1-07b4-43fb-95f5-745316bd5105-original-pull-secret\") pod \"global-pull-secret-syncer-rxg7f\" (UID: \"815986e1-07b4-43fb-95f5-745316bd5105\") " pod="kube-system/global-pull-secret-syncer-rxg7f"
Oct 30 12:51:04.966332 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:04.965934    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 12:51:04.966332 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:04.965946    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 12:51:04.966332 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:04.965970    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 12:51:04.966332 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:04.965984    2569 projected.go:196] Error preparing data for projected volume kube-api-access-jfwjf for pod openshift-network-diagnostics/network-check-target-dlxch: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:51:04.966332 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:04.965996    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/815986e1-07b4-43fb-95f5-745316bd5105-original-pull-secret podName:815986e1-07b4-43fb-95f5-745316bd5105 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:06.965977705 +0000 UTC m=+6.205355914 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/815986e1-07b4-43fb-95f5-745316bd5105-original-pull-secret") pod "global-pull-secret-syncer-rxg7f" (UID: "815986e1-07b4-43fb-95f5-745316bd5105") : object "kube-system"/"original-pull-secret" not registered
Oct 30 12:51:04.966332 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:04.966039    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c1927296-d775-43c4-a874-8842ebfeb3c8-kube-api-access-jfwjf podName:c1927296-d775-43c4-a874-8842ebfeb3c8 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:06.966022572 +0000 UTC m=+6.205400782 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-jfwjf" (UniqueName: "kubernetes.io/projected/c1927296-d775-43c4-a874-8842ebfeb3c8-kube-api-access-jfwjf") pod "network-check-target-dlxch" (UID: "c1927296-d775-43c4-a874-8842ebfeb3c8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:51:05.194944 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:05.194853    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 12:46:03 +0000 UTC" deadline="2028-06-29 08:50:25.993182911 +0000 UTC"
Oct 30 12:51:05.194944 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:05.194896    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14587h59m20.798292256s"
Oct 30 12:51:05.261568 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:05.259872    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dlxch"
Oct 30 12:51:05.261568 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:05.260015    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dlxch" podUID="c1927296-d775-43c4-a874-8842ebfeb3c8"
Oct 30 12:51:05.261568 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:05.260173    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-79wgd"
Oct 30 12:51:05.261568 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:05.260271    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-79wgd" podUID="69eab030-f488-433c-869a-52d628824391"
Oct 30 12:51:05.393784 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:05.393667    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-6.ec2.internal" event={"ID":"a60e17e4a3a53aed12f9737e4f90e05b","Type":"ContainerStarted","Data":"e0a83b472202e398c74d83049d01d00b900d7981a76198dccf8dd5fdaa7b5caa"}
Oct 30 12:51:05.441970 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:05.441909    2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-6.ec2.internal" podStartSLOduration=2.441886968 podStartE2EDuration="2.441886968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:51:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:05.440935861 +0000 UTC m=+4.680314078" watchObservedRunningTime="2026-10-30 12:51:05.441886968 +0000 UTC m=+4.681265186"
Oct 30 12:51:06.260537 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:06.259319    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rxg7f"
Oct 30 12:51:06.260537 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:06.259456    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rxg7f" podUID="815986e1-07b4-43fb-95f5-745316bd5105"
Oct 30 12:51:06.408082 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:06.407970    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zvkh9" event={"ID":"ff0a920b-f15f-4909-906b-61dc41ef1263","Type":"ContainerStarted","Data":"bd4b50582c4018c35ed971ec56705d65ece4c78b00abc6b590db3ec24bdd4eaf"}
Oct 30 12:51:06.885319 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:06.884555    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69eab030-f488-433c-869a-52d628824391-metrics-certs\") pod \"network-metrics-daemon-79wgd\" (UID: \"69eab030-f488-433c-869a-52d628824391\") " pod="openshift-multus/network-metrics-daemon-79wgd"
Oct 30 12:51:06.885319 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:06.884818    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:51:06.885319 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:06.884891    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69eab030-f488-433c-869a-52d628824391-metrics-certs podName:69eab030-f488-433c-869a-52d628824391 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:10.884869114 +0000 UTC m=+10.124247325 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/69eab030-f488-433c-869a-52d628824391-metrics-certs") pod "network-metrics-daemon-79wgd" (UID: "69eab030-f488-433c-869a-52d628824391") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:51:06.985581 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:06.985546    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfwjf\" (UniqueName: \"kubernetes.io/projected/c1927296-d775-43c4-a874-8842ebfeb3c8-kube-api-access-jfwjf\") pod \"network-check-target-dlxch\" (UID: \"c1927296-d775-43c4-a874-8842ebfeb3c8\") " pod="openshift-network-diagnostics/network-check-target-dlxch"
Oct 30 12:51:06.985758 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:06.985614    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/815986e1-07b4-43fb-95f5-745316bd5105-original-pull-secret\") pod \"global-pull-secret-syncer-rxg7f\" (UID: \"815986e1-07b4-43fb-95f5-745316bd5105\") " pod="kube-system/global-pull-secret-syncer-rxg7f"
Oct 30 12:51:06.985840 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:06.985797    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 12:51:06.985901 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:06.985871    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/815986e1-07b4-43fb-95f5-745316bd5105-original-pull-secret podName:815986e1-07b4-43fb-95f5-745316bd5105 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:10.985851274 +0000 UTC m=+10.225229493 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/815986e1-07b4-43fb-95f5-745316bd5105-original-pull-secret") pod "global-pull-secret-syncer-rxg7f" (UID: "815986e1-07b4-43fb-95f5-745316bd5105") : object "kube-system"/"original-pull-secret" not registered
Oct 30 12:51:06.986338 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:06.986292    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 12:51:06.986338 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:06.986314    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 12:51:06.986338 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:06.986328    2569 projected.go:196] Error preparing data for projected volume kube-api-access-jfwjf for pod openshift-network-diagnostics/network-check-target-dlxch: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:51:06.986555 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:06.986378    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c1927296-d775-43c4-a874-8842ebfeb3c8-kube-api-access-jfwjf podName:c1927296-d775-43c4-a874-8842ebfeb3c8 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:10.986362188 +0000 UTC m=+10.225740397 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-jfwjf" (UniqueName: "kubernetes.io/projected/c1927296-d775-43c4-a874-8842ebfeb3c8-kube-api-access-jfwjf") pod "network-check-target-dlxch" (UID: "c1927296-d775-43c4-a874-8842ebfeb3c8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:51:07.259873 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:07.259841    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dlxch"
Oct 30 12:51:07.259996 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:07.259968    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dlxch" podUID="c1927296-d775-43c4-a874-8842ebfeb3c8"
Oct 30 12:51:07.260524 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:07.260502    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-79wgd"
Oct 30 12:51:07.260669 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:07.260633    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-79wgd" podUID="69eab030-f488-433c-869a-52d628824391"
Oct 30 12:51:07.438086 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:07.437995    2569 generic.go:356] "Generic (PLEG): container finished" podID="ff0a920b-f15f-4909-906b-61dc41ef1263" containerID="bd4b50582c4018c35ed971ec56705d65ece4c78b00abc6b590db3ec24bdd4eaf" exitCode=0
Oct 30 12:51:07.438285 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:07.438089    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zvkh9" event={"ID":"ff0a920b-f15f-4909-906b-61dc41ef1263","Type":"ContainerDied","Data":"bd4b50582c4018c35ed971ec56705d65ece4c78b00abc6b590db3ec24bdd4eaf"}
Oct 30 12:51:07.450748 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:07.450717    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vlfkx" event={"ID":"7d4b7525-1c1b-4e65-b263-349ab1a0f32f","Type":"ContainerStarted","Data":"a282d884d0d67b9df523b6ef97ea332eb9f0cc3ce52b0113b1577e1f07b384d9"}
Oct 30 12:51:08.260156 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:08.260122    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rxg7f"
Oct 30 12:51:08.260347 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:08.260317    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rxg7f" podUID="815986e1-07b4-43fb-95f5-745316bd5105"
Oct 30 12:51:08.645117 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:08.644913    2569 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vlfkx"
Oct 30 12:51:08.645529 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:08.645465    2569 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vlfkx"
Oct 30 12:51:08.705640 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:08.705584    2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vlfkx" podStartSLOduration=3.051844107 podStartE2EDuration="6.705564743s" totalImagesPullingTime="3.653720636s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:51:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:07.540654677 +0000 UTC m=+6.780032896" watchObservedRunningTime="2026-10-30 12:51:08.705564743 +0000 UTC m=+7.944942962"
Oct 30 12:51:09.261691 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:09.259815    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-79wgd"
Oct 30 12:51:09.261691 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:09.259946    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-79wgd" podUID="69eab030-f488-433c-869a-52d628824391"
Oct 30 12:51:09.261691 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:09.260206    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dlxch"
Oct 30 12:51:09.261691 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:09.260287    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dlxch" podUID="c1927296-d775-43c4-a874-8842ebfeb3c8"
Oct 30 12:51:09.458711 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:09.458582    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5mdl" event={"ID":"f8f7d922-7364-444f-a932-278628e26b83","Type":"ContainerStarted","Data":"635aec90ef7e706ef73af8248a0b30ef11f34d4cd5cc1b71793877e273393e6f"}
Oct 30 12:51:09.461752 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:09.461726    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7pxn6" event={"ID":"31d09b91-c973-4cf9-a7d8-d602ff9994f6","Type":"ContainerStarted","Data":"ad401eb91b96e7f76aefaacf448cef7571284f661d410ec0e902b8a8604eabbf"}
Oct 30 12:51:10.260514 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:10.260429    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rxg7f"
Oct 30 12:51:10.260989 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:10.260580    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rxg7f" podUID="815986e1-07b4-43fb-95f5-745316bd5105"
Oct 30 12:51:10.372923 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:10.372889    2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vlfkx"
Oct 30 12:51:10.373899 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:10.373875    2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vlfkx"
Oct 30 12:51:10.427384 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:10.427147    2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7pxn6" podStartSLOduration=3.3493558119999998 podStartE2EDuration="8.427126378s" totalImagesPullingTime="5.077770566s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:51:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:09.513868473 +0000 UTC m=+8.753246695" watchObservedRunningTime="2026-10-30 12:51:10.427126378 +0000 UTC m=+9.666504598"
Oct 30 12:51:10.918862 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:10.918542    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69eab030-f488-433c-869a-52d628824391-metrics-certs\") pod \"network-metrics-daemon-79wgd\" (UID: \"69eab030-f488-433c-869a-52d628824391\") " pod="openshift-multus/network-metrics-daemon-79wgd"
Oct 30 12:51:10.918862 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:10.918738    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:51:10.918862 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:10.918803    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69eab030-f488-433c-869a-52d628824391-metrics-certs podName:69eab030-f488-433c-869a-52d628824391 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:18.918783078 +0000 UTC m=+18.158161288 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/69eab030-f488-433c-869a-52d628824391-metrics-certs") pod "network-metrics-daemon-79wgd" (UID: "69eab030-f488-433c-869a-52d628824391") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:51:10.983546 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:10.983520    2569 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 12:51:11.019110 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:11.019075    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/815986e1-07b4-43fb-95f5-745316bd5105-original-pull-secret\") pod \"global-pull-secret-syncer-rxg7f\" (UID: \"815986e1-07b4-43fb-95f5-745316bd5105\") " pod="kube-system/global-pull-secret-syncer-rxg7f"
Oct 30 12:51:11.019232 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:11.019171    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfwjf\" (UniqueName: \"kubernetes.io/projected/c1927296-d775-43c4-a874-8842ebfeb3c8-kube-api-access-jfwjf\") pod \"network-check-target-dlxch\" (UID: \"c1927296-d775-43c4-a874-8842ebfeb3c8\") " pod="openshift-network-diagnostics/network-check-target-dlxch"
Oct 30 12:51:11.019344 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:11.019327    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 12:51:11.019403 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:11.019357    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 12:51:11.019403 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:11.019372    2569 projected.go:196] Error preparing data for projected volume kube-api-access-jfwjf for pod openshift-network-diagnostics/network-check-target-dlxch: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:51:11.019509 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:11.019428    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c1927296-d775-43c4-a874-8842ebfeb3c8-kube-api-access-jfwjf podName:c1927296-d775-43c4-a874-8842ebfeb3c8 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:19.019408728 +0000 UTC m=+18.258786943 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-jfwjf" (UniqueName: "kubernetes.io/projected/c1927296-d775-43c4-a874-8842ebfeb3c8-kube-api-access-jfwjf") pod "network-check-target-dlxch" (UID: "c1927296-d775-43c4-a874-8842ebfeb3c8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:51:11.019570 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:11.019513    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 12:51:11.019570 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:11.019551    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/815986e1-07b4-43fb-95f5-745316bd5105-original-pull-secret podName:815986e1-07b4-43fb-95f5-745316bd5105 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:19.019538704 +0000 UTC m=+18.258916922 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/815986e1-07b4-43fb-95f5-745316bd5105-original-pull-secret") pod "global-pull-secret-syncer-rxg7f" (UID: "815986e1-07b4-43fb-95f5-745316bd5105") : object "kube-system"/"original-pull-secret" not registered
Oct 30 12:51:11.201859 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:11.201743    2569 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T12:51:10.983540488Z","UUID":"72dbaca0-42ab-4feb-9576-7cbb16399ba2","Handler":null,"Name":"","Endpoint":""}
Oct 30 12:51:11.203641 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:11.203615    2569 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 12:51:11.203786 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:11.203651    2569 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 12:51:11.265311 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:11.264669    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-79wgd"
Oct 30 12:51:11.265311 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:11.264812    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-79wgd" podUID="69eab030-f488-433c-869a-52d628824391"
Oct 30 12:51:11.265311 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:11.265171    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dlxch"
Oct 30 12:51:11.265311 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:11.265269    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dlxch" podUID="c1927296-d775-43c4-a874-8842ebfeb3c8"
Oct 30 12:51:11.469244 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:11.469142    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mv5cv" event={"ID":"a1ca584f-61b1-4398-a0f6-5788e476aa20","Type":"ContainerStarted","Data":"f3f5784e7c9a536bff847df061ed6712628f2226edf5d99a1a7c0c797009aa69"}
Oct 30 12:51:11.473015 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:11.472981    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5mdl" event={"ID":"f8f7d922-7364-444f-a932-278628e26b83","Type":"ContainerStarted","Data":"d5491231f50d51ae5050592ee456e250d4c7321517a47939212d83fef38fc4a7"}
Oct 30 12:51:11.524038 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:11.523971    2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-mv5cv" podStartSLOduration=1.818901416 podStartE2EDuration="9.523950806s" totalImagesPullingTime="7.70504939s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:51:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:11.522935491 +0000 UTC m=+10.762313707" watchObservedRunningTime="2026-10-30 12:51:11.523950806 +0000 UTC m=+10.763329026"
Oct 30 12:51:12.259844 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:12.259783    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rxg7f"
Oct 30 12:51:12.259962 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:12.259914    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rxg7f" podUID="815986e1-07b4-43fb-95f5-745316bd5105"
Oct 30 12:51:12.476454 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:12.476416    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7mvnz" event={"ID":"c83bcf7b-a1d6-457b-9d7c-bb03c2cbd449","Type":"ContainerStarted","Data":"fdc0c5f48f7ef78d5f3cdfc6fc215c08df0fc89640cfeb1434cbe82d4748a6e9"}
Oct 30 12:51:12.479127 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:12.479101    2569 generic.go:356] "Generic (PLEG): container finished" podID="ff0a920b-f15f-4909-906b-61dc41ef1263" containerID="653c1ffb461d29d061b2ee16375f59f04326a0766b344ff97555588878542557" exitCode=0
Oct 30 12:51:12.479264 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:12.479204    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zvkh9" event={"ID":"ff0a920b-f15f-4909-906b-61dc41ef1263","Type":"ContainerDied","Data":"653c1ffb461d29d061b2ee16375f59f04326a0766b344ff97555588878542557"}
Oct 30 12:51:12.482052 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:12.482025    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5mdl" event={"ID":"f8f7d922-7364-444f-a932-278628e26b83","Type":"ContainerStarted","Data":"be7a31babdd7ce8c6c1bb277d29ba152b70e4bcc26badc73c6584b25e2f54ff7"}
Oct 30 12:51:12.587268 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:12.586576    2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5mdl" podStartSLOduration=2.428027682 podStartE2EDuration="10.586558761s" totalImagesPullingTime="8.158531079s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:51:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:12.586233335 +0000 UTC m=+11.825611550" watchObservedRunningTime="2026-10-30 12:51:12.586558761 +0000 UTC m=+11.825936977"
Oct 30 12:51:12.587268 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:12.586732    2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-7mvnz" podStartSLOduration=2.335285616 podStartE2EDuration="10.586720719s" totalImagesPullingTime="8.251435103s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:51:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:12.52555524 +0000 UTC m=+11.764933456" watchObservedRunningTime="2026-10-30 12:51:12.586720719 +0000 UTC m=+11.826098938"
Oct 30 12:51:13.259656 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:13.259626    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-79wgd"
Oct 30 12:51:13.259804 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:13.259552    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dlxch"
Oct 30 12:51:13.259804 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:13.259777    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-79wgd" podUID="69eab030-f488-433c-869a-52d628824391"
Oct 30 12:51:13.259914 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:13.259822    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dlxch" podUID="c1927296-d775-43c4-a874-8842ebfeb3c8"
Oct 30 12:51:13.486579 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:13.486545    2569 generic.go:356] "Generic (PLEG): container finished" podID="ff0a920b-f15f-4909-906b-61dc41ef1263" containerID="e32a756ce46dd838ddf59ae449ec16cdd4f3115e33e6dd4fb233c799f38767c2" exitCode=0
Oct 30 12:51:13.487088 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:13.486636    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zvkh9" event={"ID":"ff0a920b-f15f-4909-906b-61dc41ef1263","Type":"ContainerDied","Data":"e32a756ce46dd838ddf59ae449ec16cdd4f3115e33e6dd4fb233c799f38767c2"}
Oct 30 12:51:13.489637 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:13.489567    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9p6w9" event={"ID":"19115051-3b70-47cd-a7a3-313b1b3c26c7","Type":"ContainerStarted","Data":"70011378612cfd340e705ec517941e120f9c63417fec6e17adf450967d77a9a4"}
Oct 30 12:51:13.585894 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:13.585845    2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9p6w9" podStartSLOduration=3.425190797 podStartE2EDuration="11.585831563s" totalImagesPullingTime="8.160640766s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:51:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:13.585741784 +0000 UTC m=+12.825119999" watchObservedRunningTime="2026-10-30 12:51:13.585831563 +0000 UTC m=+12.825209778"
Oct 30 12:51:14.259638 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:14.259613    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rxg7f"
Oct 30 12:51:14.259745 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:14.259727    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rxg7f" podUID="815986e1-07b4-43fb-95f5-745316bd5105"
Oct 30 12:51:14.494130 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:14.494068    2569 generic.go:356] "Generic (PLEG): container finished" podID="ff0a920b-f15f-4909-906b-61dc41ef1263" containerID="36001a7ab4546a8df2990deede34a0e2a35301e8b6b4074c12789d317f458a8e" exitCode=0
Oct 30 12:51:14.494682 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:14.494153    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zvkh9" event={"ID":"ff0a920b-f15f-4909-906b-61dc41ef1263","Type":"ContainerDied","Data":"36001a7ab4546a8df2990deede34a0e2a35301e8b6b4074c12789d317f458a8e"}
Oct 30 12:51:14.495656 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:14.495636    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7r6q9" event={"ID":"85602130-14de-443d-b237-b213c0ce8b82","Type":"ContainerStarted","Data":"dc152ae760c7249a6f3b5d91d20609bcf85ecef4c7a62441bc359b83b7d180ab"}
Oct 30 12:51:14.606023 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:14.605976    2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-7r6q9" podStartSLOduration=2.240335246 podStartE2EDuration="12.605964636s" totalImagesPullingTime="10.36562939s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:51:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:14.605871183 +0000 UTC m=+13.845249397" watchObservedRunningTime="2026-10-30 12:51:14.605964636 +0000 UTC m=+13.845342877"
Oct 30 12:51:15.259981 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:15.259947    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dlxch"
Oct 30 12:51:15.260140 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:15.260075    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dlxch" podUID="c1927296-d775-43c4-a874-8842ebfeb3c8"
Oct 30 12:51:15.260484 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:15.260466    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-79wgd"
Oct 30 12:51:15.260625 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:15.260567    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-79wgd" podUID="69eab030-f488-433c-869a-52d628824391"
Oct 30 12:51:16.259468 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:16.259438    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rxg7f"
Oct 30 12:51:16.259963 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:16.259556    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rxg7f" podUID="815986e1-07b4-43fb-95f5-745316bd5105"
Oct 30 12:51:16.503219 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:16.503186    2569 generic.go:356] "Generic (PLEG): container finished" podID="ff0a920b-f15f-4909-906b-61dc41ef1263" containerID="7479cb8d1471ab3843adde80a4ac73829345529998d131dac8adfb13e35c3727" exitCode=0
Oct 30 12:51:16.503345 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:16.503250    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zvkh9" event={"ID":"ff0a920b-f15f-4909-906b-61dc41ef1263","Type":"ContainerDied","Data":"7479cb8d1471ab3843adde80a4ac73829345529998d131dac8adfb13e35c3727"}
Oct 30 12:51:17.263275 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:17.263245    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-79wgd"
Oct 30 12:51:17.263745 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:17.263257    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dlxch"
Oct 30 12:51:17.263745 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:17.263379    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-79wgd" podUID="69eab030-f488-433c-869a-52d628824391"
Oct 30 12:51:17.263745 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:17.263437    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dlxch" podUID="c1927296-d775-43c4-a874-8842ebfeb3c8"
Oct 30 12:51:17.508989 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:17.508954    2569 generic.go:356] "Generic (PLEG): container finished" podID="ff0a920b-f15f-4909-906b-61dc41ef1263" containerID="6c53c33c80386939f77a0ac68b599495904d00ddc0278dcbcab907762add20ec" exitCode=0
Oct 30 12:51:17.509157 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:17.509024    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zvkh9" event={"ID":"ff0a920b-f15f-4909-906b-61dc41ef1263","Type":"ContainerDied","Data":"6c53c33c80386939f77a0ac68b599495904d00ddc0278dcbcab907762add20ec"}
Oct 30 12:51:18.260104 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:18.260080    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rxg7f"
Oct 30 12:51:18.260259 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:18.260167    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rxg7f" podUID="815986e1-07b4-43fb-95f5-745316bd5105"
Oct 30 12:51:18.513532 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:18.513466    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zvkh9" event={"ID":"ff0a920b-f15f-4909-906b-61dc41ef1263","Type":"ContainerStarted","Data":"6e0d3643b74d87a17cba45e9b7b8c4f437c49f29c290936c8444821a2ada177b"}
Oct 30 12:51:18.593890 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:18.593852    2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-zvkh9" podStartSLOduration=6.45260164 podStartE2EDuration="16.593839875s" totalImagesPullingTime="10.141238235s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:51:02 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:18.579134544 +0000 UTC m=+17.818512775" watchObservedRunningTime="2026-10-30 12:51:18.593839875 +0000 UTC m=+17.833218088"
Oct 30 12:51:18.979356 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:18.979195    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69eab030-f488-433c-869a-52d628824391-metrics-certs\") pod \"network-metrics-daemon-79wgd\" (UID: \"69eab030-f488-433c-869a-52d628824391\") " pod="openshift-multus/network-metrics-daemon-79wgd"
Oct 30 12:51:18.979467 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:18.979330    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:51:18.979467 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:18.979418    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69eab030-f488-433c-869a-52d628824391-metrics-certs podName:69eab030-f488-433c-869a-52d628824391 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:34.979405563 +0000 UTC m=+34.218783767 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/69eab030-f488-433c-869a-52d628824391-metrics-certs") pod "network-metrics-daemon-79wgd" (UID: "69eab030-f488-433c-869a-52d628824391") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:51:19.080311 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:19.080255    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfwjf\" (UniqueName: \"kubernetes.io/projected/c1927296-d775-43c4-a874-8842ebfeb3c8-kube-api-access-jfwjf\") pod \"network-check-target-dlxch\" (UID: \"c1927296-d775-43c4-a874-8842ebfeb3c8\") " pod="openshift-network-diagnostics/network-check-target-dlxch"
Oct 30 12:51:19.080311 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:19.080288    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/815986e1-07b4-43fb-95f5-745316bd5105-original-pull-secret\") pod \"global-pull-secret-syncer-rxg7f\" (UID: \"815986e1-07b4-43fb-95f5-745316bd5105\") " pod="kube-system/global-pull-secret-syncer-rxg7f"
Oct 30 12:51:19.080432 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:19.080404    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 12:51:19.080432 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:19.080420    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 12:51:19.080432 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:19.080429    2569 projected.go:196] Error preparing data for projected volume kube-api-access-jfwjf for pod openshift-network-diagnostics/network-check-target-dlxch: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:51:19.080519 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:19.080448    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 12:51:19.080519 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:19.080468    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c1927296-d775-43c4-a874-8842ebfeb3c8-kube-api-access-jfwjf podName:c1927296-d775-43c4-a874-8842ebfeb3c8 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:35.080456685 +0000 UTC m=+34.319834876 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-jfwjf" (UniqueName: "kubernetes.io/projected/c1927296-d775-43c4-a874-8842ebfeb3c8-kube-api-access-jfwjf") pod "network-check-target-dlxch" (UID: "c1927296-d775-43c4-a874-8842ebfeb3c8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:51:19.080519 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:19.080495    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/815986e1-07b4-43fb-95f5-745316bd5105-original-pull-secret podName:815986e1-07b4-43fb-95f5-745316bd5105 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:35.080476994 +0000 UTC m=+34.319855212 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/815986e1-07b4-43fb-95f5-745316bd5105-original-pull-secret") pod "global-pull-secret-syncer-rxg7f" (UID: "815986e1-07b4-43fb-95f5-745316bd5105") : object "kube-system"/"original-pull-secret" not registered
Oct 30 12:51:19.259982 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:19.259959    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dlxch"
Oct 30 12:51:19.260074 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:19.260044    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dlxch" podUID="c1927296-d775-43c4-a874-8842ebfeb3c8"
Oct 30 12:51:19.260144 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:19.260129    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-79wgd"
Oct 30 12:51:19.260218 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:19.260202    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-79wgd" podUID="69eab030-f488-433c-869a-52d628824391"
Oct 30 12:51:19.517329 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:19.517306    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj" event={"ID":"2d5efe23-0e0d-4a4e-9d03-42829f371c09","Type":"ContainerStarted","Data":"caca9f5050a6a6ad717f96f6cda727ba0b71771732f2088dc1f182a254df7d4e"}
Oct 30 12:51:19.517698 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:19.517378    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj" event={"ID":"2d5efe23-0e0d-4a4e-9d03-42829f371c09","Type":"ContainerStarted","Data":"f59cbe71d948dc278f5962745abafd4582359d0010d198a9ed8a5175dcb73356"}
Oct 30 12:51:19.517698 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:19.517427    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj" event={"ID":"2d5efe23-0e0d-4a4e-9d03-42829f371c09","Type":"ContainerStarted","Data":"8d70278c0655326f6202537ac935e4e95f49a553f562c50aed6d300c6906a8c1"}
Oct 30 12:51:19.517698 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:19.517474    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj" event={"ID":"2d5efe23-0e0d-4a4e-9d03-42829f371c09","Type":"ContainerStarted","Data":"55798a95f3fec44d7c5b818db7efe58a53b308c3ea8e1fa2fa97d63917d1aedf"}
Oct 30 12:51:19.517698 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:19.517519    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj" event={"ID":"2d5efe23-0e0d-4a4e-9d03-42829f371c09","Type":"ContainerStarted","Data":"d9766e98d66972294f8017d103b21ae65137c8d240365e849bdbd965c85bdf81"}
Oct 30 12:51:19.517698 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:19.517567    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj" event={"ID":"2d5efe23-0e0d-4a4e-9d03-42829f371c09","Type":"ContainerStarted","Data":"5ca5614f83455aea6714666cb33dc4e42d0339ab417e6ddd81b2f4ba632f292a"}
Oct 30 12:51:20.259611 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:20.259581    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rxg7f"
Oct 30 12:51:20.259705 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:20.259673    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rxg7f" podUID="815986e1-07b4-43fb-95f5-745316bd5105"
Oct 30 12:51:21.261096 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:21.261032    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-79wgd"
Oct 30 12:51:21.261670 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:21.261423    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dlxch"
Oct 30 12:51:21.261670 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:21.261460    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-79wgd" podUID="69eab030-f488-433c-869a-52d628824391"
Oct 30 12:51:21.261670 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:21.261615    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dlxch" podUID="c1927296-d775-43c4-a874-8842ebfeb3c8"
Oct 30 12:51:21.524340 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:21.524279    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj" event={"ID":"2d5efe23-0e0d-4a4e-9d03-42829f371c09","Type":"ContainerStarted","Data":"e4afef81b73b0b75706054c423412717c1b4053af2c0740b6f68625f6ae605a8"}
Oct 30 12:51:22.259583 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:22.259553    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rxg7f"
Oct 30 12:51:22.259764 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:22.259669    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rxg7f" podUID="815986e1-07b4-43fb-95f5-745316bd5105"
Oct 30 12:51:23.259805 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:23.259771    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-79wgd"
Oct 30 12:51:23.260217 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:23.259886    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-79wgd" podUID="69eab030-f488-433c-869a-52d628824391"
Oct 30 12:51:23.260217 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:23.259939    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dlxch"
Oct 30 12:51:23.260217 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:23.260026    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dlxch" podUID="c1927296-d775-43c4-a874-8842ebfeb3c8"
Oct 30 12:51:23.530936 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:23.530910    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj" event={"ID":"2d5efe23-0e0d-4a4e-9d03-42829f371c09","Type":"ContainerStarted","Data":"4e5afb132767b881c06027c4ab4958ab216fa73b25919d0a77de4ab7ab7869b8"}
Oct 30 12:51:23.531187 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:23.531171    2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:23.531268 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:23.531245    2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:23.546228 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:23.546209    2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:23.546425 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:23.546407    2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:23.633350 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:23.633309    2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj" podStartSLOduration=6.551802549 podStartE2EDuration="21.633298331s" totalImagesPullingTime="15.081495782s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:51:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:23.632824928 +0000 UTC m=+22.872203143" watchObservedRunningTime="2026-10-30 12:51:23.633298331 +0000 UTC m=+22.872676545"
Oct 30 12:51:24.259565 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:24.259526    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rxg7f"
Oct 30 12:51:24.259756 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:24.259660    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rxg7f" podUID="815986e1-07b4-43fb-95f5-745316bd5105"
Oct 30 12:51:24.532890 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:24.532819    2569 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 12:51:25.157519 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:25.157493    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dlxch"]
Oct 30 12:51:25.157754 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:25.157653    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dlxch"
Oct 30 12:51:25.157754 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:25.157733    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dlxch" podUID="c1927296-d775-43c4-a874-8842ebfeb3c8"
Oct 30 12:51:25.164564 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:25.164539    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rxg7f"]
Oct 30 12:51:25.164669 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:25.164652    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rxg7f"
Oct 30 12:51:25.164744 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:25.164726    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rxg7f" podUID="815986e1-07b4-43fb-95f5-745316bd5105"
Oct 30 12:51:25.171813 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:25.171793    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-79wgd"]
Oct 30 12:51:25.171903 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:25.171893    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-79wgd"
Oct 30 12:51:25.172007 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:25.171990    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-79wgd" podUID="69eab030-f488-433c-869a-52d628824391"
Oct 30 12:51:25.535129 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:25.535065    2569 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 12:51:25.712482 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:25.712440    2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:27.259958 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:27.259931    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-79wgd"
Oct 30 12:51:27.260371 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:27.259931    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rxg7f"
Oct 30 12:51:27.260371 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:27.260070    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-79wgd" podUID="69eab030-f488-433c-869a-52d628824391"
Oct 30 12:51:27.260371 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:27.259930    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dlxch"
Oct 30 12:51:27.260371 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:27.260164    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rxg7f" podUID="815986e1-07b4-43fb-95f5-745316bd5105"
Oct 30 12:51:27.260371 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:27.260207    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dlxch" podUID="c1927296-d775-43c4-a874-8842ebfeb3c8"
Oct 30 12:51:29.260165 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:29.260134    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-79wgd"
Oct 30 12:51:29.260629 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:29.260168    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dlxch"
Oct 30 12:51:29.260629 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:29.260235    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-79wgd" podUID="69eab030-f488-433c-869a-52d628824391"
Oct 30 12:51:29.260629 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:29.260329    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rxg7f"
Oct 30 12:51:29.260629 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:29.260424    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dlxch" podUID="c1927296-d775-43c4-a874-8842ebfeb3c8"
Oct 30 12:51:29.260629 ip-10-0-137-6 kubenswrapper[2569]: E1030 12:51:29.260512    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rxg7f" podUID="815986e1-07b4-43fb-95f5-745316bd5105"
Oct 30 12:51:31.005299 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:31.005252    2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-6.ec2.internal" event="NodeReady"
Oct 30 12:51:31.005622 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:31.005334    2569 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 12:51:31.260683 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:31.260497    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-79wgd"
Oct 30 12:51:31.260818 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:31.260570    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dlxch"
Oct 30 12:51:31.260887 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:31.260587    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rxg7f"
Oct 30 12:51:31.283548 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:31.283531    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 12:51:31.283751 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:31.283734    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 12:51:31.297732 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:31.297707    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 12:51:31.299302 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:31.299281    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-sb6s2\""
Oct 30 12:51:31.299391 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:31.299383    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-n54s8\""
Oct 30 12:51:31.299539 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:31.299528    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 12:51:31.299588 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:31.299555    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-rnjq7\""
Oct 30 12:51:31.793828 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:31.793801    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7mvnz_c83bcf7b-a1d6-457b-9d7c-bb03c2cbd449/dns-node-resolver/0.log"
Oct 30 12:51:32.889653 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:32.889625    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7pxn6_31d09b91-c973-4cf9-a7d8-d602ff9994f6/node-ca/0.log"
Oct 30 12:51:33.128105 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:33.128074    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vlfkx_7d4b7525-1c1b-4e65-b263-349ab1a0f32f/konnectivity-agent/0.log" limit=1024
Oct 30 12:51:33.556622 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:33.556589    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vlfkx_7d4b7525-1c1b-4e65-b263-349ab1a0f32f/konnectivity-agent/0.log" limit=1024
Oct 30 12:51:33.793415 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:33.793393    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vlfkx_7d4b7525-1c1b-4e65-b263-349ab1a0f32f/konnectivity-agent/0.log" limit=1024
Oct 30 12:51:34.094532 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:34.094505    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vlfkx_7d4b7525-1c1b-4e65-b263-349ab1a0f32f/konnectivity-agent/0.log" limit=1024
Oct 30 12:51:34.304725 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:34.304704    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vlfkx_7d4b7525-1c1b-4e65-b263-349ab1a0f32f/konnectivity-agent/0.log" limit=1024
Oct 30 12:51:35.001150 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:35.001131    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69eab030-f488-433c-869a-52d628824391-metrics-certs\") pod \"network-metrics-daemon-79wgd\" (UID: \"69eab030-f488-433c-869a-52d628824391\") " pod="openshift-multus/network-metrics-daemon-79wgd"
Oct 30 12:51:35.004450 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:35.004434    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69eab030-f488-433c-869a-52d628824391-metrics-certs\") pod \"network-metrics-daemon-79wgd\" (UID: \"69eab030-f488-433c-869a-52d628824391\") " pod="openshift-multus/network-metrics-daemon-79wgd"
Oct 30 12:51:35.101911 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:35.101864    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/815986e1-07b4-43fb-95f5-745316bd5105-original-pull-secret\") pod \"global-pull-secret-syncer-rxg7f\" (UID: \"815986e1-07b4-43fb-95f5-745316bd5105\") " pod="kube-system/global-pull-secret-syncer-rxg7f"
Oct 30 12:51:35.101911 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:35.101906    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfwjf\" (UniqueName: \"kubernetes.io/projected/c1927296-d775-43c4-a874-8842ebfeb3c8-kube-api-access-jfwjf\") pod \"network-check-target-dlxch\" (UID: \"c1927296-d775-43c4-a874-8842ebfeb3c8\") " pod="openshift-network-diagnostics/network-check-target-dlxch"
Oct 30 12:51:35.104060 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:35.104033    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/815986e1-07b4-43fb-95f5-745316bd5105-original-pull-secret\") pod \"global-pull-secret-syncer-rxg7f\" (UID: \"815986e1-07b4-43fb-95f5-745316bd5105\") " pod="kube-system/global-pull-secret-syncer-rxg7f"
Oct 30 12:51:35.104142 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:35.104075    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfwjf\" (UniqueName: \"kubernetes.io/projected/c1927296-d775-43c4-a874-8842ebfeb3c8-kube-api-access-jfwjf\") pod \"network-check-target-dlxch\" (UID: \"c1927296-d775-43c4-a874-8842ebfeb3c8\") " pod="openshift-network-diagnostics/network-check-target-dlxch"
Oct 30 12:51:35.197086 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:35.197067    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-79wgd"
Oct 30 12:51:35.199220 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:35.199203    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rxg7f"
Oct 30 12:51:35.201368 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:35.201346    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dlxch"
Oct 30 12:51:35.623485 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:35.623463    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rxg7f"]
Oct 30 12:51:35.625880 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:35.625851    2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod815986e1_07b4_43fb_95f5_745316bd5105.slice/crio-73cbd7cb7126e4b9be61ec1ed7fbfa40e963518f310672b2b89d204b8ab80b97 WatchSource:0}: Error finding container 73cbd7cb7126e4b9be61ec1ed7fbfa40e963518f310672b2b89d204b8ab80b97: Status 404 returned error can't find the container with id 73cbd7cb7126e4b9be61ec1ed7fbfa40e963518f310672b2b89d204b8ab80b97
Oct 30 12:51:35.638683 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:35.638660    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dlxch"]
Oct 30 12:51:35.641374 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:35.641347    2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1927296_d775_43c4_a874_8842ebfeb3c8.slice/crio-b3b33c2d5d904bcf841949f7bfee7e3ab8964d94f48833ca1f8fec18652e9be2 WatchSource:0}: Error finding container b3b33c2d5d904bcf841949f7bfee7e3ab8964d94f48833ca1f8fec18652e9be2: Status 404 returned error can't find the container with id b3b33c2d5d904bcf841949f7bfee7e3ab8964d94f48833ca1f8fec18652e9be2
Oct 30 12:51:35.642195 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:35.642175    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-79wgd"]
Oct 30 12:51:35.644510 ip-10-0-137-6 kubenswrapper[2569]: W1030 12:51:35.644487    2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69eab030_f488_433c_869a_52d628824391.slice/crio-bd7b201071dfe2916e1f9dbad684442224542e09ff380e06f7afa30ff071bc6f WatchSource:0}: Error finding container bd7b201071dfe2916e1f9dbad684442224542e09ff380e06f7afa30ff071bc6f: Status 404 returned error can't find the container with id bd7b201071dfe2916e1f9dbad684442224542e09ff380e06f7afa30ff071bc6f
Oct 30 12:51:36.559135 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:36.559097    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dlxch" event={"ID":"c1927296-d775-43c4-a874-8842ebfeb3c8","Type":"ContainerStarted","Data":"b3b33c2d5d904bcf841949f7bfee7e3ab8964d94f48833ca1f8fec18652e9be2"}
Oct 30 12:51:36.560648 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:36.560442    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rxg7f" event={"ID":"815986e1-07b4-43fb-95f5-745316bd5105","Type":"ContainerStarted","Data":"73cbd7cb7126e4b9be61ec1ed7fbfa40e963518f310672b2b89d204b8ab80b97"}
Oct 30 12:51:36.561637 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:36.561614    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-79wgd" event={"ID":"69eab030-f488-433c-869a-52d628824391","Type":"ContainerStarted","Data":"bd7b201071dfe2916e1f9dbad684442224542e09ff380e06f7afa30ff071bc6f"}
Oct 30 12:51:37.568937 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:37.568900    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-79wgd" event={"ID":"69eab030-f488-433c-869a-52d628824391","Type":"ContainerStarted","Data":"ec1a83ccdabeeff0c843098d044539ed906b85469f9281186407cd970fa01842"}
Oct 30 12:51:37.569374 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:37.569000    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-79wgd" event={"ID":"69eab030-f488-433c-869a-52d628824391","Type":"ContainerStarted","Data":"cfe48e05a4aa0d156343e4f3323da0ad2f85fd61daaaff87aa68ebea28628a89"}
Oct 30 12:51:37.616305 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:37.616253    2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-79wgd" podStartSLOduration=34.027768572 podStartE2EDuration="35.616236442s" totalImagesPullingTime="1.58846787s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:51:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:37.61586279 +0000 UTC m=+36.855241007" watchObservedRunningTime="2026-10-30 12:51:37.616236442 +0000 UTC m=+36.855614659"
Oct 30 12:51:39.576054 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:39.575978    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dlxch" event={"ID":"c1927296-d775-43c4-a874-8842ebfeb3c8","Type":"ContainerStarted","Data":"682c70bad8a7b8cc1a5bc5e86f2a94cda88cac5556891631e71dc57f02d0a9fd"}
Oct 30 12:51:39.576433 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:39.576129    2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-dlxch"
Oct 30 12:51:39.620791 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:39.620751    2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-dlxch" podStartSLOduration=34.119405996 podStartE2EDuration="37.620736942s" totalImagesPullingTime="3.501330946s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:51:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:39.619915318 +0000 UTC m=+38.859293532" watchObservedRunningTime="2026-10-30 12:51:39.620736942 +0000 UTC m=+38.860115155"
Oct 30 12:51:40.786747 ip-10-0-137-6 kubenswrapper[2569]: I1030 12:51:40.786655    2569 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 12:51:40.786818 ip-10-0-137-6 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 12:51:40.796979 ip-10-0-137-6 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 12:51:40.797141 ip-10-0-137-6 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 12:51:40.797188 ip-10-0-137-6 systemd[1]: kubelet.service: Consumed 2.005s CPU time.
Oct 30 12:51:40.805178 ip-10-0-137-6 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 12:51:40.891227 ip-10-0-137-6 kubenswrapper[4364]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 12:51:40.891227 ip-10-0-137-6 kubenswrapper[4364]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 12:51:40.891227 ip-10-0-137-6 kubenswrapper[4364]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 12:51:40.891227 ip-10-0-137-6 kubenswrapper[4364]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 12:51:40.893225 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893155    4364 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:51:40.893225 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893224    4364 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:51:40.893225 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893228    4364 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:51:40.893310 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893231    4364 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:51:40.893310 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893235    4364 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:51:40.893310 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893238    4364 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:51:40.893310 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893241    4364 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:51:40.893310 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893244    4364 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:51:40.893310 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893246    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:51:40.893310 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893249    4364 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:51:40.893310 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893252    4364 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:51:40.893310 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893254    4364 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:51:40.893310 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893258    4364 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:51:40.893310 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893260    4364 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:51:40.893310 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893263    4364 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:51:40.893310 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893265    4364 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:51:40.893310 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893268    4364 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:51:40.893310 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893271    4364 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:51:40.893310 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893273    4364 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:51:40.893310 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893276    4364 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:51:40.893310 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893284    4364 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:51:40.893310 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893287    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:51:40.893811 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893289    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:51:40.893811 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893292    4364 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:51:40.893811 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893294    4364 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:51:40.893811 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893297    4364 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:51:40.893811 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893300    4364 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:51:40.893811 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893303    4364 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:51:40.893811 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893305    4364 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:51:40.893811 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893308    4364 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:51:40.893811 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893311    4364 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:51:40.893811 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893314    4364 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:51:40.893811 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893316    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:51:40.893811 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893319    4364 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:51:40.893811 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893322    4364 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:51:40.893811 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893325    4364 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:51:40.893811 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893327    4364 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:51:40.893811 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893330    4364 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:51:40.893811 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893333    4364 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:51:40.893811 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893335    4364 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:51:40.893811 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893338    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:51:40.893811 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893341    4364 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:51:40.894302 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893343    4364 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:51:40.894302 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893345    4364 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:51:40.894302 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893348    4364 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:51:40.894302 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893351    4364 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:51:40.894302 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893354    4364 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:51:40.894302 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893356    4364 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:51:40.894302 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893358    4364 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:51:40.894302 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893361    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:51:40.894302 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893364    4364 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:51:40.894302 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893366    4364 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:51:40.894302 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893369    4364 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:51:40.894302 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893371    4364 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:51:40.894302 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893374    4364 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:51:40.894302 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893377    4364 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:51:40.894302 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893380    4364 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:51:40.894302 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893383    4364 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:51:40.894302 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893386    4364 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:51:40.894302 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893388    4364 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:51:40.894302 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893391    4364 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:51:40.894788 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893393    4364 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:51:40.894788 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893396    4364 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:51:40.894788 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893398    4364 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:51:40.894788 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893401    4364 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:51:40.894788 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893403    4364 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:51:40.894788 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893406    4364 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:51:40.894788 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893408    4364 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:51:40.894788 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893411    4364 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:51:40.894788 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893413    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:51:40.894788 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893422    4364 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:51:40.894788 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893425    4364 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:51:40.894788 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893427    4364 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:51:40.894788 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893430    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:51:40.894788 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893432    4364 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:51:40.894788 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893434    4364 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:51:40.894788 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893437    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:51:40.894788 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893442    4364 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:51:40.894788 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893446    4364 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:51:40.894788 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893450    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:51:40.895260 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893453    4364 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:51:40.895260 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893455    4364 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:51:40.895260 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893458    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:51:40.895260 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893460    4364 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:51:40.895260 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893463    4364 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:51:40.895260 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893465    4364 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:51:40.895260 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893468    4364 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:51:40.895260 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893472    4364 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:51:40.895260 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893475    4364 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:51:40.895260 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893477    4364 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:51:40.895260 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893480    4364 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:51:40.895260 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893483    4364 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:51:40.895260 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893485    4364 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:51:40.895260 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893488    4364 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:51:40.895260 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893490    4364 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:51:40.895260 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893493    4364 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:51:40.895260 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893495    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:51:40.895260 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893498    4364 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:51:40.895260 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893501    4364 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:51:40.895742 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893504    4364 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:51:40.895742 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893508    4364 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:51:40.895742 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893511    4364 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:51:40.895742 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893514    4364 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:51:40.895742 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893523    4364 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:51:40.895742 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893527    4364 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:51:40.895742 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.893529    4364 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:51:40.895742 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894286    4364 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:51:40.895742 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894293    4364 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:51:40.895742 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894299    4364 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:51:40.895742 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894302    4364 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:51:40.895742 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894305    4364 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:51:40.895742 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894307    4364 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:51:40.895742 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894310    4364 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:51:40.895742 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894313    4364 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:51:40.895742 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894316    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:51:40.895742 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894319    4364 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:51:40.895742 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894321    4364 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:51:40.895742 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894325    4364 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:51:40.896253 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894328    4364 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:51:40.896253 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894331    4364 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:51:40.896253 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894334    4364 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:51:40.896253 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894337    4364 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:51:40.896253 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894339    4364 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:51:40.896253 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894342    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:51:40.896253 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894345    4364 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:51:40.896253 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894347    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:51:40.896253 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894350    4364 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:51:40.896253 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894353    4364 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:51:40.896253 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894355    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:51:40.896253 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894358    4364 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:51:40.896253 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894361    4364 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:51:40.896253 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894364    4364 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:51:40.896253 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894366    4364 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:51:40.896253 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894369    4364 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:51:40.896253 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894371    4364 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:51:40.896253 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894374    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:51:40.896253 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894377    4364 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:51:40.896738 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894380    4364 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:51:40.896738 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894382    4364 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:51:40.896738 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894385    4364 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:51:40.896738 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894387    4364 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:51:40.896738 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894390    4364 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:51:40.896738 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894392    4364 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:51:40.896738 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894395    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:51:40.896738 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894397    4364 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:51:40.896738 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894400    4364 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:51:40.896738 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894403    4364 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:51:40.896738 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894405    4364 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:51:40.896738 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894408    4364 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:51:40.896738 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894410    4364 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:51:40.896738 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894414    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:51:40.896738 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894416    4364 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:51:40.896738 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894419    4364 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:51:40.896738 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894422    4364 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:51:40.896738 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894424    4364 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:51:40.896738 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894427    4364 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:51:40.897211 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894430    4364 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:51:40.897211 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894432    4364 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:51:40.897211 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894435    4364 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:51:40.897211 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894438    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:51:40.897211 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894440    4364 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:51:40.897211 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894443    4364 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:51:40.897211 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894445    4364 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:51:40.897211 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894450    4364 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:51:40.897211 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894454    4364 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:51:40.897211 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894457    4364 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:51:40.897211 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894460    4364 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:51:40.897211 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894463    4364 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:51:40.897211 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894466    4364 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:51:40.897211 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894468    4364 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:51:40.897211 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894471    4364 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:51:40.897211 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894473    4364 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:51:40.897211 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894476    4364 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:51:40.897211 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894478    4364 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:51:40.897211 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894481    4364 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:51:40.897688 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894483    4364 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:51:40.897688 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894486    4364 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:51:40.897688 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894488    4364 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:51:40.897688 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894491    4364 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:51:40.897688 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894493    4364 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:51:40.897688 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894496    4364 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:51:40.897688 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894499    4364 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:51:40.897688 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894501    4364 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:51:40.897688 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894504    4364 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:51:40.897688 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894506    4364 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:51:40.897688 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894509    4364 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:51:40.897688 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894512    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:51:40.897688 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894514    4364 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:51:40.897688 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894517    4364 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:51:40.897688 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894519    4364 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:51:40.897688 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894522    4364 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:51:40.897688 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894524    4364 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:51:40.897688 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894527    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:51:40.897688 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894529    4364 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:51:40.897688 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894532    4364 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:51:40.898165 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894534    4364 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:51:40.898165 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894537    4364 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:51:40.898165 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894539    4364 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:51:40.898165 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894542    4364 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:51:40.898165 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894544    4364 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:51:40.898165 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894547    4364 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:51:40.898165 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894549    4364 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:51:40.898165 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894552    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:51:40.898165 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894555    4364 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:51:40.898165 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894558    4364 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:51:40.898165 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894560    4364 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:51:40.898165 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894563    4364 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:51:40.898165 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894567    4364 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:51:40.898165 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894572    4364 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:51:40.898165 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894575    4364 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:51:40.898165 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894578    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:51:40.898165 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.894581    4364 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:51:40.898165 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895077    4364 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:51:40.898165 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895084    4364 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:51:40.898658 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895088    4364 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:51:40.898658 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895092    4364 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:51:40.898658 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895095    4364 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:51:40.898658 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895098    4364 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:51:40.898658 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895101    4364 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:51:40.898658 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895103    4364 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:51:40.898658 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895106    4364 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:51:40.898658 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895108    4364 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:51:40.898658 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895112    4364 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:51:40.898658 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895116    4364 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:51:40.898658 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895120    4364 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:51:40.898658 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895123    4364 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:51:40.898658 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895126    4364 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:51:40.898658 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895129    4364 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:51:40.898658 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895131    4364 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:51:40.898658 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895134    4364 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:51:40.898658 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895137    4364 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:51:40.898658 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895139    4364 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:51:40.898658 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895142    4364 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:51:40.899177 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895144    4364 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:51:40.899177 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895148    4364 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:51:40.899177 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895151    4364 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:51:40.899177 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895154    4364 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:51:40.899177 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895156    4364 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:51:40.899177 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895158    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:51:40.899177 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895161    4364 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:51:40.899177 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895164    4364 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:51:40.899177 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895170    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:51:40.899177 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895173    4364 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:51:40.899177 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895176    4364 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:51:40.899177 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895179    4364 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:51:40.899177 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895182    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:51:40.899177 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895184    4364 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:51:40.899177 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895187    4364 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:51:40.899177 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895190    4364 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:51:40.899177 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895192    4364 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:51:40.899177 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895195    4364 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:51:40.899177 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895197    4364 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:51:40.899969 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895200    4364 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:51:40.899969 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895202    4364 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:51:40.899969 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895205    4364 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:51:40.899969 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895208    4364 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:51:40.899969 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895210    4364 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:51:40.899969 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895213    4364 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:51:40.899969 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895215    4364 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:51:40.899969 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895218    4364 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:51:40.899969 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895220    4364 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:51:40.899969 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895223    4364 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:51:40.899969 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895225    4364 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:51:40.899969 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895228    4364 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:51:40.899969 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895231    4364 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:51:40.899969 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895236    4364 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:51:40.899969 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895238    4364 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:51:40.899969 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895241    4364 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:51:40.899969 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895244    4364 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:51:40.899969 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895246    4364 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:51:40.899969 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895249    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:51:40.899969 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895252    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:51:40.900790 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895255    4364 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:51:40.900790 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895259    4364 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:51:40.900790 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895262    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:51:40.900790 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895265    4364 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:51:40.900790 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895267    4364 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:51:40.900790 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895270    4364 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:51:40.900790 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895272    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:51:40.900790 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895275    4364 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:51:40.900790 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895278    4364 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:51:40.900790 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895280    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:51:40.900790 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895283    4364 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:51:40.900790 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895285    4364 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:51:40.900790 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895287    4364 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:51:40.900790 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895290    4364 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:51:40.900790 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895293    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:51:40.900790 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895295    4364 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:51:40.900790 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895298    4364 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:51:40.900790 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895300    4364 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:51:40.900790 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895303    4364 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:51:40.901611 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895307    4364 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:51:40.901611 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895311    4364 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:51:40.901611 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895314    4364 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:51:40.901611 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895317    4364 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:51:40.901611 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895319    4364 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:51:40.901611 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895322    4364 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:51:40.901611 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895326    4364 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:51:40.901611 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895328    4364 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:51:40.901611 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895331    4364 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:51:40.901611 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895334    4364 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:51:40.901611 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895337    4364 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:51:40.901611 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895339    4364 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:51:40.901611 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895342    4364 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:51:40.901611 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895344    4364 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:51:40.901611 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895348    4364 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:51:40.901611 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895351    4364 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:51:40.901611 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895354    4364 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:51:40.901611 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895356    4364 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:51:40.901611 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895359    4364 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:51:40.901611 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895361    4364 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:51:40.902466 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895365    4364 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:51:40.902466 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895367    4364 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:51:40.902466 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895370    4364 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:51:40.902466 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895372    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:51:40.902466 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895375    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:51:40.902466 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895377    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:51:40.902466 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.895380    4364 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:51:40.902466 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895468    4364 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 12:51:40.902466 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895475    4364 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 12:51:40.902466 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895482    4364 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 12:51:40.902466 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895489    4364 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 12:51:40.902466 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895495    4364 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 12:51:40.902466 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895500    4364 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 12:51:40.902466 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895506    4364 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 12:51:40.902466 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895511    4364 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 12:51:40.902466 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895514    4364 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 12:51:40.902466 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895517    4364 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 12:51:40.902466 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895521    4364 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 12:51:40.902466 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895526    4364 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 12:51:40.902466 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895529    4364 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 12:51:40.902466 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895532    4364 flags.go:64] FLAG: --cgroup-root=""
Oct 30 12:51:40.902466 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895534    4364 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 12:51:40.903519 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895538    4364 flags.go:64] FLAG: --client-ca-file=""
Oct 30 12:51:40.903519 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895541    4364 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 12:51:40.903519 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895544    4364 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 12:51:40.903519 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895548    4364 flags.go:64] FLAG: --cluster-domain=""
Oct 30 12:51:40.903519 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895552    4364 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 12:51:40.903519 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895555    4364 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 12:51:40.903519 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895559    4364 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 12:51:40.903519 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895562    4364 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 12:51:40.903519 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895566    4364 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 12:51:40.903519 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895569    4364 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 12:51:40.903519 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895572    4364 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 12:51:40.903519 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895575    4364 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 12:51:40.903519 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895578    4364 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 12:51:40.903519 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895581    4364 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 12:51:40.903519 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895584    4364 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 12:51:40.903519 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895587    4364 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 12:51:40.903519 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895590    4364 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 12:51:40.903519 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895606    4364 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 12:51:40.903519 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895610    4364 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 12:51:40.903519 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895612    4364 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 12:51:40.903519 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895615    4364 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 12:51:40.903519 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895618    4364 flags.go:64] FLAG: --enable-server="true"
Oct 30 12:51:40.903519 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895621    4364 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 12:51:40.903519 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895625    4364 flags.go:64] FLAG: --event-burst="100"
Oct 30 12:51:40.903519 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895628    4364 flags.go:64] FLAG: --event-qps="50"
Oct 30 12:51:40.904627 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895631    4364 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 12:51:40.904627 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895634    4364 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 12:51:40.904627 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895637    4364 flags.go:64] FLAG: --eviction-hard=""
Oct 30 12:51:40.904627 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895643    4364 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 12:51:40.904627 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895646    4364 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 12:51:40.904627 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895649    4364 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 12:51:40.904627 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895652    4364 flags.go:64] FLAG: --eviction-soft=""
Oct 30 12:51:40.904627 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895655    4364 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 12:51:40.904627 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895659    4364 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 12:51:40.904627 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895662    4364 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 12:51:40.904627 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895665    4364 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 12:51:40.904627 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895669    4364 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 12:51:40.904627 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895672    4364 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 12:51:40.904627 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895675    4364 flags.go:64] FLAG: --feature-gates=""
Oct 30 12:51:40.904627 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895679    4364 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 12:51:40.904627 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895682    4364 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 12:51:40.904627 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895685    4364 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 12:51:40.904627 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895689    4364 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 12:51:40.904627 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895692    4364 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 12:51:40.904627 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895695    4364 flags.go:64] FLAG: --help="false"
Oct 30 12:51:40.904627 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895698    4364 flags.go:64] FLAG: --hostname-override="ip-10-0-137-6.ec2.internal"
Oct 30 12:51:40.904627 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895702    4364 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 12:51:40.904627 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895705    4364 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 12:51:40.904627 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895707    4364 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 12:51:40.905685 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895711    4364 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 12:51:40.905685 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895714    4364 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 12:51:40.905685 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895717    4364 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 12:51:40.905685 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895720    4364 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 12:51:40.905685 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895723    4364 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 12:51:40.905685 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895726    4364 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 12:51:40.905685 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895729    4364 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 12:51:40.905685 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895732    4364 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 12:51:40.905685 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895735    4364 flags.go:64] FLAG: --kube-reserved=""
Oct 30 12:51:40.905685 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895738    4364 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 12:51:40.905685 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895741    4364 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 12:51:40.905685 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895746    4364 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 12:51:40.905685 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895749    4364 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 12:51:40.905685 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895752    4364 flags.go:64] FLAG: --lock-file=""
Oct 30 12:51:40.905685 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895755    4364 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 12:51:40.905685 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895758    4364 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 12:51:40.905685 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895761    4364 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 12:51:40.905685 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895767    4364 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 12:51:40.905685 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895770    4364 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 12:51:40.905685 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895774    4364 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 12:51:40.905685 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895777    4364 flags.go:64] FLAG: --logging-format="text"
Oct 30 12:51:40.905685 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895780    4364 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 12:51:40.905685 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895783    4364 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 12:51:40.905685 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895786    4364 flags.go:64] FLAG: --manifest-url=""
Oct 30 12:51:40.906731 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895789    4364 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 12:51:40.906731 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895793    4364 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 12:51:40.906731 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895796    4364 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 12:51:40.906731 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895800    4364 flags.go:64] FLAG: --max-pods="110"
Oct 30 12:51:40.906731 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895803    4364 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 12:51:40.906731 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895806    4364 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 12:51:40.906731 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895809    4364 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 12:51:40.906731 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895812    4364 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 12:51:40.906731 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895815    4364 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 12:51:40.906731 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895817    4364 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 12:51:40.906731 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895820    4364 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 12:51:40.906731 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895827    4364 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 12:51:40.906731 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895830    4364 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 12:51:40.906731 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895833    4364 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 12:51:40.906731 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895836    4364 flags.go:64] FLAG: --pod-cidr=""
Oct 30 12:51:40.906731 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895839    4364 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 12:51:40.906731 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895842    4364 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 12:51:40.906731 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895845    4364 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 12:51:40.906731 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895848    4364 flags.go:64] FLAG: --port="10250"
Oct 30 12:51:40.906731 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895852    4364 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 12:51:40.906731 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895855    4364 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0265679eade489590"
Oct 30 12:51:40.906731 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895858    4364 flags.go:64] FLAG: --qos-reserved=""
Oct 30 12:51:40.906731 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895861    4364 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 12:51:40.906731 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895864    4364 flags.go:64] FLAG: --register-node="true"
Oct 30 12:51:40.906731 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895867    4364 flags.go:64] FLAG: --register-with-taints=""
Oct 30 12:51:40.906731 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895871    4364 flags.go:64] FLAG: --registry-burst="10"
Oct 30 12:51:40.907839 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895878    4364 flags.go:64] FLAG: --registry-qps="5"
Oct 30 12:51:40.907839 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895881    4364 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 12:51:40.907839 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895884    4364 flags.go:64] FLAG: --reserved-memory=""
Oct 30 12:51:40.907839 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895888    4364 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 12:51:40.907839 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895891    4364 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 12:51:40.907839 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895894    4364 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 12:51:40.907839 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895897    4364 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 12:51:40.907839 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895900    4364 flags.go:64] FLAG: --runonce="false"
Oct 30 12:51:40.907839 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895903    4364 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 12:51:40.907839 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895906    4364 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 12:51:40.907839 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895909    4364 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 12:51:40.907839 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895912    4364 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 12:51:40.907839 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895914    4364 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 12:51:40.907839 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895917    4364 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 12:51:40.907839 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895920    4364 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 12:51:40.907839 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895923    4364 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 12:51:40.907839 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895926    4364 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 12:51:40.907839 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895929    4364 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 12:51:40.907839 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895932    4364 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 12:51:40.907839 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895935    4364 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 12:51:40.907839 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895938    4364 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 12:51:40.907839 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895941    4364 flags.go:64] FLAG: --system-cgroups=""
Oct 30 12:51:40.907839 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895944    4364 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 12:51:40.907839 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895949    4364 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 12:51:40.907839 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895952    4364 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 12:51:40.908775 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895956    4364 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 12:51:40.908775 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895960    4364 flags.go:64] FLAG: --tls-min-version=""
Oct 30 12:51:40.908775 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895963    4364 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 12:51:40.908775 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895966    4364 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 12:51:40.908775 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895969    4364 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 12:51:40.908775 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895971    4364 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 12:51:40.908775 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895975    4364 flags.go:64] FLAG: --v="2"
Oct 30 12:51:40.908775 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895979    4364 flags.go:64] FLAG: --version="false"
Oct 30 12:51:40.908775 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895984    4364 flags.go:64] FLAG: --vmodule=""
Oct 30 12:51:40.908775 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895988    4364 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 12:51:40.908775 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.895991    4364 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 12:51:40.908775 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896113    4364 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:51:40.908775 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896116    4364 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:51:40.908775 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896119    4364 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:51:40.908775 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896122    4364 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:51:40.908775 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896125    4364 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:51:40.908775 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896128    4364 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:51:40.908775 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896131    4364 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:51:40.908775 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896134    4364 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:51:40.908775 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896137    4364 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:51:40.908775 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896139    4364 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:51:40.908775 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896142    4364 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:51:40.909362 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896144    4364 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:51:40.909362 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896147    4364 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:51:40.909362 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896149    4364 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:51:40.909362 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896152    4364 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:51:40.909362 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896155    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:51:40.909362 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896158    4364 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:51:40.909362 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896160    4364 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:51:40.909362 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896163    4364 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:51:40.909362 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896165    4364 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:51:40.909362 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896168    4364 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:51:40.909362 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896172    4364 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:51:40.909362 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896175    4364 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:51:40.909362 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896178    4364 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:51:40.909362 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896180    4364 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:51:40.909362 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896183    4364 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:51:40.909362 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896185    4364 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:51:40.909362 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896188    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:51:40.909362 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896190    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:51:40.909362 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896193    4364 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:51:40.909362 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896196    4364 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:51:40.910035 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896198    4364 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:51:40.910035 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896201    4364 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:51:40.910035 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896203    4364 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:51:40.910035 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896206    4364 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:51:40.910035 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896208    4364 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:51:40.910035 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896211    4364 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:51:40.910035 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896214    4364 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:51:40.910035 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896216    4364 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:51:40.910035 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896218    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:51:40.910035 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896221    4364 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:51:40.910035 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896223    4364 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:51:40.910035 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896226    4364 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:51:40.910035 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896228    4364 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:51:40.910035 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896231    4364 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:51:40.910035 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896233    4364 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:51:40.910035 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896237    4364 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:51:40.910035 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896240    4364 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:51:40.910035 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896244    4364 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:51:40.910035 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896248    4364 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:51:40.910888 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896251    4364 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:51:40.910888 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896254    4364 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:51:40.910888 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896257    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:51:40.910888 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896261    4364 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:51:40.910888 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896264    4364 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:51:40.910888 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896266    4364 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:51:40.910888 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896269    4364 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:51:40.910888 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896272    4364 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:51:40.910888 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896275    4364 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:51:40.910888 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896277    4364 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:51:40.910888 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896280    4364 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:51:40.910888 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896283    4364 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:51:40.910888 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896285    4364 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:51:40.910888 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896288    4364 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:51:40.910888 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896291    4364 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:51:40.910888 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896293    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:51:40.910888 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896296    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:51:40.910888 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896298    4364 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:51:40.910888 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896301    4364 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:51:40.910888 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896303    4364 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:51:40.911667 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896306    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:51:40.911667 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896308    4364 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:51:40.911667 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896311    4364 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:51:40.911667 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896313    4364 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:51:40.911667 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896316    4364 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:51:40.911667 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896319    4364 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:51:40.911667 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896321    4364 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:51:40.911667 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896325    4364 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:51:40.911667 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896330    4364 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:51:40.911667 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896333    4364 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:51:40.911667 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896336    4364 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:51:40.911667 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896339    4364 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:51:40.911667 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896342    4364 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:51:40.911667 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896344    4364 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:51:40.911667 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896347    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:51:40.911667 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896353    4364 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:51:40.911667 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896356    4364 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:51:40.911667 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896359    4364 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:51:40.911667 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896361    4364 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:51:40.912436 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896364    4364 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:51:40.912436 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896367    4364 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:51:40.912436 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896369    4364 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:51:40.912436 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896372    4364 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:51:40.912436 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896375    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:51:40.912436 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896377    4364 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:51:40.912436 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896380    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:51:40.912436 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896383    4364 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:51:40.912436 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896385    4364 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:51:40.912436 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896388    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:51:40.912436 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896390    4364 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:51:40.912436 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896393    4364 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:51:40.912436 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896395    4364 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:51:40.912436 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896398    4364 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:51:40.912436 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896400    4364 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:51:40.912436 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896403    4364 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:51:40.912436 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896405    4364 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:51:40.913012 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.896431    4364 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 12:51:40.913012 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896466    4364 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:51:40.913012 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896471    4364 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:51:40.913012 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896475    4364 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:51:40.913012 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896478    4364 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:51:40.913012 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896482    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:51:40.913012 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896485    4364 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:51:40.913012 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896488    4364 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:51:40.913012 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896491    4364 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:51:40.913012 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896493    4364 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:51:40.913012 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896496    4364 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:51:40.913012 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896500    4364 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:51:40.913012 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896502    4364 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:51:40.913012 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896505    4364 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:51:40.913012 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896508    4364 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:51:40.913396 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896510    4364 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:51:40.913396 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896513    4364 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:51:40.913396 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896515    4364 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:51:40.913396 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896518    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:51:40.913396 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896520    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:51:40.913396 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896523    4364 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:51:40.913396 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896525    4364 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:51:40.913396 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896528    4364 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:51:40.913396 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896531    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:51:40.913396 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896534    4364 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:51:40.913396 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896536    4364 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:51:40.913396 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896539    4364 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:51:40.913396 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896541    4364 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:51:40.913396 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896544    4364 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:51:40.913396 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896546    4364 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:51:40.913396 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896550    4364 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:51:40.913396 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896553    4364 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:51:40.913396 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896556    4364 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:51:40.913396 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896560    4364 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:51:40.913924 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896563    4364 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:51:40.913924 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896566    4364 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:51:40.913924 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896568    4364 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:51:40.913924 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896572    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:51:40.913924 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896575    4364 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:51:40.913924 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896577    4364 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:51:40.913924 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896580    4364 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:51:40.913924 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896583    4364 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:51:40.913924 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896585    4364 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:51:40.913924 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896589    4364 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:51:40.913924 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896591    4364 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:51:40.913924 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896607    4364 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:51:40.913924 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896610    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:51:40.913924 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896613    4364 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:51:40.913924 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896616    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:51:40.913924 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896618    4364 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:51:40.913924 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896621    4364 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:51:40.913924 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896623    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:51:40.913924 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896626    4364 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:51:40.913924 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896629    4364 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:51:40.914418 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896631    4364 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:51:40.914418 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896634    4364 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:51:40.914418 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896637    4364 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:51:40.914418 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896640    4364 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:51:40.914418 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896642    4364 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:51:40.914418 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896645    4364 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:51:40.914418 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896648    4364 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:51:40.914418 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896651    4364 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:51:40.914418 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896653    4364 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:51:40.914418 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896656    4364 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:51:40.914418 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896658    4364 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:51:40.914418 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896661    4364 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:51:40.914418 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896663    4364 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:51:40.914418 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896666    4364 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:51:40.914418 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896668    4364 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:51:40.914418 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896672    4364 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:51:40.914418 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896675    4364 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:51:40.914418 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896677    4364 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:51:40.914418 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896680    4364 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:51:40.914892 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896682    4364 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:51:40.914892 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896685    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:51:40.914892 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896689    4364 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:51:40.914892 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896691    4364 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:51:40.914892 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896694    4364 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:51:40.914892 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896697    4364 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:51:40.914892 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896699    4364 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:51:40.914892 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896702    4364 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:51:40.914892 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896705    4364 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:51:40.914892 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896707    4364 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:51:40.914892 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896710    4364 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:51:40.914892 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896712    4364 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:51:40.914892 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896715    4364 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:51:40.914892 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896717    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:51:40.914892 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896720    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:51:40.914892 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896723    4364 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:51:40.914892 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896726    4364 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:51:40.914892 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896729    4364 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:51:40.914892 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896731    4364 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:51:40.914892 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896734    4364 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:51:40.915432 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896736    4364 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:51:40.915432 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896739    4364 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:51:40.915432 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896742    4364 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:51:40.915432 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896744    4364 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:51:40.915432 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896747    4364 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:51:40.915432 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896750    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:51:40.915432 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896752    4364 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:51:40.915432 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896755    4364 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:51:40.915432 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896759    4364 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:51:40.915432 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896761    4364 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:51:40.915432 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896764    4364 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:51:40.915432 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896766    4364 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:51:40.915432 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896769    4364 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:51:40.915432 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.896772    4364 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:51:40.915432 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.901910    4364 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 12:51:40.915432 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.901933    4364 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 12:51:40.915432 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.901957    4364 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 12:51:40.915432 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.901965    4364 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 12:51:40.915432 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902060    4364 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:51:40.916239 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902068    4364 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:51:40.916239 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902073    4364 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:51:40.916239 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902078    4364 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:51:40.916239 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902082    4364 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:51:40.916239 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902087    4364 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:51:40.916239 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902092    4364 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:51:40.916239 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902096    4364 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:51:40.916239 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902100    4364 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:51:40.916239 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902105    4364 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:51:40.916239 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902109    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:51:40.916239 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902114    4364 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:51:40.916239 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902118    4364 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:51:40.916239 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902123    4364 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:51:40.916239 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902127    4364 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:51:40.916239 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902132    4364 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:51:40.916239 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902136    4364 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:51:40.916239 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902140    4364 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:51:40.916239 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902144    4364 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:51:40.916239 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902148    4364 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:51:40.916239 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902153    4364 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:51:40.916887 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902157    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:51:40.916887 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902161    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:51:40.916887 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902166    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:51:40.916887 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902171    4364 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:51:40.916887 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902175    4364 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:51:40.916887 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902179    4364 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:51:40.916887 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902184    4364 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:51:40.916887 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902189    4364 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:51:40.916887 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902193    4364 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:51:40.916887 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902198    4364 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:51:40.916887 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902202    4364 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:51:40.916887 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902207    4364 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:51:40.916887 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902211    4364 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:51:40.916887 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902216    4364 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:51:40.916887 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902220    4364 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:51:40.916887 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902225    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:51:40.916887 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902229    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:51:40.916887 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902233    4364 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:51:40.916887 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902237    4364 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:51:40.916887 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902241    4364 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:51:40.917357 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902246    4364 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:51:40.917357 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902253    4364 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:51:40.917357 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902259    4364 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:51:40.917357 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902264    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:51:40.917357 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902268    4364 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:51:40.917357 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902272    4364 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:51:40.917357 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902280    4364 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:51:40.917357 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902286    4364 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:51:40.917357 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902290    4364 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:51:40.917357 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902295    4364 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:51:40.917357 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902299    4364 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:51:40.917357 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902304    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:51:40.917357 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902308    4364 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:51:40.917357 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902312    4364 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:51:40.917357 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902316    4364 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:51:40.917357 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902321    4364 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:51:40.917357 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902326    4364 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:51:40.917357 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902330    4364 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:51:40.917357 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902334    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:51:40.918114 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902338    4364 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:51:40.918114 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902344    4364 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:51:40.918114 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902348    4364 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:51:40.918114 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902353    4364 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:51:40.918114 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902357    4364 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:51:40.918114 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902362    4364 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:51:40.918114 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902366    4364 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:51:40.918114 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902370    4364 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:51:40.918114 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902374    4364 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:51:40.918114 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902379    4364 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:51:40.918114 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902382    4364 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:51:40.918114 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902387    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:51:40.918114 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902392    4364 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:51:40.918114 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902396    4364 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:51:40.918114 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902401    4364 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:51:40.918114 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902405    4364 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:51:40.918114 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902409    4364 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:51:40.918114 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902413    4364 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:51:40.918114 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902418    4364 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:51:40.918883 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902423    4364 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:51:40.918883 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902427    4364 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:51:40.918883 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902432    4364 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:51:40.918883 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902436    4364 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:51:40.918883 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902440    4364 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:51:40.918883 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902445    4364 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:51:40.918883 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902450    4364 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:51:40.918883 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902454    4364 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:51:40.918883 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902458    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:51:40.918883 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902463    4364 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:51:40.918883 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902468    4364 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:51:40.918883 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902472    4364 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:51:40.918883 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902476    4364 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:51:40.918883 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902479    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:51:40.918883 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902484    4364 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:51:40.918883 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902488    4364 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:51:40.918883 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902492    4364 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:51:40.918883 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902498    4364 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:51:40.918883 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902502    4364 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:51:40.918883 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902507    4364 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:51:40.919405 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902511    4364 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:51:40.919405 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902515    4364 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:51:40.919405 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902520    4364 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:51:40.919405 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902524    4364 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:51:40.919405 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902528    4364 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:51:40.919405 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902533    4364 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:51:40.919405 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902537    4364 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:51:40.919405 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.902578    4364 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 12:51:40.919405 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902832    4364 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:51:40.919405 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902841    4364 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:51:40.919405 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902846    4364 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:51:40.919405 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902850    4364 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:51:40.919405 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902855    4364 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:51:40.919405 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902868    4364 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:51:40.919405 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902873    4364 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:51:40.919808 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902877    4364 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:51:40.919808 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902882    4364 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:51:40.919808 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902887    4364 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:51:40.919808 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902891    4364 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:51:40.919808 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902895    4364 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:51:40.919808 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902900    4364 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:51:40.919808 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902904    4364 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:51:40.919808 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902909    4364 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:51:40.919808 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902913    4364 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:51:40.919808 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902917    4364 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:51:40.919808 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902921    4364 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:51:40.919808 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902926    4364 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:51:40.919808 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902931    4364 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:51:40.919808 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902935    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:51:40.919808 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902939    4364 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:51:40.919808 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902944    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:51:40.919808 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902948    4364 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:51:40.919808 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902952    4364 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:51:40.919808 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902957    4364 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:51:40.919808 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902961    4364 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:51:40.920363 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902965    4364 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:51:40.920363 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902969    4364 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:51:40.920363 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902974    4364 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:51:40.920363 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902979    4364 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:51:40.920363 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902983    4364 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:51:40.920363 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902987    4364 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:51:40.920363 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902991    4364 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:51:40.920363 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.902996    4364 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:51:40.920363 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903000    4364 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:51:40.920363 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903004    4364 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:51:40.920363 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903008    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:51:40.920363 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903019    4364 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:51:40.920363 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903024    4364 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:51:40.920363 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903028    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:51:40.920363 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903033    4364 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:51:40.920363 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903037    4364 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:51:40.920363 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903041    4364 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:51:40.920363 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903045    4364 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:51:40.920363 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903050    4364 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:51:40.920868 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903056    4364 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:51:40.920868 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903063    4364 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:51:40.920868 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903069    4364 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:51:40.920868 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903075    4364 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:51:40.920868 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903080    4364 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:51:40.920868 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903084    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:51:40.920868 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903089    4364 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:51:40.920868 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903093    4364 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:51:40.920868 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903097    4364 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:51:40.920868 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903101    4364 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:51:40.920868 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903105    4364 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:51:40.920868 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903110    4364 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:51:40.920868 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903114    4364 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:51:40.920868 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903118    4364 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:51:40.920868 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903122    4364 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:51:40.920868 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903127    4364 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:51:40.920868 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903131    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:51:40.920868 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903135    4364 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:51:40.920868 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903139    4364 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:51:40.921389 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903144    4364 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:51:40.921389 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903148    4364 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:51:40.921389 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903153    4364 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:51:40.921389 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903157    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:51:40.921389 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903161    4364 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:51:40.921389 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903166    4364 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:51:40.921389 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903171    4364 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:51:40.921389 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903175    4364 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:51:40.921389 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903179    4364 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:51:40.921389 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903183    4364 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:51:40.921389 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903187    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:51:40.921389 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903192    4364 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:51:40.921389 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903196    4364 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:51:40.921389 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903201    4364 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:51:40.921389 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903205    4364 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:51:40.921389 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903210    4364 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:51:40.921389 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903214    4364 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:51:40.921389 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903218    4364 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:51:40.921389 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903222    4364 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:51:40.921389 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903227    4364 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:51:40.922178 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903231    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:51:40.922178 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903235    4364 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:51:40.922178 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903240    4364 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:51:40.922178 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903257    4364 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:51:40.922178 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903261    4364 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:51:40.922178 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903266    4364 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:51:40.922178 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903271    4364 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:51:40.922178 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903275    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:51:40.922178 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903283    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:51:40.922178 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903288    4364 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:51:40.922178 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903292    4364 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:51:40.922178 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903298    4364 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:51:40.922178 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903304    4364 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:51:40.922178 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903309    4364 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:51:40.922178 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903314    4364 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:51:40.922178 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903319    4364 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:51:40.922178 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903323    4364 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:51:40.922178 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903328    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:51:40.922178 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903332    4364 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:51:40.923012 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903337    4364 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:51:40.923012 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903341    4364 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:51:40.923012 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.903381    4364 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 12:51:40.923012 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903436    4364 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:51:40.923012 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903443    4364 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:51:40.923012 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903448    4364 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:51:40.923012 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903453    4364 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:51:40.923012 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903457    4364 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:51:40.923012 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903462    4364 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:51:40.923012 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903467    4364 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:51:40.923012 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903472    4364 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:51:40.923012 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903476    4364 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:51:40.923012 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903480    4364 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:51:40.923012 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903484    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:51:40.923012 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903489    4364 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:51:40.923012 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903493    4364 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:51:40.923784 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903498    4364 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:51:40.923784 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903502    4364 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:51:40.923784 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903506    4364 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:51:40.923784 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903511    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:51:40.923784 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903515    4364 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:51:40.923784 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903521    4364 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:51:40.923784 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903525    4364 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:51:40.923784 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903529    4364 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:51:40.923784 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903533    4364 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:51:40.923784 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903537    4364 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:51:40.923784 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903541    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:51:40.923784 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903546    4364 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:51:40.923784 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903550    4364 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:51:40.923784 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903555    4364 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:51:40.923784 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903560    4364 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:51:40.923784 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903564    4364 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:51:40.923784 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903569    4364 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:51:40.923784 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903574    4364 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:51:40.923784 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903578    4364 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:51:40.923784 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903584    4364 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:51:40.924763 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903589    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:51:40.924763 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903593    4364 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:51:40.924763 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903618    4364 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:51:40.924763 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903622    4364 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:51:40.924763 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903626    4364 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:51:40.924763 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903631    4364 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:51:40.924763 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903635    4364 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:51:40.924763 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903640    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:51:40.924763 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903644    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:51:40.924763 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903648    4364 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:51:40.924763 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903652    4364 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:51:40.924763 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903660    4364 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:51:40.924763 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903666    4364 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:51:40.924763 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903671    4364 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:51:40.924763 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903675    4364 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:51:40.924763 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903679    4364 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:51:40.924763 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903684    4364 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:51:40.924763 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903690    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:51:40.924763 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903694    4364 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:51:40.925553 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903698    4364 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:51:40.925553 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903703    4364 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:51:40.925553 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903707    4364 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:51:40.925553 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903711    4364 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:51:40.925553 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903716    4364 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:51:40.925553 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903720    4364 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:51:40.925553 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903724    4364 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:51:40.925553 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903728    4364 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:51:40.925553 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903733    4364 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:51:40.925553 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903737    4364 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:51:40.925553 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903741    4364 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:51:40.925553 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903747    4364 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:51:40.925553 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903754    4364 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:51:40.925553 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903758    4364 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:51:40.925553 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903763    4364 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:51:40.925553 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903767    4364 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:51:40.925553 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903771    4364 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:51:40.925553 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903775    4364 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:51:40.925553 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903779    4364 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:51:40.925553 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903784    4364 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:51:40.926442 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903788    4364 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:51:40.926442 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903793    4364 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:51:40.926442 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903797    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:51:40.926442 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903802    4364 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:51:40.926442 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903806    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:51:40.926442 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903810    4364 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:51:40.926442 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903815    4364 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:51:40.926442 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903819    4364 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:51:40.926442 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903823    4364 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:51:40.926442 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903827    4364 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:51:40.926442 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903836    4364 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:51:40.926442 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903841    4364 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:51:40.926442 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903845    4364 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:51:40.926442 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903849    4364 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:51:40.926442 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903853    4364 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:51:40.926442 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903858    4364 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:51:40.926442 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903862    4364 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:51:40.926442 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903866    4364 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:51:40.926442 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903871    4364 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:51:40.926442 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903875    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:51:40.927346 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903879    4364 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:51:40.927346 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903883    4364 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:51:40.927346 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903888    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:51:40.927346 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903892    4364 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:51:40.927346 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903898    4364 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:51:40.927346 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903902    4364 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:51:40.927346 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903907    4364 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:51:40.927346 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903911    4364 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:51:40.927346 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903918    4364 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:51:40.927346 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903924    4364 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:51:40.927346 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903931    4364 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:51:40.927346 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903935    4364 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:51:40.927346 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903940    4364 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:51:40.927346 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:40.903947    4364 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:51:40.927346 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.904030    4364 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 12:51:40.927346 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.905552    4364 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 12:51:40.927346 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.905640    4364 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 12:51:40.927346 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.905916    4364 server.go:1014] "Starting client certificate rotation"
Oct 30 12:51:40.928127 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.906047    4364 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 12:51:40.928127 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.906092    4364 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 12:46:01 +0000 UTC" deadline="2028-05-30 02:38:59.321817956 +0000 UTC"
Oct 30 12:51:40.928127 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.906127    4364 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13861h47m18.415693027s"
Oct 30 12:51:40.928127 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.906632    4364 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 12:51:40.928127 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.908273    4364 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 12:51:40.928127 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.909745    4364 log.go:25] "Validated CRI v1 runtime API"
Oct 30 12:51:40.928127 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.911616    4364 log.go:25] "Validated CRI v1 image API"
Oct 30 12:51:40.928127 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.912303    4364 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 12:51:40.928127 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.913613    4364 fs.go:135] Filesystem UUIDs: map[79892a95-1ec4-412c-98aa-4788f9cef552:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 cc1aad2c-dca2-41db-bf03-471d34c82d3d:/dev/nvme0n1p3]
Oct 30 12:51:40.928489 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.913633    4364 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/209dbc8895cd9800cf2706ae099c9d409f916b6439290d9964aab249e7d3dd51/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/209dbc8895cd9800cf2706ae099c9d409f916b6439290d9964aab249e7d3dd51/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5db4412bd71a6d83812f0301b973336ae78cdeadaac587d366503762741963b7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5db4412bd71a6d83812f0301b973336ae78cdeadaac587d366503762741963b7/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/73cbd7cb7126e4b9be61ec1ed7fbfa40e963518f310672b2b89d204b8ab80b97/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/73cbd7cb7126e4b9be61ec1ed7fbfa40e963518f310672b2b89d204b8ab80b97/userdata/shm major:0 minor:204 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/75919339ef0b9c35042d1e42d475595815f0bec1ff4bfee178334d65fe6c9e99/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/75919339ef0b9c35042d1e42d475595815f0bec1ff4bfee178334d65fe6c9e99/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/841c7315696222e35d4b041031582f10b1b24393ed6535120d67e8c0f88c6f4a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/841c7315696222e35d4b041031582f10b1b24393ed6535120d67e8c0f88c6f4a/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/85ca31ad3a144e45ff7a5c955c9f421b21596c0a6dbf03d3496fab6c5afeb18a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/85ca31ad3a144e45ff7a5c955c9f421b21596c0a6dbf03d3496fab6c5afeb18a/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9fb14d5547656f78b7318c2bab087962d514662e903b402003c3830713792fe5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9fb14d5547656f78b7318c2bab087962d514662e903b402003c3830713792fe5/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a103dec3bf04fc7374a948e56580a14754926df6a11fdc6fa78b06ab10452ed4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a103dec3bf04fc7374a948e56580a14754926df6a11fdc6fa78b06ab10452ed4/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b3b33c2d5d904bcf841949f7bfee7e3ab8964d94f48833ca1f8fec18652e9be2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b3b33c2d5d904bcf841949f7bfee7e3ab8964d94f48833ca1f8fec18652e9be2/userdata/shm major:0 minor:206 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ba1b5a455d610b60c5d2e4e14e0168f6a3555dfacf0ec593869f1489f8c588d1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ba1b5a455d610b60c5d2e4e14e0168f6a3555dfacf0ec593869f1489f8c588d1/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bd7b201071dfe2916e1f9dbad684442224542e09ff380e06f7afa30ff071bc6f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bd7b201071dfe2916e1f9dbad684442224542e09ff380e06f7afa30ff071bc6f/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c883f63b50ff78fea1cbad9c51a48d375c9dab138bf49c1e08b0a71211be2a8e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c883f63b50ff78fea1cbad9c51a48d375c9dab138bf49c1e08b0a71211be2a8e/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/febe2ba051c1b55bf7ad7e46d5651b755cb0e91501337c63e0d74aea981ca5dd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/febe2ba051c1b55bf7ad7e46d5651b755cb0e91501337c63e0d74aea981ca5dd/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/19115051-3b70-47cd-a7a3-313b1b3c26c7/volumes/kubernetes.io~projected/kube-api-access-78v2l:{mountpoint:/var/lib/kubelet/pods/19115051-3b70-47cd-a7a3-313b1b3c26c7/volumes/kubernetes.io~projected/kube-api-access-78v2l major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2d5efe23-0e0d-4a4e-9d03-42829f371c09/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/2d5efe23-0e0d-4a4e-9d03-42829f371c09/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2d5efe23-0e0d-4a4e-9d03-42829f371c09/volumes/kubernetes.io~projected/kube-api-access-x5s5m:{mountpoint:/var/lib/kubelet/pods/2d5efe23-0e0d-4a4e-9d03-42829f371c09/volumes/kubernetes.io~projected/kube-api-access-x5s5m major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2d5efe23-0e0d-4a4e-9d03-42829f371c09/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/2d5efe23-0e0d-4a4e-9d03-42829f371c09/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31d09b91-c973-4cf9-a7d8-d602ff9994f6/volumes/kubernetes.io~projected/kube-api-access-zdplx:{mountpoint:/var/lib/kubelet/pods/31d09b91-c973-4cf9-a7d8-d602ff9994f6/volumes/kubernetes.io~projected/kube-api-access-zdplx major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/69eab030-f488-433c-869a-52d628824391/volumes/kubernetes.io~projected/kube-api-access-hxq79:{mountpoint:/var/lib/kubelet/pods/69eab030-f488-433c-869a-52d628824391/volumes/kubernetes.io~projected/kube-api-access-hxq79 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/69eab030-f488-433c-869a-52d628824391/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/69eab030-f488-433c-869a-52d628824391/volumes/kubernetes.io~secret/metrics-certs major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7d4b7525-1c1b-4e65-b263-349ab1a0f32f/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/7d4b7525-1c1b-4e65-b263-349ab1a0f32f/volumes/kubernetes.io~secret/agent-certs major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/815986e1-07b4-43fb-95f5-745316bd5105/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/815986e1-07b4-43fb-95f5-745316bd5105/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/85602130-14de-443d-b237-b213c0ce8b82/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/85602130-14de-443d-b237-b213c0ce8b82/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/85602130-14de-443d-b237-b213c0ce8b82/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/85602130-14de-443d-b237-b213c0ce8b82/volumes/kubernetes.io~empty-dir/tmp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/85602130-14de-443d-b237-b213c0ce8b82/volumes/kubernetes.io~projected/kube-api-access-749nf:{mountpoint:/var/lib/kubelet/pods/85602130-14de-443d-b237-b213c0ce8b82/volumes/kubernetes.io~projected/kube-api-access-749nf major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a1ca584f-61b1-4398-a0f6-5788e476aa20/volumes/kubernetes.io~projected/kube-api-access-7pppc:{mountpoint:/var/lib/kubelet/pods/a1ca584f-61b1-4398-a0f6-5788e476aa20/volumes/kubernetes.io~projected/kube-api-access-7pppc major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c1927296-d775-43c4-a874-8842ebfeb3c8/volumes/kubernetes.io~projected/kube-api-access-jfwjf:{mountpoint:/var/lib/kubelet/pods/c1927296-d775-43c4-a874-8842ebfeb3c8/volumes/kubernetes.io~projected/kube-api-access-jfwjf major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c83bcf7b-a1d6-457b-9d7c-bb03c2cbd449/volumes/kubernetes.io~projected/kube-api-access-ls6hw:{mountpoint:/var/lib/kubelet/pods/c83bcf7b-a1d6-457b-9d7c-bb03c2cbd449/volumes/kubernetes.io~projected/kube-api-access-ls6hw major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f8f7d922-7364-444f-a932-278628e26b83/volumes/kubernetes.io~projected/kube-api-access-wtppn:{mountpoint:/var/lib/kubelet/pods/f8f7d922-7364-444f-a932-278628e26b83/volumes/kubernetes.io~projected/kube-api-access-wtppn major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ff0a920b-f15f-4909-906b-61dc41ef1263/volumes/kubernetes.io~projected/kube-api-access-q6s7g:{mountpoint:/var/lib/kubelet/pods/ff0a920b-f15f-4909-906b-61dc41ef1263/volumes/kubernetes.io~projected/kube-api-access-q6s7g major:0 minor:61 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/83a488b588b79bf53fd54bfeab7c2a9dc3441e6f84f19da34ab90b8e80bb731e/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/75570a7b54fc371ec2cac9e925c7c085d0668ec23bbcb4e36d94e458abcb51b4/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/b6c4bc61cdb992cf7f484fd21428a216e825f0285b01ca150dea5117e65244dd/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/08b498136811bfffb1267c118ce8ac88ecde3a6d02af8c29b1837165b782cc2e/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/d001306e681b28eff8e7e2b42aa0453ad89c6a36f41159f722dec25e25516eff/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/daf017e5e12a83217087d599cd97cda0a02c6dc763c4eb22ee5788f64225175e/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/3385fd65ef4d1ceb524bd69a891047fbc0c95d8feb57e6ffe2e17a49a6b2e2f1/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/42293928268b51075a8def69c9fe13f874abf6a797842e8ce7e26d9690b80b21/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/d5d107ea64dda67618efb900a1742c1c94e1ae2dc44ef16f10fbc46473c12ee5/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/f1800f58b7a531dcfc6597aced4fa23a87ae12be466ee6b4dce128dddbf22621/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/f787bc4ab83b5ab90a8a3f14abb2ff22e5005a0df656e5cf65eeee6741206281/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-145:{mountpoint:/var/lib/containers/storage/overlay/edaaab821499949777cb18cd287a335e9834ae6cda8f5450110e2cf6579b650e/merged major:0 minor:145 fsType:overlay blockSize:0} overlay_0-148:{mountpoint:/var/lib/containers/storage/overlay/dacbd003b372fb2c83380fd77c0b27947a328c9d2821b2ac6836245d697e1d5c/merged major:0 minor:148 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/1d9aa647e2373059999bced69e46860ded4acc9c83566cf2acbf53623ae905c9/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/29963cc72c43ebba1eb3f8b39ef6527e22b672f8c1850825d0c52d7b8ac6c1b4/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/6c8e806c89359c2295ea5e0a09d64a1e8aa1e7c15b440437c4f7701d80f3c529/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/d88776b4fbdb0842f8c9fb9b36fd83a49eb3b4ece1fff2e5d921dd713fb13c9a/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/108afffca84c4f79efceb9e5771638818d878f23dc9d992b0e658e18e242b830/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/a2759211cf74070b51b886a4b99a55c34ec5f808f2f16ce2a72741110360b0fa/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/1fdbf3dbea9d7a77ea1c8d316d8629ed4fc558af24a1c345046c2949948a73f7/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/806a12b37143946d6cb1026eab4a2f9d43c3ef1775d5a535772d8e1f0d98a911/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-208:{mountpoint:/var/lib/containers/storage/overlay/ef3b38422c889589ffe44844345401eaf7a742c2deb432823071458f9a2921fd/merged major:0 minor:208 fsType:overlay blockSize:0} overlay_0-210:{mountpoint:/var/lib/containers/storage/overlay/29733b966514f504c1c91f5cf03d55d8d01d790f05607ded257eb260a09698d5/merged major:0 minor:210 fsType:overlay blockSize:0} overlay_0-212:{mountpoint:/var/lib/containers/storage/overlay/c9a07fafa28ca5136190abe2d9195747f81ecef0e8975723860e2b36a461dd12/merged major:0 minor:212 fsType:overlay blockSize:0} overlay_0-214:{mountpoint:/var/lib/containers/storage/overlay/14cc1fa63a0c11badb74d497220718e6da0697ee0aaf5168663e58723507cadc/merged major:0 minor:214 fsType:overlay blockSize:0} overlay_0-216:{mountpoint:/var/lib/containers/storage/overlay/6fea8ff004e33b290700d764375d85571c70520f8f41b6ded7b50753bce3d718/merged major:0 minor:216 fsType:overlay blockSize:0} overlay_0-222:{mountpoint:/var/lib/containers/storage/overlay/a227aeefef48932b780d4aa04fb0bc9039e5f4a8aa28ac24cc479fc7412100f3/merged major:0 minor:222 fsType:overlay blockSize:0} overlay_0-227:{mountpoint:/var/lib/containers/storage/overlay/1d0ff86a3c9f518de742359c45ba83ff6c6b3fb5d3d351ffd1db9449f0654570/merged major:0 minor:227 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/db0ab48f7932765be1b0f9e88726cad02f54573fe59960dea8dd9ee8e41afe8a/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/013d19b410b03b5ea11212c013e77183d293d4835e0950d7b9be68b48da5bc2a/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/377c0726d1a44696257e0b94aed41b27eb713aa2f1cadd974bf5e7894dfd38b0/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/325d90a27d5fb3ec099c9aa6b72d8f7744476f255e42ad97ee1617adfd90ca5c/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/2b05d60fc633b2d027220251440861df9dee2583f89bcf70e9bc86b8ad30f87c/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/9ed9fbbf66e682d08b289be7a34b20ab072b3023f58c557a0d31ffe04dd5bf6f/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/a07fab6c22f88c51ede4c0720e2a9f4ec8e4cf308a29764d73593361db9f43ce/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/37203866fb11b55c5cf3db328445914a44a593d1c29e1582e0abdb7302424d78/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/2a3ca58359e11fe2c932f5a5f9188f8903769115feb3e932d210a5958867f141/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/bde2087469b2c3b7abe9bc70d4d6da63cf5bb0c33c248285fab44e5dee0b9b76/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 12:51:40.928489 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.919734    4364 manager.go:217] Machine: {Timestamp:2026-10-30 12:51:40.919037377 +0000 UTC m=+0.085402016 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097997 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2025572fc46201b88d8a24bd390f6e SystemUUID:ec202557-2fc4-6201-b88d-8a24bd390f6e BootID:6b0b2603-de76-426d-a575-92fc5ddc5aed Filesystems:[{Device:/var/lib/kubelet/pods/7d4b7525-1c1b-4e65-b263-349ab1a0f32f/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/2d5efe23-0e0d-4a4e-9d03-42829f371c09/volumes/kubernetes.io~projected/kube-api-access-x5s5m DeviceMajor:0 DeviceMinor:62 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/19115051-3b70-47cd-a7a3-313b1b3c26c7/volumes/kubernetes.io~projected/kube-api-access-78v2l DeviceMajor:0 DeviceMinor:67 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/75919339ef0b9c35042d1e42d475595815f0bec1ff4bfee178334d65fe6c9e99/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-222 DeviceMajor:0 DeviceMinor:222 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5db4412bd71a6d83812f0301b973336ae78cdeadaac587d366503762741963b7/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9fb14d5547656f78b7318c2bab087962d514662e903b402003c3830713792fe5/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bd7b201071dfe2916e1f9dbad684442224542e09ff380e06f7afa30ff071bc6f/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-212 DeviceMajor:0 DeviceMinor:212 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/febe2ba051c1b55bf7ad7e46d5651b755cb0e91501337c63e0d74aea981ca5dd/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/31d09b91-c973-4cf9-a7d8-d602ff9994f6/volumes/kubernetes.io~projected/kube-api-access-zdplx DeviceMajor:0 DeviceMinor:65 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c1927296-d775-43c4-a874-8842ebfeb3c8/volumes/kubernetes.io~projected/kube-api-access-jfwjf DeviceMajor:0 DeviceMinor:197 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2d5efe23-0e0d-4a4e-9d03-42829f371c09/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/69eab030-f488-433c-869a-52d628824391/volumes/kubernetes.io~projected/kube-api-access-hxq79 DeviceMajor:0 DeviceMinor:60 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ff0a920b-f15f-4909-906b-61dc41ef1263/volumes/kubernetes.io~projected/kube-api-access-q6s7g DeviceMajor:0 DeviceMinor:61 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/85ca31ad3a144e45ff7a5c955c9f421b21596c0a6dbf03d3496fab6c5afeb18a/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ba1b5a455d610b60c5d2e4e14e0168f6a3555dfacf0ec593869f1489f8c588d1/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c83bcf7b-a1d6-457b-9d7c-bb03c2cbd449/volumes/kubernetes.io~projected/kube-api-access-ls6hw DeviceMajor:0 DeviceMinor:64 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/841c7315696222e35d4b041031582f10b1b24393ed6535120d67e8c0f88c6f4a/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-145 DeviceMajor:0 DeviceMinor:145 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/815986e1-07b4-43fb-95f5-745316bd5105/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-227 DeviceMajor:0 DeviceMinor:227 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a1ca584f-61b1-4398-a0f6-5788e476aa20/volumes/kubernetes.io~projected/kube-api-access-7pppc DeviceMajor:0 DeviceMinor:63 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/209dbc8895cd9800cf2706ae099c9d409f916b6439290d9964aab249e7d3dd51/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/85602130-14de-443d-b237-b213c0ce8b82/volumes/kubernetes.io~projected/kube-api-access-749nf DeviceMajor:0 DeviceMinor:66 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2d5efe23-0e0d-4a4e-9d03-42829f371c09/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b3b33c2d5d904bcf841949f7bfee7e3ab8964d94f48833ca1f8fec18652e9be2/userdata/shm DeviceMajor:0 DeviceMinor:206 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-208 DeviceMajor:0 DeviceMinor:208 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-214 DeviceMajor:0 DeviceMinor:214 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/85602130-14de-443d-b237-b213c0ce8b82/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c883f63b50ff78fea1cbad9c51a48d375c9dab138bf49c1e08b0a71211be2a8e/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a103dec3bf04fc7374a948e56580a14754926df6a11fdc6fa78b06ab10452ed4/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/69eab030-f488-433c-869a-52d628824391/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/85602130-14de-443d-b237-b213c0ce8b82/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:58 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f8f7d922-7364-444f-a932-278628e26b83/volumes/kubernetes.io~projected/kube-api-access-wtppn DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/73cbd7cb7126e4b9be61ec1ed7fbfa40e963518f310672b2b89d204b8ab80b97/userdata/shm DeviceMajor:0 DeviceMinor:204 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-210 DeviceMajor:0 DeviceMinor:210 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-216 DeviceMajor:0 DeviceMinor:216 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-148 DeviceMajor:0 DeviceMinor:148 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:73cbd7cb7126e4b MacAddress:de:d0:c6:93:e8:4c Speed:10000 Mtu:8901} {Name:b3b33c2d5d904bc MacAddress:22:f7:2b:e2:fd:5b Speed:10000 Mtu:8901} {Name:bd7b201071dfe29 MacAddress:ba:21:aa:98:71:82 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:f6:ed:f9:5b Speed:0 Mtu:9001} {Name:br-int MacAddress:7e:f0:aa:7c:ac:04 Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:ff:f6:ed:f9:5b Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:7a:93:b4:e7:53:f6 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:fa:63:eb:6b:b2:87 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 12:51:40.928489 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.919968    4364 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 12:51:40.928489 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.920010    4364 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 12:51:40.928489 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.920265    4364 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 12:51:40.928954 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.920290    4364 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-6.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 12:51:40.928954 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.920401    4364 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 12:51:40.928954 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.920408    4364 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 12:51:40.928954 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.920418    4364 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 12:51:40.928954 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.920430    4364 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 12:51:40.928954 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.920515    4364 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 12:51:40.928954 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.920674    4364 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 12:51:40.928954 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.920764    4364 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 12:51:40.928954 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.920794    4364 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 12:51:40.928954 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.920803    4364 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 12:51:40.928954 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.920813    4364 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 12:51:40.928954 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.920822    4364 kubelet.go:405] "Adding apiserver pod source"
Oct 30 12:51:40.928954 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.920829    4364 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 12:51:40.928954 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.921478    4364 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 12:51:40.928954 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.921490    4364 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 12:51:40.928954 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.921970    4364 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 12:51:40.928954 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.924615    4364 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 12:51:40.928954 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.924628    4364 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 12:51:40.928954 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.924740    4364 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 12:51:40.928954 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.924990    4364 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 12:51:40.928954 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.925034    4364 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 12:51:40.928954 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.925146    4364 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 12:51:40.928954 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.925165    4364 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 12:51:40.928954 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.925174    4364 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 12:51:40.928954 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.925183    4364 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 12:51:40.928954 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.925193    4364 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 12:51:40.928954 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.925205    4364 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 12:51:40.928954 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.925214    4364 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 12:51:40.928954 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.925223    4364 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 12:51:40.928954 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.925234    4364 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 12:51:40.928954 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.925244    4364 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 12:51:40.928954 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.925267    4364 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 12:51:40.928954 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.925281    4364 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 12:51:40.928954 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.925314    4364 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 12:51:40.928954 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.925323    4364 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 12:51:40.930489 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.929393    4364 server.go:1263] "Started kubelet"
Oct 30 12:51:40.930489 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.929449    4364 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 12:51:40.930489 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.929559    4364 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 12:51:40.930489 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.929650    4364 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 12:51:40.930489 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.929979    4364 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 12:51:40.930489 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.930490    4364 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 12:51:40.929948 ip-10-0-137-6 systemd[1]: Started Kubernetes Kubelet.
Oct 30 12:51:40.932554 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.932536    4364 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 12:51:40.932875 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.932643    4364 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 12:51:40.932875 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.932654    4364 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 12:51:40.932875 ip-10-0-137-6 kubenswrapper[4364]: E1030 12:51:40.932704    4364 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-6.ec2.internal\" not found"
Oct 30 12:51:40.932875 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.932754    4364 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 12:51:40.933560 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.933521    4364 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 12:51:40.933871 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.933819    4364 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 12:46:03 +0000 UTC" deadline="2028-05-31 22:38:46.7601055 +0000 UTC"
Oct 30 12:51:40.933871 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.933857    4364 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13905h47m5.826252213s"
Oct 30 12:51:40.936102 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.935242    4364 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85602130-14de-443d-b237-b213c0ce8b82" volumeName="kubernetes.io/empty-dir/85602130-14de-443d-b237-b213c0ce8b82-etc-tuned" seLinuxMountContext=""
Oct 30 12:51:40.936102 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.935288    4364 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85602130-14de-443d-b237-b213c0ce8b82" volumeName="kubernetes.io/empty-dir/85602130-14de-443d-b237-b213c0ce8b82-tmp" seLinuxMountContext=""
Oct 30 12:51:40.936102 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.935304    4364 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85602130-14de-443d-b237-b213c0ce8b82" volumeName="kubernetes.io/projected/85602130-14de-443d-b237-b213c0ce8b82-kube-api-access-749nf" seLinuxMountContext=""
Oct 30 12:51:40.936102 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.935325    4364 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1ca584f-61b1-4398-a0f6-5788e476aa20" volumeName="kubernetes.io/configmap/a1ca584f-61b1-4398-a0f6-5788e476aa20-cni-binary-copy" seLinuxMountContext=""
Oct 30 12:51:40.936102 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.935338    4364 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1ca584f-61b1-4398-a0f6-5788e476aa20" volumeName="kubernetes.io/configmap/a1ca584f-61b1-4398-a0f6-5788e476aa20-multus-daemon-config" seLinuxMountContext=""
Oct 30 12:51:40.936102 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.935354    4364 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1927296-d775-43c4-a874-8842ebfeb3c8" volumeName="kubernetes.io/projected/c1927296-d775-43c4-a874-8842ebfeb3c8-kube-api-access-jfwjf" seLinuxMountContext=""
Oct 30 12:51:40.936102 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.935366    4364 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c83bcf7b-a1d6-457b-9d7c-bb03c2cbd449" volumeName="kubernetes.io/projected/c83bcf7b-a1d6-457b-9d7c-bb03c2cbd449-kube-api-access-ls6hw" seLinuxMountContext=""
Oct 30 12:51:40.936102 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.935379    4364 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff0a920b-f15f-4909-906b-61dc41ef1263" volumeName="kubernetes.io/projected/ff0a920b-f15f-4909-906b-61dc41ef1263-kube-api-access-q6s7g" seLinuxMountContext=""
Oct 30 12:51:40.936102 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.935398    4364 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d5efe23-0e0d-4a4e-9d03-42829f371c09" volumeName="kubernetes.io/configmap/2d5efe23-0e0d-4a4e-9d03-42829f371c09-ovnkube-config" seLinuxMountContext=""
Oct 30 12:51:40.936102 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.935413    4364 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d5efe23-0e0d-4a4e-9d03-42829f371c09" volumeName="kubernetes.io/secret/2d5efe23-0e0d-4a4e-9d03-42829f371c09-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 12:51:40.936102 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.935430    4364 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="69eab030-f488-433c-869a-52d628824391" volumeName="kubernetes.io/projected/69eab030-f488-433c-869a-52d628824391-kube-api-access-hxq79" seLinuxMountContext=""
Oct 30 12:51:40.936102 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.935442    4364 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="69eab030-f488-433c-869a-52d628824391" volumeName="kubernetes.io/secret/69eab030-f488-433c-869a-52d628824391-metrics-certs" seLinuxMountContext=""
Oct 30 12:51:40.936102 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.935458    4364 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff0a920b-f15f-4909-906b-61dc41ef1263" volumeName="kubernetes.io/configmap/ff0a920b-f15f-4909-906b-61dc41ef1263-cni-binary-copy" seLinuxMountContext=""
Oct 30 12:51:40.936102 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.935469    4364 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff0a920b-f15f-4909-906b-61dc41ef1263" volumeName="kubernetes.io/configmap/ff0a920b-f15f-4909-906b-61dc41ef1263-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 12:51:40.936102 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.935491    4364 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff0a920b-f15f-4909-906b-61dc41ef1263" volumeName="kubernetes.io/configmap/ff0a920b-f15f-4909-906b-61dc41ef1263-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 12:51:40.936102 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.935521    4364 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19115051-3b70-47cd-a7a3-313b1b3c26c7" volumeName="kubernetes.io/configmap/19115051-3b70-47cd-a7a3-313b1b3c26c7-iptables-alerter-script" seLinuxMountContext=""
Oct 30 12:51:40.936102 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.935533    4364 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d5efe23-0e0d-4a4e-9d03-42829f371c09" volumeName="kubernetes.io/configmap/2d5efe23-0e0d-4a4e-9d03-42829f371c09-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 12:51:40.936102 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.935551    4364 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d5efe23-0e0d-4a4e-9d03-42829f371c09" volumeName="kubernetes.io/projected/2d5efe23-0e0d-4a4e-9d03-42829f371c09-kube-api-access-x5s5m" seLinuxMountContext=""
Oct 30 12:51:40.936102 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.935563    4364 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d4b7525-1c1b-4e65-b263-349ab1a0f32f" volumeName="kubernetes.io/configmap/7d4b7525-1c1b-4e65-b263-349ab1a0f32f-konnectivity-ca" seLinuxMountContext=""
Oct 30 12:51:40.936102 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.935581    4364 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="815986e1-07b4-43fb-95f5-745316bd5105" volumeName="kubernetes.io/secret/815986e1-07b4-43fb-95f5-745316bd5105-original-pull-secret" seLinuxMountContext=""
Oct 30 12:51:40.936102 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.935594    4364 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1ca584f-61b1-4398-a0f6-5788e476aa20" volumeName="kubernetes.io/projected/a1ca584f-61b1-4398-a0f6-5788e476aa20-kube-api-access-7pppc" seLinuxMountContext=""
Oct 30 12:51:40.936102 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.935628    4364 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c83bcf7b-a1d6-457b-9d7c-bb03c2cbd449" volumeName="kubernetes.io/empty-dir/c83bcf7b-a1d6-457b-9d7c-bb03c2cbd449-tmp-dir" seLinuxMountContext=""
Oct 30 12:51:40.936102 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.935644    4364 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8f7d922-7364-444f-a932-278628e26b83" volumeName="kubernetes.io/projected/f8f7d922-7364-444f-a932-278628e26b83-kube-api-access-wtppn" seLinuxMountContext=""
Oct 30 12:51:40.936102 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.935656    4364 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19115051-3b70-47cd-a7a3-313b1b3c26c7" volumeName="kubernetes.io/projected/19115051-3b70-47cd-a7a3-313b1b3c26c7-kube-api-access-78v2l" seLinuxMountContext=""
Oct 30 12:51:40.936102 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.935673    4364 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d5efe23-0e0d-4a4e-9d03-42829f371c09" volumeName="kubernetes.io/configmap/2d5efe23-0e0d-4a4e-9d03-42829f371c09-env-overrides" seLinuxMountContext=""
Oct 30 12:51:40.936102 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.935684    4364 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31d09b91-c973-4cf9-a7d8-d602ff9994f6" volumeName="kubernetes.io/configmap/31d09b91-c973-4cf9-a7d8-d602ff9994f6-serviceca" seLinuxMountContext=""
Oct 30 12:51:40.936102 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.935696    4364 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31d09b91-c973-4cf9-a7d8-d602ff9994f6" volumeName="kubernetes.io/projected/31d09b91-c973-4cf9-a7d8-d602ff9994f6-kube-api-access-zdplx" seLinuxMountContext=""
Oct 30 12:51:40.936102 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.935713    4364 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d4b7525-1c1b-4e65-b263-349ab1a0f32f" volumeName="kubernetes.io/secret/7d4b7525-1c1b-4e65-b263-349ab1a0f32f-agent-certs" seLinuxMountContext=""
Oct 30 12:51:40.936102 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.935724    4364 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 12:51:40.936102 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.935732    4364 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 12:51:40.937515 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.936242    4364 factory.go:55] Registering systemd factory
Oct 30 12:51:40.937515 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.936258    4364 factory.go:223] Registration of the systemd container factory successfully
Oct 30 12:51:40.938221 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.938194    4364 factory.go:153] Registering CRI-O factory
Oct 30 12:51:40.938221 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.938218    4364 factory.go:223] Registration of the crio container factory successfully
Oct 30 12:51:40.938416 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.938277    4364 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 12:51:40.938416 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.938307    4364 factory.go:103] Registering Raw factory
Oct 30 12:51:40.938416 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.938321    4364 manager.go:1199] Started watching for new ooms in manager
Oct 30 12:51:40.939697 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.938680    4364 manager.go:319] Starting recovery of all containers
Oct 30 12:51:40.960511 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.960481    4364 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 12:51:40.961788 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.961763    4364 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 12:51:40.961883 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.961792    4364 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 12:51:40.961883 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.961815    4364 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 12:51:40.961883 ip-10-0-137-6 kubenswrapper[4364]: E1030 12:51:40.961857    4364 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 12:51:40.972550 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.972524    4364 generic.go:356] "Generic (PLEG): container finished" podID="ff0a920b-f15f-4909-906b-61dc41ef1263" containerID="6c53c33c80386939f77a0ac68b599495904d00ddc0278dcbcab907762add20ec" exitCode=0
Oct 30 12:51:40.972550 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.972546    4364 generic.go:356] "Generic (PLEG): container finished" podID="ff0a920b-f15f-4909-906b-61dc41ef1263" containerID="7479cb8d1471ab3843adde80a4ac73829345529998d131dac8adfb13e35c3727" exitCode=0
Oct 30 12:51:40.972550 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.972552    4364 generic.go:356] "Generic (PLEG): container finished" podID="ff0a920b-f15f-4909-906b-61dc41ef1263" containerID="36001a7ab4546a8df2990deede34a0e2a35301e8b6b4074c12789d317f458a8e" exitCode=0
Oct 30 12:51:40.972754 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.972559    4364 generic.go:356] "Generic (PLEG): container finished" podID="ff0a920b-f15f-4909-906b-61dc41ef1263" containerID="e32a756ce46dd838ddf59ae449ec16cdd4f3115e33e6dd4fb233c799f38767c2" exitCode=0
Oct 30 12:51:40.972754 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.972565    4364 generic.go:356] "Generic (PLEG): container finished" podID="ff0a920b-f15f-4909-906b-61dc41ef1263" containerID="653c1ffb461d29d061b2ee16375f59f04326a0766b344ff97555588878542557" exitCode=0
Oct 30 12:51:40.972754 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.972571    4364 generic.go:356] "Generic (PLEG): container finished" podID="ff0a920b-f15f-4909-906b-61dc41ef1263" containerID="bd4b50582c4018c35ed971ec56705d65ece4c78b00abc6b590db3ec24bdd4eaf" exitCode=0
Oct 30 12:51:40.989293 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.989272    4364 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 12:51:40.992236 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.992209    4364 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 12:51:40.992715 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.992692    4364 generic.go:356] "Generic (PLEG): container finished" podID="a60e17e4a3a53aed12f9737e4f90e05b" containerID="7dfc08fd708e7f6ca6f640380cce379aee75b71617a39e48cd64aa11a308153d" exitCode=0
Oct 30 12:51:40.996164 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:40.996148    4364 manager.go:324] Recovery completed
Oct 30 12:51:41.003093 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:41.003072    4364 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 12:51:41.003176 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:41.003131    4364 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 12:51:41.006236 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:41.006214    4364 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 12:51:41.009524 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:41.009510    4364 cpu_manager.go:226] "Starting" policy="none"
Oct 30 12:51:41.009524 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:41.009521    4364 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 12:51:41.009687 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:41.009540    4364 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 12:51:41.009746 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:41.009696    4364 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 12:51:41.009746 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:41.009711    4364 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 12:51:41.009746 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:41.009734    4364 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 12:51:41.009746 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:41.009742    4364 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 12:51:41.009920 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:41.009749    4364 policy_none.go:50] "Start"
Oct 30 12:51:41.009920 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:41.009761    4364 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 12:51:41.009920 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:41.009774    4364 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 12:51:41.010057 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:41.009959    4364 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 12:51:41.010057 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:41.009969    4364 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 12:51:41.010057 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:41.009977    4364 policy_none.go:44] "Start"
Oct 30 12:51:41.013809 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:41.013784    4364 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 12:51:41.016402 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:41.016389    4364 manager.go:341] "Starting Device Plugin manager"
Oct 30 12:51:41.016849 ip-10-0-137-6 kubenswrapper[4364]: E1030 12:51:41.016414    4364 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 12:51:41.016936 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:41.016851    4364 server.go:87] "Starting device plugin registration server"
Oct 30 12:51:41.017163 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:41.017150    4364 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 12:51:41.017247 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:41.017165    4364 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 12:51:41.017299 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:41.017279    4364 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 12:51:41.017400 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:41.017368    4364 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 12:51:41.017400 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:41.017401    4364 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 12:51:41.017535 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:41.017407    4364 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 12:51:41.017535 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:41.017452    4364 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T12:51:41.017391846Z","UUID":"850d8c4e-9e37-42fb-903d-1270d266cf4c","Handler":null,"Name":"","Endpoint":""}
Oct 30 12:51:41.018448 ip-10-0-137-6 kubenswrapper[4364]: E1030 12:51:41.018427    4364 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 12:51:41.018863 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:41.018838    4364 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 12:51:41.018939 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:41.018873    4364 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 12:51:41.062851 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:41.062785    4364 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-6.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-6.ec2.internal"]
Oct 30 12:51:41.063278 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:41.063243    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-6.ec2.internal" event={"ID":"c8cfc7a7881c11ddfae618f888b1edd8","Type":"ContainerStarted","Data":"b5ddeb7249724e94c8420e2cd96674ca97900fa8dfcc1c7608aafe9fbf0dcfde"}
Oct 30 12:51:41.063347 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:41.063290    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-6.ec2.internal" event={"ID":"c8cfc7a7881c11ddfae618f888b1edd8","Type":"ContainerStarted","Data":"febe2ba051c1b55bf7ad7e46d5651b755cb0e91501337c63e0d74aea981ca5dd"}
Oct 30 12:51:41.063347 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:41.063322    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-6.ec2.internal" event={"ID":"a60e17e4a3a53aed12f9737e4f90e05b","Type":"ContainerStarted","Data":"e0a83b472202e398c74d83049d01d00b900d7981a76198dccf8dd5fdaa7b5caa"}
Oct 30 12:51:41.063347 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:41.063333    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-6.ec2.internal" event={"ID":"a60e17e4a3a53aed12f9737e4f90e05b","Type":"ContainerDied","Data":"7dfc08fd708e7f6ca6f640380cce379aee75b71617a39e48cd64aa11a308153d"}
Oct 30 12:51:41.063347 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:41.063344    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-6.ec2.internal" event={"ID":"a60e17e4a3a53aed12f9737e4f90e05b","Type":"ContainerStarted","Data":"9fb14d5547656f78b7318c2bab087962d514662e903b402003c3830713792fe5"}
Oct 30 12:51:41.063514 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:41.063417    4364 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-6.ec2.internal"
Oct 30 12:51:41.063514 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:41.063429    4364 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-6.ec2.internal"
Oct 30 12:51:41.091572 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:41.091554    4364 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 12:51:41.091661 ip-10-0-137-6 kubenswrapper[4364]: E1030 12:51:41.091618    4364 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-137-6.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-6.ec2.internal"
Oct 30 12:51:41.095423 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:41.095409    4364 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 12:51:41.095470 ip-10-0-137-6 kubenswrapper[4364]: E1030 12:51:41.095440    4364 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-137-6.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-6.ec2.internal"
Oct 30 12:51:41.118218 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:41.118200    4364 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 12:51:41.119332 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:41.119310    4364 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-6.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 12:51:41.119424 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:41.119343    4364 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-6.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 12:51:41.119424 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:41.119359    4364 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-6.ec2.internal" event="NodeHasSufficientPID"
Oct 30 12:51:41.119424 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:41.119391    4364 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-6.ec2.internal"
Oct 30 12:51:41.160889 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:41.160868    4364 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-137-6.ec2.internal"
Oct 30 12:51:41.160955 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:41.160913    4364 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-137-6.ec2.internal"
Oct 30 12:51:41.189883 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:41.189861    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c8cfc7a7881c11ddfae618f888b1edd8-config\") pod \"kube-apiserver-proxy-ip-10-0-137-6.ec2.internal\" (UID: \"c8cfc7a7881c11ddfae618f888b1edd8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-6.ec2.internal"
Oct 30 12:51:41.189965 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:41.189885    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a60e17e4a3a53aed12f9737e4f90e05b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-6.ec2.internal\" (UID: \"a60e17e4a3a53aed12f9737e4f90e05b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-6.ec2.internal"
Oct 30 12:51:41.189965 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:41.189902    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a60e17e4a3a53aed12f9737e4f90e05b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-6.ec2.internal\" (UID: \"a60e17e4a3a53aed12f9737e4f90e05b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-6.ec2.internal"
Oct 30 12:51:41.290880 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:41.290851    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a60e17e4a3a53aed12f9737e4f90e05b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-6.ec2.internal\" (UID: \"a60e17e4a3a53aed12f9737e4f90e05b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-6.ec2.internal"
Oct 30 12:51:41.290948 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:41.290886    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a60e17e4a3a53aed12f9737e4f90e05b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-6.ec2.internal\" (UID: \"a60e17e4a3a53aed12f9737e4f90e05b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-6.ec2.internal"
Oct 30 12:51:41.290982 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:41.290949    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a60e17e4a3a53aed12f9737e4f90e05b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-6.ec2.internal\" (UID: \"a60e17e4a3a53aed12f9737e4f90e05b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-6.ec2.internal"
Oct 30 12:51:41.291023 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:41.290983    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c8cfc7a7881c11ddfae618f888b1edd8-config\") pod \"kube-apiserver-proxy-ip-10-0-137-6.ec2.internal\" (UID: \"c8cfc7a7881c11ddfae618f888b1edd8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-6.ec2.internal"
Oct 30 12:51:41.291023 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:41.290952    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c8cfc7a7881c11ddfae618f888b1edd8-config\") pod \"kube-apiserver-proxy-ip-10-0-137-6.ec2.internal\" (UID: \"c8cfc7a7881c11ddfae618f888b1edd8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-6.ec2.internal"
Oct 30 12:51:41.291023 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:41.290926    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a60e17e4a3a53aed12f9737e4f90e05b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-6.ec2.internal\" (UID: \"a60e17e4a3a53aed12f9737e4f90e05b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-6.ec2.internal"
Oct 30 12:51:41.921702 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:41.921681    4364 apiserver.go:52] "Watching apiserver"
Oct 30 12:51:41.994774 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:41.994751    4364 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-6.ec2.internal"
Oct 30 12:51:41.994904 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:41.994889    4364 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-6.ec2.internal"
Oct 30 12:51:42.014824 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.014799    4364 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 12:51:42.015139 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.015120    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-dlxch","kube-system/konnectivity-agent-vlfkx","kube-system/kube-apiserver-proxy-ip-10-0-137-6.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5mdl","openshift-multus/network-metrics-daemon-79wgd","openshift-network-operator/iptables-alerter-9p6w9","openshift-ovn-kubernetes/ovnkube-node-wpkrj","openshift-dns/node-resolver-7mvnz","kube-system/global-pull-secret-syncer-rxg7f","openshift-cluster-node-tuning-operator/tuned-7r6q9","openshift-image-registry/node-ca-7pxn6","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-6.ec2.internal","openshift-multus/multus-additional-cni-plugins-zvkh9","openshift-multus/multus-mv5cv"]
Oct 30 12:51:42.017214 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.017197    4364 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 12:51:42.017306 ip-10-0-137-6 kubenswrapper[4364]: E1030 12:51:42.017240    4364 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-137-6.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-6.ec2.internal"
Oct 30 12:51:42.025978 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.025957    4364 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 12:51:42.026074 ip-10-0-137-6 kubenswrapper[4364]: E1030 12:51:42.026003    4364 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-137-6.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-6.ec2.internal"
Oct 30 12:51:42.032290 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.032265    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-d8ng4\""
Oct 30 12:51:42.032372 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.032344    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pdl9w\""
Oct 30 12:51:42.033132 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.033119    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 12:51:42.033701 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.033680    4364 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 12:51:42.039951 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.039927    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 12:51:42.039951 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.039928    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 12:51:42.040103 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.039926    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 12:51:42.040103 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.040006    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 12:51:42.040103 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.040007    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 12:51:42.040103 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.040027    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 12:51:42.040103 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.040045    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 12:51:42.040277 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.040108    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 12:51:42.048662 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.048641    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 12:51:42.048735 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.048722    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 12:51:42.049722 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.049704    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 12:51:42.049796 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.049784    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 12:51:42.049846 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.049824    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 12:51:42.049846 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.049826    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 12:51:42.049932 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.049866    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 12:51:42.049980 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.049943    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 12:51:42.050062 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.050050    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-fv4k6\""
Oct 30 12:51:42.050062 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.050056    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-rnjq7\""
Oct 30 12:51:42.050512 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.050495    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 12:51:42.050826 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.050812    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 12:51:42.051283 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.051267    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 12:51:42.051357 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.051301    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 12:51:42.051357 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.051306    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 12:51:42.051451 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.051440    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 12:51:42.051501 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.051440    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 12:51:42.051501 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.051460    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 12:51:42.051501 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.051468    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 12:51:42.051501 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.051484    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 12:51:42.051668 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.051503    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 12:51:42.051987 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.051971    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-78z4l\""
Oct 30 12:51:42.052054 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.052015    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 12:51:42.052108 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.052093    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-g6csn\""
Oct 30 12:51:42.052223 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.052211    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-nklv7\""
Oct 30 12:51:42.052255 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.052244    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-sb6s2\""
Oct 30 12:51:42.052363 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.052350    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-hfgh7\""
Oct 30 12:51:42.052452 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.052440    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-n54s8\""
Oct 30 12:51:42.052488 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.052474    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-588rd\""
Oct 30 12:51:42.053671 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.053657    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9fz8p\""
Oct 30 12:51:42.055322 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.055308    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 12:51:42.070461 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.070444    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 12:51:42.094709 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.094683    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-host-var-lib-cni-multus\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:42.094790 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.094720    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/19115051-3b70-47cd-a7a3-313b1b3c26c7-iptables-alerter-script\") pod \"iptables-alerter-9p6w9\" (UID: \"19115051-3b70-47cd-a7a3-313b1b3c26c7\") " pod="openshift-network-operator/iptables-alerter-9p6w9"
Oct 30 12:51:42.094790 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.094779    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85602130-14de-443d-b237-b213c0ce8b82-etc-kubernetes\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:42.094895 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.094815    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtppn\" (UniqueName: \"kubernetes.io/projected/f8f7d922-7364-444f-a932-278628e26b83-kube-api-access-wtppn\") pod \"aws-ebs-csi-driver-node-p5mdl\" (UID: \"f8f7d922-7364-444f-a932-278628e26b83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5mdl"
Oct 30 12:51:42.094895 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.094846    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ff0a920b-f15f-4909-906b-61dc41ef1263-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zvkh9\" (UID: \"ff0a920b-f15f-4909-906b-61dc41ef1263\") " pod="openshift-multus/multus-additional-cni-plugins-zvkh9"
Oct 30 12:51:42.094987 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.094902    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6s7g\" (UniqueName: \"kubernetes.io/projected/ff0a920b-f15f-4909-906b-61dc41ef1263-kube-api-access-q6s7g\") pod \"multus-additional-cni-plugins-zvkh9\" (UID: \"ff0a920b-f15f-4909-906b-61dc41ef1263\") " pod="openshift-multus/multus-additional-cni-plugins-zvkh9"
Oct 30 12:51:42.094987 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.094978    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-system-cni-dir\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:42.095068 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.095010    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-host-slash\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:42.095114 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.095085    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-log-socket\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:42.095172 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.095121    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-host-run-ovn-kubernetes\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:42.095223 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.095164    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-host-cni-netd\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:42.095223 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.095203    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2d5efe23-0e0d-4a4e-9d03-42829f371c09-ovn-node-metrics-cert\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:42.095310 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.095242    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/19115051-3b70-47cd-a7a3-313b1b3c26c7-iptables-alerter-script\") pod \"iptables-alerter-9p6w9\" (UID: \"19115051-3b70-47cd-a7a3-313b1b3c26c7\") " pod="openshift-network-operator/iptables-alerter-9p6w9"
Oct 30 12:51:42.095379 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.095361    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2d5efe23-0e0d-4a4e-9d03-42829f371c09-ovnkube-script-lib\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:42.095446 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.095425    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a1ca584f-61b1-4398-a0f6-5788e476aa20-cni-binary-copy\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:42.095478 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.095462    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/85602130-14de-443d-b237-b213c0ce8b82-etc-sysctl-conf\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:42.095577 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.095558    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7d4b7525-1c1b-4e65-b263-349ab1a0f32f-agent-certs\") pod \"konnectivity-agent-vlfkx\" (UID: \"7d4b7525-1c1b-4e65-b263-349ab1a0f32f\") " pod="kube-system/konnectivity-agent-vlfkx"
Oct 30 12:51:42.095676 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.095588    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2d5efe23-0e0d-4a4e-9d03-42829f371c09-ovn-node-metrics-cert\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:42.095676 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.095617    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2d5efe23-0e0d-4a4e-9d03-42829f371c09-ovnkube-script-lib\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:42.095676 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.095616    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-multus-cni-dir\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:42.095811 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.095691    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78v2l\" (UniqueName: \"kubernetes.io/projected/19115051-3b70-47cd-a7a3-313b1b3c26c7-kube-api-access-78v2l\") pod \"iptables-alerter-9p6w9\" (UID: \"19115051-3b70-47cd-a7a3-313b1b3c26c7\") " pod="openshift-network-operator/iptables-alerter-9p6w9"
Oct 30 12:51:42.095811 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.095718    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/85602130-14de-443d-b237-b213c0ce8b82-etc-modprobe-d\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:42.095811 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.095744    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/85602130-14de-443d-b237-b213c0ce8b82-lib-modules\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:42.095811 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.095783    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/31d09b91-c973-4cf9-a7d8-d602ff9994f6-serviceca\") pod \"node-ca-7pxn6\" (UID: \"31d09b91-c973-4cf9-a7d8-d602ff9994f6\") " pod="openshift-image-registry/node-ca-7pxn6"
Oct 30 12:51:42.096004 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.095784    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a1ca584f-61b1-4398-a0f6-5788e476aa20-cni-binary-copy\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:42.096004 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.095814    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-host-kubelet\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:42.096004 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.095867    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f8f7d922-7364-444f-a932-278628e26b83-registration-dir\") pod \"aws-ebs-csi-driver-node-p5mdl\" (UID: \"f8f7d922-7364-444f-a932-278628e26b83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5mdl"
Oct 30 12:51:42.096004 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.095898    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-multus-socket-dir-parent\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:42.096004 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.095925    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-host-run-netns\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:42.096004 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.095964    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-multus-conf-dir\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:42.096259 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.096009    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7d4b7525-1c1b-4e65-b263-349ab1a0f32f-agent-certs\") pod \"konnectivity-agent-vlfkx\" (UID: \"7d4b7525-1c1b-4e65-b263-349ab1a0f32f\") " pod="kube-system/konnectivity-agent-vlfkx"
Oct 30 12:51:42.096259 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.096014    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7d4b7525-1c1b-4e65-b263-349ab1a0f32f-konnectivity-ca\") pod \"konnectivity-agent-vlfkx\" (UID: \"7d4b7525-1c1b-4e65-b263-349ab1a0f32f\") " pod="kube-system/konnectivity-agent-vlfkx"
Oct 30 12:51:42.096259 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.096054    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/85602130-14de-443d-b237-b213c0ce8b82-sys\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:42.096259 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.096126    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-749nf\" (UniqueName: \"kubernetes.io/projected/85602130-14de-443d-b237-b213c0ce8b82-kube-api-access-749nf\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:42.096259 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.096165    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-host-run-netns\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:42.096259 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.096193    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-run-openvswitch\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:42.096259 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.096200    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7d4b7525-1c1b-4e65-b263-349ab1a0f32f-konnectivity-ca\") pod \"konnectivity-agent-vlfkx\" (UID: \"7d4b7525-1c1b-4e65-b263-349ab1a0f32f\") " pod="kube-system/konnectivity-agent-vlfkx"
Oct 30 12:51:42.096259 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.096217    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-host-cni-bin\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:42.096259 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.096247    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/815986e1-07b4-43fb-95f5-745316bd5105-dbus\") pod \"global-pull-secret-syncer-rxg7f\" (UID: \"815986e1-07b4-43fb-95f5-745316bd5105\") " pod="kube-system/global-pull-secret-syncer-rxg7f"
Oct 30 12:51:42.096627 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.096277    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f8f7d922-7364-444f-a932-278628e26b83-device-dir\") pod \"aws-ebs-csi-driver-node-p5mdl\" (UID: \"f8f7d922-7364-444f-a932-278628e26b83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5mdl"
Oct 30 12:51:42.096627 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.096295    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f8f7d922-7364-444f-a932-278628e26b83-etc-selinux\") pod \"aws-ebs-csi-driver-node-p5mdl\" (UID: \"f8f7d922-7364-444f-a932-278628e26b83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5mdl"
Oct 30 12:51:42.096627 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.096299    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/31d09b91-c973-4cf9-a7d8-d602ff9994f6-serviceca\") pod \"node-ca-7pxn6\" (UID: \"31d09b91-c973-4cf9-a7d8-d602ff9994f6\") " pod="openshift-image-registry/node-ca-7pxn6"
Oct 30 12:51:42.096627 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.096334    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f8f7d922-7364-444f-a932-278628e26b83-sys-fs\") pod \"aws-ebs-csi-driver-node-p5mdl\" (UID: \"f8f7d922-7364-444f-a932-278628e26b83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5mdl"
Oct 30 12:51:42.096627 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.096363    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-var-lib-openvswitch\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:42.096627 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.096399    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-etc-openvswitch\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:42.096627 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.096444    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/85602130-14de-443d-b237-b213c0ce8b82-host\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:42.096627 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.096474    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/85602130-14de-443d-b237-b213c0ce8b82-etc-tuned\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:42.096627 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.096493    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/85602130-14de-443d-b237-b213c0ce8b82-etc-sysctl-d\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:42.096627 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.096507    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/85602130-14de-443d-b237-b213c0ce8b82-etc-systemd\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:42.096627 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.096542    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/85602130-14de-443d-b237-b213c0ce8b82-etc-tuned\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:42.096627 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.096562    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ff0a920b-f15f-4909-906b-61dc41ef1263-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zvkh9\" (UID: \"ff0a920b-f15f-4909-906b-61dc41ef1263\") " pod="openshift-multus/multus-additional-cni-plugins-zvkh9"
Oct 30 12:51:42.096627 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.096634    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-host-var-lib-cni-bin\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:42.097134 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.096652    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-host-run-multus-certs\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:42.097134 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.096667    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-run-ovn\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:42.097134 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.096681    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:42.097134 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.096699    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/815986e1-07b4-43fb-95f5-745316bd5105-original-pull-secret\") pod \"global-pull-secret-syncer-rxg7f\" (UID: \"815986e1-07b4-43fb-95f5-745316bd5105\") " pod="kube-system/global-pull-secret-syncer-rxg7f"
Oct 30 12:51:42.097134 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.096724    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/19115051-3b70-47cd-a7a3-313b1b3c26c7-host-slash\") pod \"iptables-alerter-9p6w9\" (UID: \"19115051-3b70-47cd-a7a3-313b1b3c26c7\") " pod="openshift-network-operator/iptables-alerter-9p6w9"
Oct 30 12:51:42.097134 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.096739    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfwjf\" (UniqueName: \"kubernetes.io/projected/c1927296-d775-43c4-a874-8842ebfeb3c8-kube-api-access-jfwjf\") pod \"network-check-target-dlxch\" (UID: \"c1927296-d775-43c4-a874-8842ebfeb3c8\") " pod="openshift-network-diagnostics/network-check-target-dlxch"
Oct 30 12:51:42.097134 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.096799    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2d5efe23-0e0d-4a4e-9d03-42829f371c09-env-overrides\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:42.097134 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.096838    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69eab030-f488-433c-869a-52d628824391-metrics-certs\") pod \"network-metrics-daemon-79wgd\" (UID: \"69eab030-f488-433c-869a-52d628824391\") " pod="openshift-multus/network-metrics-daemon-79wgd"
Oct 30 12:51:42.097134 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.096842    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ff0a920b-f15f-4909-906b-61dc41ef1263-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zvkh9\" (UID: \"ff0a920b-f15f-4909-906b-61dc41ef1263\") " pod="openshift-multus/multus-additional-cni-plugins-zvkh9"
Oct 30 12:51:42.097134 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.096954    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/815986e1-07b4-43fb-95f5-745316bd5105-original-pull-secret\") pod \"global-pull-secret-syncer-rxg7f\" (UID: \"815986e1-07b4-43fb-95f5-745316bd5105\") " pod="kube-system/global-pull-secret-syncer-rxg7f"
Oct 30 12:51:42.097134 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.096961    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ff0a920b-f15f-4909-906b-61dc41ef1263-cni-binary-copy\") pod \"multus-additional-cni-plugins-zvkh9\" (UID: \"ff0a920b-f15f-4909-906b-61dc41ef1263\") " pod="openshift-multus/multus-additional-cni-plugins-zvkh9"
Oct 30 12:51:42.097134 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.097009    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-host-run-k8s-cni-cncf-io\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:42.097134 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.097015    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69eab030-f488-433c-869a-52d628824391-metrics-certs\") pod \"network-metrics-daemon-79wgd\" (UID: \"69eab030-f488-433c-869a-52d628824391\") " pod="openshift-multus/network-metrics-daemon-79wgd"
Oct 30 12:51:42.097134 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.097029    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2d5efe23-0e0d-4a4e-9d03-42829f371c09-env-overrides\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:42.097134 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.097043    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-host-var-lib-kubelet\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:42.097134 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.097070    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pppc\" (UniqueName: \"kubernetes.io/projected/a1ca584f-61b1-4398-a0f6-5788e476aa20-kube-api-access-7pppc\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:42.097134 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.097095    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2d5efe23-0e0d-4a4e-9d03-42829f371c09-ovnkube-config\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:42.097134 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.097118    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ff0a920b-f15f-4909-906b-61dc41ef1263-system-cni-dir\") pod \"multus-additional-cni-plugins-zvkh9\" (UID: \"ff0a920b-f15f-4909-906b-61dc41ef1263\") " pod="openshift-multus/multus-additional-cni-plugins-zvkh9"
Oct 30 12:51:42.097134 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.097138    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ff0a920b-f15f-4909-906b-61dc41ef1263-cni-binary-copy\") pod \"multus-additional-cni-plugins-zvkh9\" (UID: \"ff0a920b-f15f-4909-906b-61dc41ef1263\") " pod="openshift-multus/multus-additional-cni-plugins-zvkh9"
Oct 30 12:51:42.097867 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.097207    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-cnibin\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:42.097867 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.097245    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-etc-kubernetes\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:42.097867 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.097277    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2d5efe23-0e0d-4a4e-9d03-42829f371c09-ovnkube-config\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:42.097867 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.097284    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5s5m\" (UniqueName: \"kubernetes.io/projected/2d5efe23-0e0d-4a4e-9d03-42829f371c09-kube-api-access-x5s5m\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:42.097867 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.097314    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c83bcf7b-a1d6-457b-9d7c-bb03c2cbd449-tmp-dir\") pod \"node-resolver-7mvnz\" (UID: \"c83bcf7b-a1d6-457b-9d7c-bb03c2cbd449\") " pod="openshift-dns/node-resolver-7mvnz"
Oct 30 12:51:42.097867 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.097399    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-hostroot\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:42.097867 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.097418    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a1ca584f-61b1-4398-a0f6-5788e476aa20-multus-daemon-config\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:42.097867 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.097459    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c83bcf7b-a1d6-457b-9d7c-bb03c2cbd449-hosts-file\") pod \"node-resolver-7mvnz\" (UID: \"c83bcf7b-a1d6-457b-9d7c-bb03c2cbd449\") " pod="openshift-dns/node-resolver-7mvnz"
Oct 30 12:51:42.097867 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.097471    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c83bcf7b-a1d6-457b-9d7c-bb03c2cbd449-tmp-dir\") pod \"node-resolver-7mvnz\" (UID: \"c83bcf7b-a1d6-457b-9d7c-bb03c2cbd449\") " pod="openshift-dns/node-resolver-7mvnz"
Oct 30 12:51:42.097867 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.097488    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-node-log\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:42.097867 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.097515    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxq79\" (UniqueName: \"kubernetes.io/projected/69eab030-f488-433c-869a-52d628824391-kube-api-access-hxq79\") pod \"network-metrics-daemon-79wgd\" (UID: \"69eab030-f488-433c-869a-52d628824391\") " pod="openshift-multus/network-metrics-daemon-79wgd"
Oct 30 12:51:42.097867 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.097551    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ff0a920b-f15f-4909-906b-61dc41ef1263-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zvkh9\" (UID: \"ff0a920b-f15f-4909-906b-61dc41ef1263\") " pod="openshift-multus/multus-additional-cni-plugins-zvkh9"
Oct 30 12:51:42.097867 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.097566    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a1ca584f-61b1-4398-a0f6-5788e476aa20-multus-daemon-config\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:42.097867 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.097590    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/815986e1-07b4-43fb-95f5-745316bd5105-kubelet-config\") pod \"global-pull-secret-syncer-rxg7f\" (UID: \"815986e1-07b4-43fb-95f5-745316bd5105\") " pod="kube-system/global-pull-secret-syncer-rxg7f"
Oct 30 12:51:42.097867 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.097630    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/85602130-14de-443d-b237-b213c0ce8b82-var-lib-kubelet\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:42.097867 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.097654    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85602130-14de-443d-b237-b213c0ce8b82-tmp\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:42.097867 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.097673    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/31d09b91-c973-4cf9-a7d8-d602ff9994f6-host\") pod \"node-ca-7pxn6\" (UID: \"31d09b91-c973-4cf9-a7d8-d602ff9994f6\") " pod="openshift-image-registry/node-ca-7pxn6"
Oct 30 12:51:42.097867 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.097721    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ff0a920b-f15f-4909-906b-61dc41ef1263-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zvkh9\" (UID: \"ff0a920b-f15f-4909-906b-61dc41ef1263\") " pod="openshift-multus/multus-additional-cni-plugins-zvkh9"
Oct 30 12:51:42.097867 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.097736    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85602130-14de-443d-b237-b213c0ce8b82-tmp\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:42.097867 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.097733    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls6hw\" (UniqueName: \"kubernetes.io/projected/c83bcf7b-a1d6-457b-9d7c-bb03c2cbd449-kube-api-access-ls6hw\") pod \"node-resolver-7mvnz\" (UID: \"c83bcf7b-a1d6-457b-9d7c-bb03c2cbd449\") " pod="openshift-dns/node-resolver-7mvnz"
Oct 30 12:51:42.097867 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.097781    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ff0a920b-f15f-4909-906b-61dc41ef1263-cnibin\") pod \"multus-additional-cni-plugins-zvkh9\" (UID: \"ff0a920b-f15f-4909-906b-61dc41ef1263\") " pod="openshift-multus/multus-additional-cni-plugins-zvkh9"
Oct 30 12:51:42.097867 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.097873    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/85602130-14de-443d-b237-b213c0ce8b82-run\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:42.098565 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.097906    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdplx\" (UniqueName: \"kubernetes.io/projected/31d09b91-c973-4cf9-a7d8-d602ff9994f6-kube-api-access-zdplx\") pod \"node-ca-7pxn6\" (UID: \"31d09b91-c973-4cf9-a7d8-d602ff9994f6\") " pod="openshift-image-registry/node-ca-7pxn6"
Oct 30 12:51:42.098565 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.098000    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f8f7d922-7364-444f-a932-278628e26b83-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p5mdl\" (UID: \"f8f7d922-7364-444f-a932-278628e26b83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5mdl"
Oct 30 12:51:42.098565 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.098017    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/85602130-14de-443d-b237-b213c0ce8b82-etc-sysconfig\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:42.098565 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.098033    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-systemd-units\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:42.098565 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.098049    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-run-systemd\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:42.098565 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.098077    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f8f7d922-7364-444f-a932-278628e26b83-socket-dir\") pod \"aws-ebs-csi-driver-node-p5mdl\" (UID: \"f8f7d922-7364-444f-a932-278628e26b83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5mdl"
Oct 30 12:51:42.120629 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.120584    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtppn\" (UniqueName: \"kubernetes.io/projected/f8f7d922-7364-444f-a932-278628e26b83-kube-api-access-wtppn\") pod \"aws-ebs-csi-driver-node-p5mdl\" (UID: \"f8f7d922-7364-444f-a932-278628e26b83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5mdl"
Oct 30 12:51:42.121647 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.121632    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6s7g\" (UniqueName: \"kubernetes.io/projected/ff0a920b-f15f-4909-906b-61dc41ef1263-kube-api-access-q6s7g\") pod \"multus-additional-cni-plugins-zvkh9\" (UID: \"ff0a920b-f15f-4909-906b-61dc41ef1263\") " pod="openshift-multus/multus-additional-cni-plugins-zvkh9"
Oct 30 12:51:42.130904 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.130886    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78v2l\" (UniqueName: \"kubernetes.io/projected/19115051-3b70-47cd-a7a3-313b1b3c26c7-kube-api-access-78v2l\") pod \"iptables-alerter-9p6w9\" (UID: \"19115051-3b70-47cd-a7a3-313b1b3c26c7\") " pod="openshift-network-operator/iptables-alerter-9p6w9"
Oct 30 12:51:42.134964 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.134949    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdplx\" (UniqueName: \"kubernetes.io/projected/31d09b91-c973-4cf9-a7d8-d602ff9994f6-kube-api-access-zdplx\") pod \"node-ca-7pxn6\" (UID: \"31d09b91-c973-4cf9-a7d8-d602ff9994f6\") " pod="openshift-image-registry/node-ca-7pxn6"
Oct 30 12:51:42.135009 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.134963    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls6hw\" (UniqueName: \"kubernetes.io/projected/c83bcf7b-a1d6-457b-9d7c-bb03c2cbd449-kube-api-access-ls6hw\") pod \"node-resolver-7mvnz\" (UID: \"c83bcf7b-a1d6-457b-9d7c-bb03c2cbd449\") " pod="openshift-dns/node-resolver-7mvnz"
Oct 30 12:51:42.135009 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.134948    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pppc\" (UniqueName: \"kubernetes.io/projected/a1ca584f-61b1-4398-a0f6-5788e476aa20-kube-api-access-7pppc\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:42.138081 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.138066    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxq79\" (UniqueName: \"kubernetes.io/projected/69eab030-f488-433c-869a-52d628824391-kube-api-access-hxq79\") pod \"network-metrics-daemon-79wgd\" (UID: \"69eab030-f488-433c-869a-52d628824391\") " pod="openshift-multus/network-metrics-daemon-79wgd"
Oct 30 12:51:42.139279 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.139258    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-749nf\" (UniqueName: \"kubernetes.io/projected/85602130-14de-443d-b237-b213c0ce8b82-kube-api-access-749nf\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:42.140438 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.140423    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfwjf\" (UniqueName: \"kubernetes.io/projected/c1927296-d775-43c4-a874-8842ebfeb3c8-kube-api-access-jfwjf\") pod \"network-check-target-dlxch\" (UID: \"c1927296-d775-43c4-a874-8842ebfeb3c8\") " pod="openshift-network-diagnostics/network-check-target-dlxch"
Oct 30 12:51:42.140679 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.140665    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5s5m\" (UniqueName: \"kubernetes.io/projected/2d5efe23-0e0d-4a4e-9d03-42829f371c09-kube-api-access-x5s5m\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:42.199144 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.199096    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-host-run-netns\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:42.199144 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.199120    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-run-openvswitch\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:42.199285 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.199151    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-host-cni-bin\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:42.199285 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.199171    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/815986e1-07b4-43fb-95f5-745316bd5105-dbus\") pod \"global-pull-secret-syncer-rxg7f\" (UID: \"815986e1-07b4-43fb-95f5-745316bd5105\") " pod="kube-system/global-pull-secret-syncer-rxg7f"
Oct 30 12:51:42.199285 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.199185    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f8f7d922-7364-444f-a932-278628e26b83-device-dir\") pod \"aws-ebs-csi-driver-node-p5mdl\" (UID: \"f8f7d922-7364-444f-a932-278628e26b83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5mdl"
Oct 30 12:51:42.199285 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.199205    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-host-run-netns\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:42.199285 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.199206    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-run-openvswitch\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:42.199285 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.199240    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-host-cni-bin\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:42.199285 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.199260    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/815986e1-07b4-43fb-95f5-745316bd5105-dbus\") pod \"global-pull-secret-syncer-rxg7f\" (UID: \"815986e1-07b4-43fb-95f5-745316bd5105\") " pod="kube-system/global-pull-secret-syncer-rxg7f"
Oct 30 12:51:42.199285 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.199259    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f8f7d922-7364-444f-a932-278628e26b83-etc-selinux\") pod \"aws-ebs-csi-driver-node-p5mdl\" (UID: \"f8f7d922-7364-444f-a932-278628e26b83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5mdl"
Oct 30 12:51:42.199515 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.199245    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f8f7d922-7364-444f-a932-278628e26b83-device-dir\") pod \"aws-ebs-csi-driver-node-p5mdl\" (UID: \"f8f7d922-7364-444f-a932-278628e26b83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5mdl"
Oct 30 12:51:42.199515 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.199299    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f8f7d922-7364-444f-a932-278628e26b83-sys-fs\") pod \"aws-ebs-csi-driver-node-p5mdl\" (UID: \"f8f7d922-7364-444f-a932-278628e26b83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5mdl"
Oct 30 12:51:42.199515 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.199310    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f8f7d922-7364-444f-a932-278628e26b83-etc-selinux\") pod \"aws-ebs-csi-driver-node-p5mdl\" (UID: \"f8f7d922-7364-444f-a932-278628e26b83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5mdl"
Oct 30 12:51:42.199515 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.199328    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f8f7d922-7364-444f-a932-278628e26b83-sys-fs\") pod \"aws-ebs-csi-driver-node-p5mdl\" (UID: \"f8f7d922-7364-444f-a932-278628e26b83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5mdl"
Oct 30 12:51:42.199515 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.199333    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-var-lib-openvswitch\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:42.199515 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.199352    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-etc-openvswitch\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:42.199515 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.199366    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/85602130-14de-443d-b237-b213c0ce8b82-host\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:42.199515 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.199383    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/85602130-14de-443d-b237-b213c0ce8b82-etc-sysctl-d\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:42.199515 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.199407    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/85602130-14de-443d-b237-b213c0ce8b82-etc-systemd\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:42.199515 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.199427    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-etc-openvswitch\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:42.199515 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.199434    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-var-lib-openvswitch\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:42.199515 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.199458    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/85602130-14de-443d-b237-b213c0ce8b82-etc-systemd\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:42.199515 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.199488    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/85602130-14de-443d-b237-b213c0ce8b82-host\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:42.199515 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.199498    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/85602130-14de-443d-b237-b213c0ce8b82-etc-sysctl-d\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:42.199515 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.199504    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-host-var-lib-cni-bin\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:42.200124 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.199529    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-host-var-lib-cni-bin\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:42.200124 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.199553    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-host-run-multus-certs\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:42.200124 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.199567    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-run-ovn\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:42.200124 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.199582    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:42.200124 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.199592    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-host-run-multus-certs\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:42.200124 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.199636    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/19115051-3b70-47cd-a7a3-313b1b3c26c7-host-slash\") pod \"iptables-alerter-9p6w9\" (UID: \"19115051-3b70-47cd-a7a3-313b1b3c26c7\") " pod="openshift-network-operator/iptables-alerter-9p6w9"
Oct 30 12:51:42.200124 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.199642    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-run-ovn\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:42.200124 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.199653    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-host-run-k8s-cni-cncf-io\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:42.200124 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.199666    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/19115051-3b70-47cd-a7a3-313b1b3c26c7-host-slash\") pod \"iptables-alerter-9p6w9\" (UID: \"19115051-3b70-47cd-a7a3-313b1b3c26c7\") " pod="openshift-network-operator/iptables-alerter-9p6w9"
Oct 30 12:51:42.200124 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.199668    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-host-var-lib-kubelet\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:42.200124 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.199668    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:42.200124 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.199695    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ff0a920b-f15f-4909-906b-61dc41ef1263-system-cni-dir\") pod \"multus-additional-cni-plugins-zvkh9\" (UID: \"ff0a920b-f15f-4909-906b-61dc41ef1263\") " pod="openshift-multus/multus-additional-cni-plugins-zvkh9"
Oct 30 12:51:42.200124 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.199693    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-host-run-k8s-cni-cncf-io\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:42.200124 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.199714    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-cnibin\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:42.200124 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.199738    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-host-var-lib-kubelet\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:42.200124 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.199738    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ff0a920b-f15f-4909-906b-61dc41ef1263-system-cni-dir\") pod \"multus-additional-cni-plugins-zvkh9\" (UID: \"ff0a920b-f15f-4909-906b-61dc41ef1263\") " pod="openshift-multus/multus-additional-cni-plugins-zvkh9"
Oct 30 12:51:42.200124 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.199766    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-cnibin\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:42.200124 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.199774    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-etc-kubernetes\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:42.200124 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.199801    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-hostroot\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:42.200124 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.199824    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c83bcf7b-a1d6-457b-9d7c-bb03c2cbd449-hosts-file\") pod \"node-resolver-7mvnz\" (UID: \"c83bcf7b-a1d6-457b-9d7c-bb03c2cbd449\") " pod="openshift-dns/node-resolver-7mvnz"
Oct 30 12:51:42.200124 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.199844    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-etc-kubernetes\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:42.200124 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.199845    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-node-log\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:42.200124 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.199864    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-hostroot\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:42.200124 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.199879    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-node-log\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:42.200124 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.199895    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c83bcf7b-a1d6-457b-9d7c-bb03c2cbd449-hosts-file\") pod \"node-resolver-7mvnz\" (UID: \"c83bcf7b-a1d6-457b-9d7c-bb03c2cbd449\") " pod="openshift-dns/node-resolver-7mvnz"
Oct 30 12:51:42.200124 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.199899    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/815986e1-07b4-43fb-95f5-745316bd5105-kubelet-config\") pod \"global-pull-secret-syncer-rxg7f\" (UID: \"815986e1-07b4-43fb-95f5-745316bd5105\") " pod="kube-system/global-pull-secret-syncer-rxg7f"
Oct 30 12:51:42.200124 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.199940    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/815986e1-07b4-43fb-95f5-745316bd5105-kubelet-config\") pod \"global-pull-secret-syncer-rxg7f\" (UID: \"815986e1-07b4-43fb-95f5-745316bd5105\") " pod="kube-system/global-pull-secret-syncer-rxg7f"
Oct 30 12:51:42.200124 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.199973    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/85602130-14de-443d-b237-b213c0ce8b82-var-lib-kubelet\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:42.200124 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.199988    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/31d09b91-c973-4cf9-a7d8-d602ff9994f6-host\") pod \"node-ca-7pxn6\" (UID: \"31d09b91-c973-4cf9-a7d8-d602ff9994f6\") " pod="openshift-image-registry/node-ca-7pxn6"
Oct 30 12:51:42.200124 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.200002    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ff0a920b-f15f-4909-906b-61dc41ef1263-cnibin\") pod \"multus-additional-cni-plugins-zvkh9\" (UID: \"ff0a920b-f15f-4909-906b-61dc41ef1263\") " pod="openshift-multus/multus-additional-cni-plugins-zvkh9"
Oct 30 12:51:42.200124 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.200015    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/85602130-14de-443d-b237-b213c0ce8b82-run\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:42.200124 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.200029    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f8f7d922-7364-444f-a932-278628e26b83-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p5mdl\" (UID: \"f8f7d922-7364-444f-a932-278628e26b83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5mdl"
Oct 30 12:51:42.200124 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.200039    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/31d09b91-c973-4cf9-a7d8-d602ff9994f6-host\") pod \"node-ca-7pxn6\" (UID: \"31d09b91-c973-4cf9-a7d8-d602ff9994f6\") " pod="openshift-image-registry/node-ca-7pxn6"
Oct 30 12:51:42.200124 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.200070    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/85602130-14de-443d-b237-b213c0ce8b82-run\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:42.200124 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.200069    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ff0a920b-f15f-4909-906b-61dc41ef1263-cnibin\") pod \"multus-additional-cni-plugins-zvkh9\" (UID: \"ff0a920b-f15f-4909-906b-61dc41ef1263\") " pod="openshift-multus/multus-additional-cni-plugins-zvkh9"
Oct 30 12:51:42.200124 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.200101    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f8f7d922-7364-444f-a932-278628e26b83-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p5mdl\" (UID: \"f8f7d922-7364-444f-a932-278628e26b83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5mdl"
Oct 30 12:51:42.200124 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.200085    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/85602130-14de-443d-b237-b213c0ce8b82-var-lib-kubelet\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:42.200124 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.200105    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/85602130-14de-443d-b237-b213c0ce8b82-etc-sysconfig\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:42.200124 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.200068    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/85602130-14de-443d-b237-b213c0ce8b82-etc-sysconfig\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:42.201487 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.200153    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-systemd-units\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:42.201487 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.200167    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-run-systemd\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:42.201487 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.200181    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f8f7d922-7364-444f-a932-278628e26b83-socket-dir\") pod \"aws-ebs-csi-driver-node-p5mdl\" (UID: \"f8f7d922-7364-444f-a932-278628e26b83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5mdl"
Oct 30 12:51:42.201487 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.200195    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-host-var-lib-cni-multus\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:42.201487 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.200205    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-run-systemd\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:42.201487 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.200209    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-systemd-units\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:42.201487 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.200224    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85602130-14de-443d-b237-b213c0ce8b82-etc-kubernetes\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:42.201487 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.200239    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ff0a920b-f15f-4909-906b-61dc41ef1263-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zvkh9\" (UID: \"ff0a920b-f15f-4909-906b-61dc41ef1263\") " pod="openshift-multus/multus-additional-cni-plugins-zvkh9"
Oct 30 12:51:42.201487 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.200261    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f8f7d922-7364-444f-a932-278628e26b83-socket-dir\") pod \"aws-ebs-csi-driver-node-p5mdl\" (UID: \"f8f7d922-7364-444f-a932-278628e26b83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5mdl"
Oct 30 12:51:42.201487 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.200266    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-system-cni-dir\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:42.201487 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.200260    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-host-var-lib-cni-multus\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:42.201487 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.200293    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-host-slash\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:42.201487 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.200298    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85602130-14de-443d-b237-b213c0ce8b82-etc-kubernetes\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:42.201487 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.200312    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-log-socket\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:42.201487 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.200325    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-system-cni-dir\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:42.201487 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.200326    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-host-slash\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:42.201487 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.200328    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ff0a920b-f15f-4909-906b-61dc41ef1263-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zvkh9\" (UID: \"ff0a920b-f15f-4909-906b-61dc41ef1263\") " pod="openshift-multus/multus-additional-cni-plugins-zvkh9"
Oct 30 12:51:42.201487 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.200347    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-host-run-ovn-kubernetes\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:42.201487 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.200349    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-log-socket\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:42.201487 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.200373    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-host-run-ovn-kubernetes\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:42.201487 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.200382    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-host-cni-netd\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:42.201487 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.200409    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/85602130-14de-443d-b237-b213c0ce8b82-etc-sysctl-conf\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:42.201487 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.200431    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-multus-cni-dir\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:42.201487 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.200441    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-host-cni-netd\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:42.201487 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.200502    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/85602130-14de-443d-b237-b213c0ce8b82-etc-modprobe-d\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:42.201487 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.200521    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-multus-cni-dir\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:42.201487 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.200523    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/85602130-14de-443d-b237-b213c0ce8b82-etc-sysctl-conf\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:42.201487 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.200534    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/85602130-14de-443d-b237-b213c0ce8b82-lib-modules\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:42.201487 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.200628    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/85602130-14de-443d-b237-b213c0ce8b82-lib-modules\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:42.201487 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.200619    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-host-kubelet\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:42.201487 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.200647    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d5efe23-0e0d-4a4e-9d03-42829f371c09-host-kubelet\") pod \"ovnkube-node-wpkrj\" (UID: \"2d5efe23-0e0d-4a4e-9d03-42829f371c09\") " pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:42.201487 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.200655    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f8f7d922-7364-444f-a932-278628e26b83-registration-dir\") pod \"aws-ebs-csi-driver-node-p5mdl\" (UID: \"f8f7d922-7364-444f-a932-278628e26b83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5mdl"
Oct 30 12:51:42.201487 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.200693    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-multus-socket-dir-parent\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:42.201487 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.200705    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/85602130-14de-443d-b237-b213c0ce8b82-etc-modprobe-d\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:42.201487 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.200717    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-host-run-netns\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:42.201487 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.200726    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f8f7d922-7364-444f-a932-278628e26b83-registration-dir\") pod \"aws-ebs-csi-driver-node-p5mdl\" (UID: \"f8f7d922-7364-444f-a932-278628e26b83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5mdl"
Oct 30 12:51:42.201487 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.200745    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-host-run-netns\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:42.201487 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.200749    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-multus-conf-dir\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:42.201487 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.200764    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-multus-socket-dir-parent\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:42.201487 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.200777    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/85602130-14de-443d-b237-b213c0ce8b82-sys\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:42.201487 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.200783    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a1ca584f-61b1-4398-a0f6-5788e476aa20-multus-conf-dir\") pod \"multus-mv5cv\" (UID: \"a1ca584f-61b1-4398-a0f6-5788e476aa20\") " pod="openshift-multus/multus-mv5cv"
Oct 30 12:51:42.201487 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.200799    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/85602130-14de-443d-b237-b213c0ce8b82-sys\") pod \"tuned-7r6q9\" (UID: \"85602130-14de-443d-b237-b213c0ce8b82\") " pod="openshift-cluster-node-tuning-operator/tuned-7r6q9"
Oct 30 12:51:42.384246 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:42.384223    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-dlxch"
Oct 30 12:51:48.741463 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:48.741430    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:49.771727 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:49.771700    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:51:49.772101 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:49.771871    4364 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 12:51:54.105531 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.105501    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-22gcb"]
Oct 30 12:51:54.107216 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.107188    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-22gcb"
Oct 30 12:51:54.129571 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.129550    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-bdxlj\""
Oct 30 12:51:54.132336 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.132313    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Oct 30 12:51:54.143247 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.143216    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-22gcb"]
Oct 30 12:51:54.202246 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.202223    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-ffvhz"]
Oct 30 12:51:54.203982 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.203956    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-p566k"]
Oct 30 12:51:54.204187 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.204169    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ffvhz"
Oct 30 12:51:54.205277 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.205260    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-64f798964-m7xhn"]
Oct 30 12:51:54.205408 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.205392    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-p566k"
Oct 30 12:51:54.206802 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.206787    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64f798964-m7xhn"
Oct 30 12:51:54.217417 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.217395    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-54cf5557c-9c668"]
Oct 30 12:51:54.219278 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.219262    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54cf5557c-9c668"
Oct 30 12:51:54.223704 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.223686    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-j8984"]
Oct 30 12:51:54.225169 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.225155    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-j8984"
Oct 30 12:51:54.236303 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.236285    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 12:51:54.236433 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.236418    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 12:51:54.236556 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.236539    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 12:51:54.237056 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.237037    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ffvhz"]
Oct 30 12:51:54.238015 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.237972    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 30 12:51:54.238167 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.238085    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 30 12:51:54.242190 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.242169    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-9mpc4\""
Oct 30 12:51:54.242190 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.242180    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 12:51:54.242342 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.242245    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 30 12:51:54.242342 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.242260    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 30 12:51:54.242445 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.242397    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 30 12:51:54.242861 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.242844    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-p566k"]
Oct 30 12:51:54.249885 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.249869    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-s894x\""
Oct 30 12:51:54.249885 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.249874    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6vr75\""
Oct 30 12:51:54.255865 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.255842    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/cd7629fa-bfd4-44ab-b51b-39e5eea7032d-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-22gcb\" (UID: \"cd7629fa-bfd4-44ab-b51b-39e5eea7032d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-22gcb"
Oct 30 12:51:54.267166 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.267147    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 12:51:54.267280 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.267257    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 12:51:54.267505 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.267490    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 12:51:54.267689 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.267663    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-j8984"]
Oct 30 12:51:54.268016 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.268000    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-54cf5557c-9c668"]
Oct 30 12:51:54.268794 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.268774    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64f798964-m7xhn"]
Oct 30 12:51:54.276159 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.276142    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-zn76j"]
Oct 30 12:51:54.277857 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.277843    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zn76j"
Oct 30 12:51:54.295388 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.295370    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 12:51:54.295388 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.295386    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 12:51:54.295522 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.295405    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 12:51:54.303104 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.303085    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zn76j"]
Oct 30 12:51:54.330178 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.330157    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 12:51:54.330267 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.330214    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 12:51:54.330267 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.330216    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 12:51:54.330267 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.330219    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-hhkwq\""
Oct 30 12:51:54.330267 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.330247    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 12:51:54.330444 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.330326    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 12:51:54.330444 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.330365    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rq622\""
Oct 30 12:51:54.330444 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.330416    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kq2jq\""
Oct 30 12:51:54.356423 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.356385    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfgf7\" (UniqueName: \"kubernetes.io/projected/62adaa28-5f97-4e1b-865f-d931ffd19d04-kube-api-access-dfgf7\") pod \"dns-default-ffvhz\" (UID: \"62adaa28-5f97-4e1b-865f-d931ffd19d04\") " pod="openshift-dns/dns-default-ffvhz"
Oct 30 12:51:54.356423 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.356407    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/03052418-91d3-44c9-95f2-f0b014dcbcbe-tmp\") pod \"downloads-6dd789c8b7-p566k\" (UID: \"03052418-91d3-44c9-95f2-f0b014dcbcbe\") " pod="openshift-console/downloads-6dd789c8b7-p566k"
Oct 30 12:51:54.356516 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.356427    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0448750b-7ffc-43c1-bdd8-198f73d940be-console-serving-cert\") pod \"console-64f798964-m7xhn\" (UID: \"0448750b-7ffc-43c1-bdd8-198f73d940be\") " pod="openshift-console/console-64f798964-m7xhn"
Oct 30 12:51:54.356516 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.356442    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f3dfe649-cc32-4dc7-8a09-b4790f17f2fb-cert\") pod \"ingress-canary-j8984\" (UID: \"f3dfe649-cc32-4dc7-8a09-b4790f17f2fb\") " pod="openshift-ingress-canary/ingress-canary-j8984"
Oct 30 12:51:54.356516 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.356457    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/62adaa28-5f97-4e1b-865f-d931ffd19d04-metrics-tls\") pod \"dns-default-ffvhz\" (UID: \"62adaa28-5f97-4e1b-865f-d931ffd19d04\") " pod="openshift-dns/dns-default-ffvhz"
Oct 30 12:51:54.356625 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.356535    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0448750b-7ffc-43c1-bdd8-198f73d940be-service-ca\") pod \"console-64f798964-m7xhn\" (UID: \"0448750b-7ffc-43c1-bdd8-198f73d940be\") " pod="openshift-console/console-64f798964-m7xhn"
Oct 30 12:51:54.356625 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.356577    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6fe75d93-6ca5-4051-91f9-a02766ce12cc-bound-sa-token\") pod \"image-registry-54cf5557c-9c668\" (UID: \"6fe75d93-6ca5-4051-91f9-a02766ce12cc\") " pod="openshift-image-registry/image-registry-54cf5557c-9c668"
Oct 30 12:51:54.356688 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.356638    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0448750b-7ffc-43c1-bdd8-198f73d940be-console-oauth-config\") pod \"console-64f798964-m7xhn\" (UID: \"0448750b-7ffc-43c1-bdd8-198f73d940be\") " pod="openshift-console/console-64f798964-m7xhn"
Oct 30 12:51:54.356688 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.356679    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/029281be-58e5-4f0e-a7b7-bbb7804ef32e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zn76j\" (UID: \"029281be-58e5-4f0e-a7b7-bbb7804ef32e\") " pod="openshift-insights/insights-runtime-extractor-zn76j"
Oct 30 12:51:54.356746 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.356731    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/cd7629fa-bfd4-44ab-b51b-39e5eea7032d-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-22gcb\" (UID: \"cd7629fa-bfd4-44ab-b51b-39e5eea7032d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-22gcb"
Oct 30 12:51:54.356779 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.356754    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/029281be-58e5-4f0e-a7b7-bbb7804ef32e-data-volume\") pod \"insights-runtime-extractor-zn76j\" (UID: \"029281be-58e5-4f0e-a7b7-bbb7804ef32e\") " pod="openshift-insights/insights-runtime-extractor-zn76j"
Oct 30 12:51:54.356810 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.356774    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6fe75d93-6ca5-4051-91f9-a02766ce12cc-registry-tls\") pod \"image-registry-54cf5557c-9c668\" (UID: \"6fe75d93-6ca5-4051-91f9-a02766ce12cc\") " pod="openshift-image-registry/image-registry-54cf5557c-9c668"
Oct 30 12:51:54.356810 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.356795    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6fe75d93-6ca5-4051-91f9-a02766ce12cc-installation-pull-secrets\") pod \"image-registry-54cf5557c-9c668\" (UID: \"6fe75d93-6ca5-4051-91f9-a02766ce12cc\") " pod="openshift-image-registry/image-registry-54cf5557c-9c668"
Oct 30 12:51:54.356896 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.356811    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmv89\" (UniqueName: \"kubernetes.io/projected/03052418-91d3-44c9-95f2-f0b014dcbcbe-kube-api-access-tmv89\") pod \"downloads-6dd789c8b7-p566k\" (UID: \"03052418-91d3-44c9-95f2-f0b014dcbcbe\") " pod="openshift-console/downloads-6dd789c8b7-p566k"
Oct 30 12:51:54.356896 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.356827    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nggsh\" (UniqueName: \"kubernetes.io/projected/0448750b-7ffc-43c1-bdd8-198f73d940be-kube-api-access-nggsh\") pod \"console-64f798964-m7xhn\" (UID: \"0448750b-7ffc-43c1-bdd8-198f73d940be\") " pod="openshift-console/console-64f798964-m7xhn"
Oct 30 12:51:54.356896 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.356843    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/029281be-58e5-4f0e-a7b7-bbb7804ef32e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zn76j\" (UID: \"029281be-58e5-4f0e-a7b7-bbb7804ef32e\") " pod="openshift-insights/insights-runtime-extractor-zn76j"
Oct 30 12:51:54.356896 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.356879    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkbsw\" (UniqueName: \"kubernetes.io/projected/6fe75d93-6ca5-4051-91f9-a02766ce12cc-kube-api-access-hkbsw\") pod \"image-registry-54cf5557c-9c668\" (UID: \"6fe75d93-6ca5-4051-91f9-a02766ce12cc\") " pod="openshift-image-registry/image-registry-54cf5557c-9c668"
Oct 30 12:51:54.357049 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.356905    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62adaa28-5f97-4e1b-865f-d931ffd19d04-tmp-dir\") pod \"dns-default-ffvhz\" (UID: \"62adaa28-5f97-4e1b-865f-d931ffd19d04\") " pod="openshift-dns/dns-default-ffvhz"
Oct 30 12:51:54.357049 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.356960    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0448750b-7ffc-43c1-bdd8-198f73d940be-console-config\") pod \"console-64f798964-m7xhn\" (UID: \"0448750b-7ffc-43c1-bdd8-198f73d940be\") " pod="openshift-console/console-64f798964-m7xhn"
Oct 30 12:51:54.357049 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.357004    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6fe75d93-6ca5-4051-91f9-a02766ce12cc-image-registry-private-configuration\") pod \"image-registry-54cf5557c-9c668\" (UID: \"6fe75d93-6ca5-4051-91f9-a02766ce12cc\") " pod="openshift-image-registry/image-registry-54cf5557c-9c668"
Oct 30 12:51:54.357185 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.357062    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0448750b-7ffc-43c1-bdd8-198f73d940be-tmp\") pod \"console-64f798964-m7xhn\" (UID: \"0448750b-7ffc-43c1-bdd8-198f73d940be\") " pod="openshift-console/console-64f798964-m7xhn"
Oct 30 12:51:54.357185 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.357094    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/62adaa28-5f97-4e1b-865f-d931ffd19d04-config-volume\") pod \"dns-default-ffvhz\" (UID: \"62adaa28-5f97-4e1b-865f-d931ffd19d04\") " pod="openshift-dns/dns-default-ffvhz"
Oct 30 12:51:54.357185 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.357098    4364 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 12:51:54.357185 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.357124    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/029281be-58e5-4f0e-a7b7-bbb7804ef32e-crio-socket\") pod \"insights-runtime-extractor-zn76j\" (UID: \"029281be-58e5-4f0e-a7b7-bbb7804ef32e\") " pod="openshift-insights/insights-runtime-extractor-zn76j"
Oct 30 12:51:54.357185 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.357162    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq786\" (UniqueName: \"kubernetes.io/projected/029281be-58e5-4f0e-a7b7-bbb7804ef32e-kube-api-access-sq786\") pod \"insights-runtime-extractor-zn76j\" (UID: \"029281be-58e5-4f0e-a7b7-bbb7804ef32e\") " pod="openshift-insights/insights-runtime-extractor-zn76j"
Oct 30 12:51:54.357384 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.357194    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0448750b-7ffc-43c1-bdd8-198f73d940be-oauth-serving-cert\") pod \"console-64f798964-m7xhn\" (UID: \"0448750b-7ffc-43c1-bdd8-198f73d940be\") " pod="openshift-console/console-64f798964-m7xhn"
Oct 30 12:51:54.357384 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.357241    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6fe75d93-6ca5-4051-91f9-a02766ce12cc-trusted-ca\") pod \"image-registry-54cf5557c-9c668\" (UID: \"6fe75d93-6ca5-4051-91f9-a02766ce12cc\") " pod="openshift-image-registry/image-registry-54cf5557c-9c668"
Oct 30 12:51:54.357384 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.357260    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj8jj\" (UniqueName: \"kubernetes.io/projected/f3dfe649-cc32-4dc7-8a09-b4790f17f2fb-kube-api-access-gj8jj\") pod \"ingress-canary-j8984\" (UID: \"f3dfe649-cc32-4dc7-8a09-b4790f17f2fb\") " pod="openshift-ingress-canary/ingress-canary-j8984"
Oct 30 12:51:54.357384 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.357282    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6fe75d93-6ca5-4051-91f9-a02766ce12cc-registry-certificates\") pod \"image-registry-54cf5557c-9c668\" (UID: \"6fe75d93-6ca5-4051-91f9-a02766ce12cc\") " pod="openshift-image-registry/image-registry-54cf5557c-9c668"
Oct 30 12:51:54.357384 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.357352    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6fe75d93-6ca5-4051-91f9-a02766ce12cc-ca-trust-extracted\") pod \"image-registry-54cf5557c-9c668\" (UID: \"6fe75d93-6ca5-4051-91f9-a02766ce12cc\") " pod="openshift-image-registry/image-registry-54cf5557c-9c668"
Oct 30 12:51:54.359271 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.359256    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/cd7629fa-bfd4-44ab-b51b-39e5eea7032d-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-22gcb\" (UID: \"cd7629fa-bfd4-44ab-b51b-39e5eea7032d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-22gcb"
Oct 30 12:51:54.416561 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.416536    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-22gcb"
Oct 30 12:51:54.458678 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.458644    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0448750b-7ffc-43c1-bdd8-198f73d940be-service-ca\") pod \"console-64f798964-m7xhn\" (UID: \"0448750b-7ffc-43c1-bdd8-198f73d940be\") " pod="openshift-console/console-64f798964-m7xhn"
Oct 30 12:51:54.458678 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.458680    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6fe75d93-6ca5-4051-91f9-a02766ce12cc-bound-sa-token\") pod \"image-registry-54cf5557c-9c668\" (UID: \"6fe75d93-6ca5-4051-91f9-a02766ce12cc\") " pod="openshift-image-registry/image-registry-54cf5557c-9c668"
Oct 30 12:51:54.459514 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.458707    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0448750b-7ffc-43c1-bdd8-198f73d940be-console-oauth-config\") pod \"console-64f798964-m7xhn\" (UID: \"0448750b-7ffc-43c1-bdd8-198f73d940be\") " pod="openshift-console/console-64f798964-m7xhn"
Oct 30 12:51:54.459514 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.458733    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/029281be-58e5-4f0e-a7b7-bbb7804ef32e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zn76j\" (UID: \"029281be-58e5-4f0e-a7b7-bbb7804ef32e\") " pod="openshift-insights/insights-runtime-extractor-zn76j"
Oct 30 12:51:54.459514 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.458771    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/029281be-58e5-4f0e-a7b7-bbb7804ef32e-data-volume\") pod \"insights-runtime-extractor-zn76j\" (UID: \"029281be-58e5-4f0e-a7b7-bbb7804ef32e\") " pod="openshift-insights/insights-runtime-extractor-zn76j"
Oct 30 12:51:54.459514 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.458796    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6fe75d93-6ca5-4051-91f9-a02766ce12cc-registry-tls\") pod \"image-registry-54cf5557c-9c668\" (UID: \"6fe75d93-6ca5-4051-91f9-a02766ce12cc\") " pod="openshift-image-registry/image-registry-54cf5557c-9c668"
Oct 30 12:51:54.459514 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.458818    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6fe75d93-6ca5-4051-91f9-a02766ce12cc-installation-pull-secrets\") pod \"image-registry-54cf5557c-9c668\" (UID: \"6fe75d93-6ca5-4051-91f9-a02766ce12cc\") " pod="openshift-image-registry/image-registry-54cf5557c-9c668"
Oct 30 12:51:54.459514 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.458847    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmv89\" (UniqueName: \"kubernetes.io/projected/03052418-91d3-44c9-95f2-f0b014dcbcbe-kube-api-access-tmv89\") pod \"downloads-6dd789c8b7-p566k\" (UID: \"03052418-91d3-44c9-95f2-f0b014dcbcbe\") " pod="openshift-console/downloads-6dd789c8b7-p566k"
Oct 30 12:51:54.459514 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.458879    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nggsh\" (UniqueName: \"kubernetes.io/projected/0448750b-7ffc-43c1-bdd8-198f73d940be-kube-api-access-nggsh\") pod \"console-64f798964-m7xhn\" (UID: \"0448750b-7ffc-43c1-bdd8-198f73d940be\") " pod="openshift-console/console-64f798964-m7xhn"
Oct 30 12:51:54.459514 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.458902    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/029281be-58e5-4f0e-a7b7-bbb7804ef32e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zn76j\" (UID: \"029281be-58e5-4f0e-a7b7-bbb7804ef32e\") " pod="openshift-insights/insights-runtime-extractor-zn76j"
Oct 30 12:51:54.459514 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.458929    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkbsw\" (UniqueName: \"kubernetes.io/projected/6fe75d93-6ca5-4051-91f9-a02766ce12cc-kube-api-access-hkbsw\") pod \"image-registry-54cf5557c-9c668\" (UID: \"6fe75d93-6ca5-4051-91f9-a02766ce12cc\") " pod="openshift-image-registry/image-registry-54cf5557c-9c668"
Oct 30 12:51:54.459514 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.458952    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62adaa28-5f97-4e1b-865f-d931ffd19d04-tmp-dir\") pod \"dns-default-ffvhz\" (UID: \"62adaa28-5f97-4e1b-865f-d931ffd19d04\") " pod="openshift-dns/dns-default-ffvhz"
Oct 30 12:51:54.459514 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.458978    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0448750b-7ffc-43c1-bdd8-198f73d940be-console-config\") pod \"console-64f798964-m7xhn\" (UID: \"0448750b-7ffc-43c1-bdd8-198f73d940be\") " pod="openshift-console/console-64f798964-m7xhn"
Oct 30 12:51:54.459514 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.459007    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6fe75d93-6ca5-4051-91f9-a02766ce12cc-image-registry-private-configuration\") pod \"image-registry-54cf5557c-9c668\" (UID: \"6fe75d93-6ca5-4051-91f9-a02766ce12cc\") " pod="openshift-image-registry/image-registry-54cf5557c-9c668"
Oct 30 12:51:54.459514 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.459033    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0448750b-7ffc-43c1-bdd8-198f73d940be-tmp\") pod \"console-64f798964-m7xhn\" (UID: \"0448750b-7ffc-43c1-bdd8-198f73d940be\") " pod="openshift-console/console-64f798964-m7xhn"
Oct 30 12:51:54.459514 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.459055    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/62adaa28-5f97-4e1b-865f-d931ffd19d04-config-volume\") pod \"dns-default-ffvhz\" (UID: \"62adaa28-5f97-4e1b-865f-d931ffd19d04\") " pod="openshift-dns/dns-default-ffvhz"
Oct 30 12:51:54.459514 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.459086    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/029281be-58e5-4f0e-a7b7-bbb7804ef32e-crio-socket\") pod \"insights-runtime-extractor-zn76j\" (UID: \"029281be-58e5-4f0e-a7b7-bbb7804ef32e\") " pod="openshift-insights/insights-runtime-extractor-zn76j"
Oct 30 12:51:54.459514 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.459114    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq786\" (UniqueName: \"kubernetes.io/projected/029281be-58e5-4f0e-a7b7-bbb7804ef32e-kube-api-access-sq786\") pod \"insights-runtime-extractor-zn76j\" (UID: \"029281be-58e5-4f0e-a7b7-bbb7804ef32e\") " pod="openshift-insights/insights-runtime-extractor-zn76j"
Oct 30 12:51:54.459514 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.459138    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0448750b-7ffc-43c1-bdd8-198f73d940be-oauth-serving-cert\") pod \"console-64f798964-m7xhn\" (UID: \"0448750b-7ffc-43c1-bdd8-198f73d940be\") " pod="openshift-console/console-64f798964-m7xhn"
Oct 30 12:51:54.459514 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.459161    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6fe75d93-6ca5-4051-91f9-a02766ce12cc-trusted-ca\") pod \"image-registry-54cf5557c-9c668\" (UID: \"6fe75d93-6ca5-4051-91f9-a02766ce12cc\") " pod="openshift-image-registry/image-registry-54cf5557c-9c668"
Oct 30 12:51:54.459514 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.459183    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj8jj\" (UniqueName: \"kubernetes.io/projected/f3dfe649-cc32-4dc7-8a09-b4790f17f2fb-kube-api-access-gj8jj\") pod \"ingress-canary-j8984\" (UID: \"f3dfe649-cc32-4dc7-8a09-b4790f17f2fb\") " pod="openshift-ingress-canary/ingress-canary-j8984"
Oct 30 12:51:54.459514 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.459208    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6fe75d93-6ca5-4051-91f9-a02766ce12cc-registry-certificates\") pod \"image-registry-54cf5557c-9c668\" (UID: \"6fe75d93-6ca5-4051-91f9-a02766ce12cc\") " pod="openshift-image-registry/image-registry-54cf5557c-9c668"
Oct 30 12:51:54.459514 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.459234    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6fe75d93-6ca5-4051-91f9-a02766ce12cc-ca-trust-extracted\") pod \"image-registry-54cf5557c-9c668\" (UID: \"6fe75d93-6ca5-4051-91f9-a02766ce12cc\") " pod="openshift-image-registry/image-registry-54cf5557c-9c668"
Oct 30 12:51:54.459514 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.459262    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfgf7\" (UniqueName: \"kubernetes.io/projected/62adaa28-5f97-4e1b-865f-d931ffd19d04-kube-api-access-dfgf7\") pod \"dns-default-ffvhz\" (UID: \"62adaa28-5f97-4e1b-865f-d931ffd19d04\") " pod="openshift-dns/dns-default-ffvhz"
Oct 30 12:51:54.459514 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.459285    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/03052418-91d3-44c9-95f2-f0b014dcbcbe-tmp\") pod \"downloads-6dd789c8b7-p566k\" (UID: \"03052418-91d3-44c9-95f2-f0b014dcbcbe\") " pod="openshift-console/downloads-6dd789c8b7-p566k"
Oct 30 12:51:54.459514 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.459313    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0448750b-7ffc-43c1-bdd8-198f73d940be-console-serving-cert\") pod \"console-64f798964-m7xhn\" (UID: \"0448750b-7ffc-43c1-bdd8-198f73d940be\") " pod="openshift-console/console-64f798964-m7xhn"
Oct 30 12:51:54.459514 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.459334    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f3dfe649-cc32-4dc7-8a09-b4790f17f2fb-cert\") pod \"ingress-canary-j8984\" (UID: \"f3dfe649-cc32-4dc7-8a09-b4790f17f2fb\") " pod="openshift-ingress-canary/ingress-canary-j8984"
Oct 30 12:51:54.459514 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.459360    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/62adaa28-5f97-4e1b-865f-d931ffd19d04-metrics-tls\") pod \"dns-default-ffvhz\" (UID: \"62adaa28-5f97-4e1b-865f-d931ffd19d04\") " pod="openshift-dns/dns-default-ffvhz"
Oct 30 12:51:54.461495 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.459755    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/029281be-58e5-4f0e-a7b7-bbb7804ef32e-data-volume\") pod \"insights-runtime-extractor-zn76j\" (UID: \"029281be-58e5-4f0e-a7b7-bbb7804ef32e\") " pod="openshift-insights/insights-runtime-extractor-zn76j"
Oct 30 12:51:54.461495 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.460252    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/029281be-58e5-4f0e-a7b7-bbb7804ef32e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zn76j\" (UID: \"029281be-58e5-4f0e-a7b7-bbb7804ef32e\") " pod="openshift-insights/insights-runtime-extractor-zn76j"
Oct 30 12:51:54.461495 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.460560    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/029281be-58e5-4f0e-a7b7-bbb7804ef32e-crio-socket\") pod \"insights-runtime-extractor-zn76j\" (UID: \"029281be-58e5-4f0e-a7b7-bbb7804ef32e\") " pod="openshift-insights/insights-runtime-extractor-zn76j"
Oct 30 12:51:54.461495 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.461012    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0448750b-7ffc-43c1-bdd8-198f73d940be-service-ca\") pod \"console-64f798964-m7xhn\" (UID: \"0448750b-7ffc-43c1-bdd8-198f73d940be\") " pod="openshift-console/console-64f798964-m7xhn"
Oct 30 12:51:54.464061 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.462135    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6fe75d93-6ca5-4051-91f9-a02766ce12cc-registry-certificates\") pod \"image-registry-54cf5557c-9c668\" (UID: \"6fe75d93-6ca5-4051-91f9-a02766ce12cc\") " pod="openshift-image-registry/image-registry-54cf5557c-9c668"
Oct 30 12:51:54.464061 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.462318    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/03052418-91d3-44c9-95f2-f0b014dcbcbe-tmp\") pod \"downloads-6dd789c8b7-p566k\" (UID: \"03052418-91d3-44c9-95f2-f0b014dcbcbe\") " pod="openshift-console/downloads-6dd789c8b7-p566k"
Oct 30 12:51:54.464061 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.462922    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0448750b-7ffc-43c1-bdd8-198f73d940be-oauth-serving-cert\") pod \"console-64f798964-m7xhn\" (UID: \"0448750b-7ffc-43c1-bdd8-198f73d940be\") " pod="openshift-console/console-64f798964-m7xhn"
Oct 30 12:51:54.464061 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.463269    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0448750b-7ffc-43c1-bdd8-198f73d940be-console-config\") pod \"console-64f798964-m7xhn\" (UID: \"0448750b-7ffc-43c1-bdd8-198f73d940be\") " pod="openshift-console/console-64f798964-m7xhn"
Oct 30 12:51:54.464061 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.463516    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6fe75d93-6ca5-4051-91f9-a02766ce12cc-ca-trust-extracted\") pod \"image-registry-54cf5557c-9c668\" (UID: \"6fe75d93-6ca5-4051-91f9-a02766ce12cc\") " pod="openshift-image-registry/image-registry-54cf5557c-9c668"
Oct 30 12:51:54.464061 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.463536    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62adaa28-5f97-4e1b-865f-d931ffd19d04-tmp-dir\") pod \"dns-default-ffvhz\" (UID: \"62adaa28-5f97-4e1b-865f-d931ffd19d04\") " pod="openshift-dns/dns-default-ffvhz"
Oct 30 12:51:54.464061 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.463723    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6fe75d93-6ca5-4051-91f9-a02766ce12cc-installation-pull-secrets\") pod \"image-registry-54cf5557c-9c668\" (UID: \"6fe75d93-6ca5-4051-91f9-a02766ce12cc\") " pod="openshift-image-registry/image-registry-54cf5557c-9c668"
Oct 30 12:51:54.464061 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.463850    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0448750b-7ffc-43c1-bdd8-198f73d940be-tmp\") pod \"console-64f798964-m7xhn\" (UID: \"0448750b-7ffc-43c1-bdd8-198f73d940be\") " pod="openshift-console/console-64f798964-m7xhn"
Oct 30 12:51:54.464903 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.464197    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/62adaa28-5f97-4e1b-865f-d931ffd19d04-config-volume\") pod \"dns-default-ffvhz\" (UID: \"62adaa28-5f97-4e1b-865f-d931ffd19d04\") " pod="openshift-dns/dns-default-ffvhz"
Oct 30 12:51:54.464903 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.464657    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6fe75d93-6ca5-4051-91f9-a02766ce12cc-trusted-ca\") pod \"image-registry-54cf5557c-9c668\" (UID: \"6fe75d93-6ca5-4051-91f9-a02766ce12cc\") " pod="openshift-image-registry/image-registry-54cf5557c-9c668"
Oct 30 12:51:54.465014 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.464923    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/029281be-58e5-4f0e-a7b7-bbb7804ef32e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zn76j\" (UID: \"029281be-58e5-4f0e-a7b7-bbb7804ef32e\") " pod="openshift-insights/insights-runtime-extractor-zn76j"
Oct 30 12:51:54.466663 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.466637    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6fe75d93-6ca5-4051-91f9-a02766ce12cc-image-registry-private-configuration\") pod \"image-registry-54cf5557c-9c668\" (UID: \"6fe75d93-6ca5-4051-91f9-a02766ce12cc\") " pod="openshift-image-registry/image-registry-54cf5557c-9c668"
Oct 30 12:51:54.467409 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.467384    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0448750b-7ffc-43c1-bdd8-198f73d940be-console-serving-cert\") pod \"console-64f798964-m7xhn\" (UID: \"0448750b-7ffc-43c1-bdd8-198f73d940be\") " pod="openshift-console/console-64f798964-m7xhn"
Oct 30 12:51:54.467648 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.467627    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f3dfe649-cc32-4dc7-8a09-b4790f17f2fb-cert\") pod \"ingress-canary-j8984\" (UID: \"f3dfe649-cc32-4dc7-8a09-b4790f17f2fb\") " pod="openshift-ingress-canary/ingress-canary-j8984"
Oct 30 12:51:54.468337 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.468313    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/62adaa28-5f97-4e1b-865f-d931ffd19d04-metrics-tls\") pod \"dns-default-ffvhz\" (UID: \"62adaa28-5f97-4e1b-865f-d931ffd19d04\") " pod="openshift-dns/dns-default-ffvhz"
Oct 30 12:51:54.469372 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.469351    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6fe75d93-6ca5-4051-91f9-a02766ce12cc-registry-tls\") pod \"image-registry-54cf5557c-9c668\" (UID: \"6fe75d93-6ca5-4051-91f9-a02766ce12cc\") " pod="openshift-image-registry/image-registry-54cf5557c-9c668"
Oct 30 12:51:54.471221 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.471198    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0448750b-7ffc-43c1-bdd8-198f73d940be-console-oauth-config\") pod \"console-64f798964-m7xhn\" (UID: \"0448750b-7ffc-43c1-bdd8-198f73d940be\") " pod="openshift-console/console-64f798964-m7xhn"
Oct 30 12:51:54.488818 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.488797    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6fe75d93-6ca5-4051-91f9-a02766ce12cc-bound-sa-token\") pod \"image-registry-54cf5557c-9c668\" (UID: \"6fe75d93-6ca5-4051-91f9-a02766ce12cc\") " pod="openshift-image-registry/image-registry-54cf5557c-9c668"
Oct 30 12:51:54.492442 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.492419    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj8jj\" (UniqueName: \"kubernetes.io/projected/f3dfe649-cc32-4dc7-8a09-b4790f17f2fb-kube-api-access-gj8jj\") pod \"ingress-canary-j8984\" (UID: \"f3dfe649-cc32-4dc7-8a09-b4790f17f2fb\") " pod="openshift-ingress-canary/ingress-canary-j8984"
Oct 30 12:51:54.493459 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.493436    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmv89\" (UniqueName: \"kubernetes.io/projected/03052418-91d3-44c9-95f2-f0b014dcbcbe-kube-api-access-tmv89\") pod \"downloads-6dd789c8b7-p566k\" (UID: \"03052418-91d3-44c9-95f2-f0b014dcbcbe\") " pod="openshift-console/downloads-6dd789c8b7-p566k"
Oct 30 12:51:54.493534 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.493496    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nggsh\" (UniqueName: \"kubernetes.io/projected/0448750b-7ffc-43c1-bdd8-198f73d940be-kube-api-access-nggsh\") pod \"console-64f798964-m7xhn\" (UID: \"0448750b-7ffc-43c1-bdd8-198f73d940be\") " pod="openshift-console/console-64f798964-m7xhn"
Oct 30 12:51:54.494294 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.494274    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq786\" (UniqueName: \"kubernetes.io/projected/029281be-58e5-4f0e-a7b7-bbb7804ef32e-kube-api-access-sq786\") pod \"insights-runtime-extractor-zn76j\" (UID: \"029281be-58e5-4f0e-a7b7-bbb7804ef32e\") " pod="openshift-insights/insights-runtime-extractor-zn76j"
Oct 30 12:51:54.494548 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.494521    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfgf7\" (UniqueName: \"kubernetes.io/projected/62adaa28-5f97-4e1b-865f-d931ffd19d04-kube-api-access-dfgf7\") pod \"dns-default-ffvhz\" (UID: \"62adaa28-5f97-4e1b-865f-d931ffd19d04\") " pod="openshift-dns/dns-default-ffvhz"
Oct 30 12:51:54.495075 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.495058    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkbsw\" (UniqueName: \"kubernetes.io/projected/6fe75d93-6ca5-4051-91f9-a02766ce12cc-kube-api-access-hkbsw\") pod \"image-registry-54cf5557c-9c668\" (UID: \"6fe75d93-6ca5-4051-91f9-a02766ce12cc\") " pod="openshift-image-registry/image-registry-54cf5557c-9c668"
Oct 30 12:51:54.515315 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.515297    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ffvhz"
Oct 30 12:51:54.521228 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.521207    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-p566k"
Oct 30 12:51:54.527161 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.527138    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64f798964-m7xhn"
Oct 30 12:51:54.534502 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.534365    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54cf5557c-9c668"
Oct 30 12:51:54.539022 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.539003    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-j8984"
Oct 30 12:51:54.592939 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.588764    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zn76j"
Oct 30 12:51:54.717868 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.717844    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-22gcb"]
Oct 30 12:51:54.720360 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:54.720337    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd7629fa_bfd4_44ab_b51b_39e5eea7032d.slice/crio-f554aa52e62fb668cdb53489c9c999288fce078f35354ce1b2a1bf46b155d282 WatchSource:0}: Error finding container f554aa52e62fb668cdb53489c9c999288fce078f35354ce1b2a1bf46b155d282: Status 404 returned error can't find the container with id f554aa52e62fb668cdb53489c9c999288fce078f35354ce1b2a1bf46b155d282
Oct 30 12:51:54.722350 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.722330    4364 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 12:51:54.828394 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.828330    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ffvhz"]
Oct 30 12:51:54.830661 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:54.830630    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62adaa28_5f97_4e1b_865f_d931ffd19d04.slice/crio-1349a0014d172fd83a70769e489e0056aa173d13d7f3beed90aaddf5efc94ad3 WatchSource:0}: Error finding container 1349a0014d172fd83a70769e489e0056aa173d13d7f3beed90aaddf5efc94ad3: Status 404 returned error can't find the container with id 1349a0014d172fd83a70769e489e0056aa173d13d7f3beed90aaddf5efc94ad3
Oct 30 12:51:54.840038 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.840013    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-p566k"]
Oct 30 12:51:54.843663 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:54.843634    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03052418_91d3_44c9_95f2_f0b014dcbcbe.slice/crio-6927627b36c36fc1391e9117fa88d574e022be542eeec1d165a36d72f24cf713 WatchSource:0}: Error finding container 6927627b36c36fc1391e9117fa88d574e022be542eeec1d165a36d72f24cf713: Status 404 returned error can't find the container with id 6927627b36c36fc1391e9117fa88d574e022be542eeec1d165a36d72f24cf713
Oct 30 12:51:54.856654 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.856630    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64f798964-m7xhn"]
Oct 30 12:51:54.884019 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.879239    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-j8984"]
Oct 30 12:51:54.884019 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.879503    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-54cf5557c-9c668"]
Oct 30 12:51:54.887328 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:54.886531    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fe75d93_6ca5_4051_91f9_a02766ce12cc.slice/crio-b5516549c2bcd351ded8b0e58665cae2426093546d01faf1783c80a63e822d82 WatchSource:0}: Error finding container b5516549c2bcd351ded8b0e58665cae2426093546d01faf1783c80a63e822d82: Status 404 returned error can't find the container with id b5516549c2bcd351ded8b0e58665cae2426093546d01faf1783c80a63e822d82
Oct 30 12:51:54.927111 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:54.927090    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zn76j"]
Oct 30 12:51:54.929402 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:54.929384    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod029281be_58e5_4f0e_a7b7_bbb7804ef32e.slice/crio-c3217a54ffae8b1dd446d8191d15ba58ef3fd432e71df12e45ccc5a00abef7f3 WatchSource:0}: Error finding container c3217a54ffae8b1dd446d8191d15ba58ef3fd432e71df12e45ccc5a00abef7f3: Status 404 returned error can't find the container with id c3217a54ffae8b1dd446d8191d15ba58ef3fd432e71df12e45ccc5a00abef7f3
Oct 30 12:51:55.030063 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:55.030028    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zn76j" event={"ID":"029281be-58e5-4f0e-a7b7-bbb7804ef32e","Type":"ContainerStarted","Data":"909758a8f7a46473cbc5fa888448f1b1361ed5277d7da929345b74210f651439"}
Oct 30 12:51:55.030198 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:55.030088    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zn76j" event={"ID":"029281be-58e5-4f0e-a7b7-bbb7804ef32e","Type":"ContainerStarted","Data":"c3217a54ffae8b1dd446d8191d15ba58ef3fd432e71df12e45ccc5a00abef7f3"}
Oct 30 12:51:55.031160 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:55.031140    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-j8984" event={"ID":"f3dfe649-cc32-4dc7-8a09-b4790f17f2fb","Type":"ContainerStarted","Data":"add9d2dda48247d56bec690bd534d721c3b5083c9c63873bc5978fa01edeafec"}
Oct 30 12:51:55.032505 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:55.032487    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54cf5557c-9c668" event={"ID":"6fe75d93-6ca5-4051-91f9-a02766ce12cc","Type":"ContainerStarted","Data":"b103f70580cd6bd2f998d52ec41fce4dfa802f49fae4183024c7bfcda4c7df86"}
Oct 30 12:51:55.032637 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:55.032518    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54cf5557c-9c668" event={"ID":"6fe75d93-6ca5-4051-91f9-a02766ce12cc","Type":"ContainerStarted","Data":"b5516549c2bcd351ded8b0e58665cae2426093546d01faf1783c80a63e822d82"}
Oct 30 12:51:55.032703 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:55.032632    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-54cf5557c-9c668"
Oct 30 12:51:55.033829 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:55.033805    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-p566k" event={"ID":"03052418-91d3-44c9-95f2-f0b014dcbcbe","Type":"ContainerStarted","Data":"6927627b36c36fc1391e9117fa88d574e022be542eeec1d165a36d72f24cf713"}
Oct 30 12:51:55.035009 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:55.034986    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ffvhz" event={"ID":"62adaa28-5f97-4e1b-865f-d931ffd19d04","Type":"ContainerStarted","Data":"1349a0014d172fd83a70769e489e0056aa173d13d7f3beed90aaddf5efc94ad3"}
Oct 30 12:51:55.036113 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:55.036087    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-22gcb" event={"ID":"cd7629fa-bfd4-44ab-b51b-39e5eea7032d","Type":"ContainerStarted","Data":"f554aa52e62fb668cdb53489c9c999288fce078f35354ce1b2a1bf46b155d282"}
Oct 30 12:51:55.037198 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:55.037171    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64f798964-m7xhn" event={"ID":"0448750b-7ffc-43c1-bdd8-198f73d940be","Type":"ContainerStarted","Data":"7d123d21a8b350dc7912aa521924ef0938c8c61e883fc7bd575856e96b618678"}
Oct 30 12:51:55.082835 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:55.082775    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-54cf5557c-9c668" podStartSLOduration=1.082759712 podStartE2EDuration="1.082759712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:51:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:55.082011935 +0000 UTC m=+14.248376592" watchObservedRunningTime="2026-10-30 12:51:55.082759712 +0000 UTC m=+14.249124364"
Oct 30 12:51:56.045352 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:56.045270    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zn76j" event={"ID":"029281be-58e5-4f0e-a7b7-bbb7804ef32e","Type":"ContainerStarted","Data":"606267dbd28478a7f8593a3fca451775af443d0f1cc3df51a47d309b76b15864"}
Oct 30 12:51:57.053592 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:57.053558    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-22gcb" event={"ID":"cd7629fa-bfd4-44ab-b51b-39e5eea7032d","Type":"ContainerStarted","Data":"55af4c9b939f39515b21c53ba8b48f2b65d2f37f4ee7569b79dcaebd9b18884e"}
Oct 30 12:51:57.054979 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:57.054521    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-22gcb"
Oct 30 12:51:57.061246 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:57.061224    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-22gcb"
Oct 30 12:51:57.180727 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:57.180676    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-22gcb" podStartSLOduration=1.6089246689999999 podStartE2EDuration="3.180661669s" totalImagesPullingTime="1.571737s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:51:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:57.098171003 +0000 UTC m=+16.264535651" watchObservedRunningTime="2026-10-30 12:51:57.180661669 +0000 UTC m=+16.347026311"
Oct 30 12:51:57.686545 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:57.685446    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-cmngs"]
Oct 30 12:51:57.688215 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:57.688194    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-cmngs"
Oct 30 12:51:57.709749 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:57.709724    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-cmngs"]
Oct 30 12:51:57.722157 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:57.721696    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 12:51:57.722157 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:57.721849    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 30 12:51:57.722157 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:57.721973    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 12:51:57.722157 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:57.722110    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 30 12:51:57.722989 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:57.722814    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-blhrz\""
Oct 30 12:51:57.722989 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:57.722919    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 12:51:57.803215 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:57.803183    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnwcx\" (UniqueName: \"kubernetes.io/projected/6bb9bf93-b7c0-4cf8-b1f3-f802c6d91d36-kube-api-access-xnwcx\") pod \"prometheus-operator-9cd89b74c-cmngs\" (UID: \"6bb9bf93-b7c0-4cf8-b1f3-f802c6d91d36\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-cmngs"
Oct 30 12:51:57.803339 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:57.803249    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6bb9bf93-b7c0-4cf8-b1f3-f802c6d91d36-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-cmngs\" (UID: \"6bb9bf93-b7c0-4cf8-b1f3-f802c6d91d36\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-cmngs"
Oct 30 12:51:57.803339 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:57.803291    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6bb9bf93-b7c0-4cf8-b1f3-f802c6d91d36-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-cmngs\" (UID: \"6bb9bf93-b7c0-4cf8-b1f3-f802c6d91d36\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-cmngs"
Oct 30 12:51:57.803339 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:57.803330    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6bb9bf93-b7c0-4cf8-b1f3-f802c6d91d36-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-cmngs\" (UID: \"6bb9bf93-b7c0-4cf8-b1f3-f802c6d91d36\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-cmngs"
Oct 30 12:51:57.904294 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:57.904265    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnwcx\" (UniqueName: \"kubernetes.io/projected/6bb9bf93-b7c0-4cf8-b1f3-f802c6d91d36-kube-api-access-xnwcx\") pod \"prometheus-operator-9cd89b74c-cmngs\" (UID: \"6bb9bf93-b7c0-4cf8-b1f3-f802c6d91d36\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-cmngs"
Oct 30 12:51:57.904402 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:57.904338    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6bb9bf93-b7c0-4cf8-b1f3-f802c6d91d36-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-cmngs\" (UID: \"6bb9bf93-b7c0-4cf8-b1f3-f802c6d91d36\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-cmngs"
Oct 30 12:51:57.904402 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:57.904384    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6bb9bf93-b7c0-4cf8-b1f3-f802c6d91d36-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-cmngs\" (UID: \"6bb9bf93-b7c0-4cf8-b1f3-f802c6d91d36\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-cmngs"
Oct 30 12:51:57.904515 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:57.904429    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6bb9bf93-b7c0-4cf8-b1f3-f802c6d91d36-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-cmngs\" (UID: \"6bb9bf93-b7c0-4cf8-b1f3-f802c6d91d36\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-cmngs"
Oct 30 12:51:57.905455 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:57.905383    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6bb9bf93-b7c0-4cf8-b1f3-f802c6d91d36-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-cmngs\" (UID: \"6bb9bf93-b7c0-4cf8-b1f3-f802c6d91d36\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-cmngs"
Oct 30 12:51:57.908779 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:57.908735    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6bb9bf93-b7c0-4cf8-b1f3-f802c6d91d36-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-cmngs\" (UID: \"6bb9bf93-b7c0-4cf8-b1f3-f802c6d91d36\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-cmngs"
Oct 30 12:51:57.909007 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:57.908985    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6bb9bf93-b7c0-4cf8-b1f3-f802c6d91d36-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-cmngs\" (UID: \"6bb9bf93-b7c0-4cf8-b1f3-f802c6d91d36\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-cmngs"
Oct 30 12:51:57.930473 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:57.930278    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnwcx\" (UniqueName: \"kubernetes.io/projected/6bb9bf93-b7c0-4cf8-b1f3-f802c6d91d36-kube-api-access-xnwcx\") pod \"prometheus-operator-9cd89b74c-cmngs\" (UID: \"6bb9bf93-b7c0-4cf8-b1f3-f802c6d91d36\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-cmngs"
Oct 30 12:51:57.980628 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:57.980591    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7mvnz_c83bcf7b-a1d6-457b-9d7c-bb03c2cbd449/dns-node-resolver/0.log"
Oct 30 12:51:58.007477 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:58.007295    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-cmngs"
Oct 30 12:51:58.067793 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:58.066935    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zn76j" event={"ID":"029281be-58e5-4f0e-a7b7-bbb7804ef32e","Type":"ContainerStarted","Data":"8033faee4038fd46a5e27acfc3b94ee08ca2512f516ea0fe65ca4e104d39bfaa"}
Oct 30 12:51:58.073249 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:58.073222    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ffvhz" event={"ID":"62adaa28-5f97-4e1b-865f-d931ffd19d04","Type":"ContainerStarted","Data":"62244a59ad76e85c9f4c1251ff05581d3b125673ae7bba921b0cc866dca12b96"}
Oct 30 12:51:58.073350 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:58.073275    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ffvhz" event={"ID":"62adaa28-5f97-4e1b-865f-d931ffd19d04","Type":"ContainerStarted","Data":"7c48f1710800fbd24c4663b66d06d02f478f44ba22f671466f62cb979fb38117"}
Oct 30 12:51:58.073350 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:58.073307    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-ffvhz"
Oct 30 12:51:58.118320 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:58.118277    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-zn76j" podStartSLOduration=1.34764002 podStartE2EDuration="4.11826509s" totalImagesPullingTime="2.77062507s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:51:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:58.117778713 +0000 UTC m=+17.284143364" watchObservedRunningTime="2026-10-30 12:51:58.11826509 +0000 UTC m=+17.284629762"
Oct 30 12:51:58.163650 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:58.163576    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-ffvhz" podStartSLOduration=1.32510866 podStartE2EDuration="4.163556889s" totalImagesPullingTime="2.838448229s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:51:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:58.162668172 +0000 UTC m=+17.329032821" watchObservedRunningTime="2026-10-30 12:51:58.163556889 +0000 UTC m=+17.329921544"
Oct 30 12:51:58.319003 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:58.318935    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-cmngs"]
Oct 30 12:51:58.322626 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:51:58.322566    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bb9bf93_b7c0_4cf8_b1f3_f802c6d91d36.slice/crio-2b19b7760cf125c3ac246a7f99f34006cfdda347b22a244a12ac3026e35ff0ab WatchSource:0}: Error finding container 2b19b7760cf125c3ac246a7f99f34006cfdda347b22a244a12ac3026e35ff0ab: Status 404 returned error can't find the container with id 2b19b7760cf125c3ac246a7f99f34006cfdda347b22a244a12ac3026e35ff0ab
Oct 30 12:51:58.573363 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:58.573291    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-54cf5557c-9c668_6fe75d93-6ca5-4051-91f9-a02766ce12cc/registry/0.log"
Oct 30 12:51:59.080949 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:59.080911    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-cmngs" event={"ID":"6bb9bf93-b7c0-4cf8-b1f3-f802c6d91d36","Type":"ContainerStarted","Data":"2b19b7760cf125c3ac246a7f99f34006cfdda347b22a244a12ac3026e35ff0ab"}
Oct 30 12:51:59.084641 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:59.084612    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-j8984" event={"ID":"f3dfe649-cc32-4dc7-8a09-b4790f17f2fb","Type":"ContainerStarted","Data":"eee5f1fad57d099795591629935a6b350897eedee6ab5dec3b76f4ddbbb5378c"}
Oct 30 12:51:59.376010 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:59.375896    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7pxn6_31d09b91-c973-4cf9-a7d8-d602ff9994f6/node-ca/0.log"
Oct 30 12:51:59.383292 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:59.383232    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-j8984" podStartSLOduration=1.354761336 podStartE2EDuration="5.38321482s" totalImagesPullingTime="4.028453484s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:51:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:59.129487542 +0000 UTC m=+18.295852192" watchObservedRunningTime="2026-10-30 12:51:59.38321482 +0000 UTC m=+18.549579468"
Oct 30 12:51:59.383764 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:59.383739    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-d4d8455c7-25km7"]
Oct 30 12:51:59.385835 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:59.385811    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d4d8455c7-25km7"
Oct 30 12:51:59.445396 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:59.445366    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 30 12:51:59.447131 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:59.447057    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d4d8455c7-25km7"]
Oct 30 12:51:59.514540 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:59.514509    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c-service-ca\") pod \"console-d4d8455c7-25km7\" (UID: \"4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c\") " pod="openshift-console/console-d4d8455c7-25km7"
Oct 30 12:51:59.514679 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:59.514558    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c-trusted-ca-bundle\") pod \"console-d4d8455c7-25km7\" (UID: \"4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c\") " pod="openshift-console/console-d4d8455c7-25km7"
Oct 30 12:51:59.514679 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:59.514586    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c-console-config\") pod \"console-d4d8455c7-25km7\" (UID: \"4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c\") " pod="openshift-console/console-d4d8455c7-25km7"
Oct 30 12:51:59.514787 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:59.514712    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c-oauth-serving-cert\") pod \"console-d4d8455c7-25km7\" (UID: \"4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c\") " pod="openshift-console/console-d4d8455c7-25km7"
Oct 30 12:51:59.514787 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:59.514747    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c-tmp\") pod \"console-d4d8455c7-25km7\" (UID: \"4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c\") " pod="openshift-console/console-d4d8455c7-25km7"
Oct 30 12:51:59.514887 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:59.514832    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c-console-serving-cert\") pod \"console-d4d8455c7-25km7\" (UID: \"4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c\") " pod="openshift-console/console-d4d8455c7-25km7"
Oct 30 12:51:59.514887 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:59.514875    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drkzh\" (UniqueName: \"kubernetes.io/projected/4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c-kube-api-access-drkzh\") pod \"console-d4d8455c7-25km7\" (UID: \"4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c\") " pod="openshift-console/console-d4d8455c7-25km7"
Oct 30 12:51:59.514958 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:59.514921    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c-console-oauth-config\") pod \"console-d4d8455c7-25km7\" (UID: \"4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c\") " pod="openshift-console/console-d4d8455c7-25km7"
Oct 30 12:51:59.615622 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:59.615576    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c-oauth-serving-cert\") pod \"console-d4d8455c7-25km7\" (UID: \"4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c\") " pod="openshift-console/console-d4d8455c7-25km7"
Oct 30 12:51:59.615784 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:59.615637    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c-tmp\") pod \"console-d4d8455c7-25km7\" (UID: \"4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c\") " pod="openshift-console/console-d4d8455c7-25km7"
Oct 30 12:51:59.615784 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:59.615676    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c-console-serving-cert\") pod \"console-d4d8455c7-25km7\" (UID: \"4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c\") " pod="openshift-console/console-d4d8455c7-25km7"
Oct 30 12:51:59.615784 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:59.615744    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drkzh\" (UniqueName: \"kubernetes.io/projected/4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c-kube-api-access-drkzh\") pod \"console-d4d8455c7-25km7\" (UID: \"4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c\") " pod="openshift-console/console-d4d8455c7-25km7"
Oct 30 12:51:59.615784 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:59.615776    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c-console-oauth-config\") pod \"console-d4d8455c7-25km7\" (UID: \"4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c\") " pod="openshift-console/console-d4d8455c7-25km7"
Oct 30 12:51:59.615994 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:59.615803    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c-service-ca\") pod \"console-d4d8455c7-25km7\" (UID: \"4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c\") " pod="openshift-console/console-d4d8455c7-25km7"
Oct 30 12:51:59.615994 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:59.615826    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c-trusted-ca-bundle\") pod \"console-d4d8455c7-25km7\" (UID: \"4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c\") " pod="openshift-console/console-d4d8455c7-25km7"
Oct 30 12:51:59.615994 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:59.615907    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c-console-config\") pod \"console-d4d8455c7-25km7\" (UID: \"4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c\") " pod="openshift-console/console-d4d8455c7-25km7"
Oct 30 12:51:59.616540 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:59.616521    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c-console-config\") pod \"console-d4d8455c7-25km7\" (UID: \"4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c\") " pod="openshift-console/console-d4d8455c7-25km7"
Oct 30 12:51:59.617042 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:59.617023    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c-oauth-serving-cert\") pod \"console-d4d8455c7-25km7\" (UID: \"4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c\") " pod="openshift-console/console-d4d8455c7-25km7"
Oct 30 12:51:59.617288 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:59.617270    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c-tmp\") pod \"console-d4d8455c7-25km7\" (UID: \"4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c\") " pod="openshift-console/console-d4d8455c7-25km7"
Oct 30 12:51:59.618232 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:59.618208    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c-service-ca\") pod \"console-d4d8455c7-25km7\" (UID: \"4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c\") " pod="openshift-console/console-d4d8455c7-25km7"
Oct 30 12:51:59.618635 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:59.618594    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c-trusted-ca-bundle\") pod \"console-d4d8455c7-25km7\" (UID: \"4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c\") " pod="openshift-console/console-d4d8455c7-25km7"
Oct 30 12:51:59.620201 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:59.620181    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c-console-serving-cert\") pod \"console-d4d8455c7-25km7\" (UID: \"4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c\") " pod="openshift-console/console-d4d8455c7-25km7"
Oct 30 12:51:59.620383 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:59.620364    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c-console-oauth-config\") pod \"console-d4d8455c7-25km7\" (UID: \"4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c\") " pod="openshift-console/console-d4d8455c7-25km7"
Oct 30 12:51:59.643581 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:59.643507    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drkzh\" (UniqueName: \"kubernetes.io/projected/4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c-kube-api-access-drkzh\") pod \"console-d4d8455c7-25km7\" (UID: \"4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c\") " pod="openshift-console/console-d4d8455c7-25km7"
Oct 30 12:51:59.700625 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:59.700341    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d4d8455c7-25km7"
Oct 30 12:51:59.973885 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:51:59.973856    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-j8984_f3dfe649-cc32-4dc7-8a09-b4790f17f2fb/serve-healthcheck-canary/0.log"
Oct 30 12:52:00.024131 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:00.024101    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d4d8455c7-25km7"]
Oct 30 12:52:00.040738 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:52:00.040695    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eb7de99_cbc7_4af4_82b7_92d3ffd73b7c.slice/crio-7de3d34d6d02bd0960d88da9619735ef60ac868bbb616de205a903e19b8b00dc WatchSource:0}: Error finding container 7de3d34d6d02bd0960d88da9619735ef60ac868bbb616de205a903e19b8b00dc: Status 404 returned error can't find the container with id 7de3d34d6d02bd0960d88da9619735ef60ac868bbb616de205a903e19b8b00dc
Oct 30 12:52:00.089460 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:00.089430    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-cmngs" event={"ID":"6bb9bf93-b7c0-4cf8-b1f3-f802c6d91d36","Type":"ContainerStarted","Data":"bfc47753bc428d6a8a3d264a08f667ec515a5e25d79d4580bfab1a0da67bbc05"}
Oct 30 12:52:00.089857 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:00.089494    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-cmngs" event={"ID":"6bb9bf93-b7c0-4cf8-b1f3-f802c6d91d36","Type":"ContainerStarted","Data":"42256be16c35b713878424941dd9a9a2e6e1a61ea8e8bdb85cd6c9107f686270"}
Oct 30 12:52:00.090947 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:00.090897    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d4d8455c7-25km7" event={"ID":"4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c","Type":"ContainerStarted","Data":"7de3d34d6d02bd0960d88da9619735ef60ac868bbb616de205a903e19b8b00dc"}
Oct 30 12:52:00.092635 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:00.092613    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64f798964-m7xhn" event={"ID":"0448750b-7ffc-43c1-bdd8-198f73d940be","Type":"ContainerStarted","Data":"e919605fe23c1151389421ed93967b2282b3544e56f5a37ad209b6ed71163a52"}
Oct 30 12:52:00.134656 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:00.134582    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-cmngs" podStartSLOduration=1.5952508600000002 podStartE2EDuration="3.134564352s" totalImagesPullingTime="1.539313492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:51:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:00.133971703 +0000 UTC m=+19.300336353" watchObservedRunningTime="2026-10-30 12:52:00.134564352 +0000 UTC m=+19.300929005"
Oct 30 12:52:00.183617 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:00.183524    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-64f798964-m7xhn" podStartSLOduration=1.549325292 podStartE2EDuration="6.183510228s" totalImagesPullingTime="4.634184936s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:51:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:00.182427251 +0000 UTC m=+19.348791900" watchObservedRunningTime="2026-10-30 12:52:00.183510228 +0000 UTC m=+19.349874874"
Oct 30 12:52:01.097666 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:01.097580    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d4d8455c7-25km7" event={"ID":"4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c","Type":"ContainerStarted","Data":"378179c75d50d7e6272494ca137f1e32195c260a99415c687dfbe4782558e23e"}
Oct 30 12:52:01.144356 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:01.144300    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-d4d8455c7-25km7" podStartSLOduration=2.144284557 podStartE2EDuration="2.144284557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:51:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:01.143892435 +0000 UTC m=+20.310257085" watchObservedRunningTime="2026-10-30 12:52:01.144284557 +0000 UTC m=+20.310649205"
Oct 30 12:52:02.400639 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:02.400592    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-275zk"]
Oct 30 12:52:02.403612 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:02.403586    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-275zk"
Oct 30 12:52:02.457857 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:02.457827    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 12:52:02.458051 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:02.458027    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 12:52:02.470228 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:02.470207    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 12:52:02.470355 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:02.470212    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-hxn87\""
Oct 30 12:52:02.537569 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:02.537542    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e10e4c9c-5a7b-4a00-8515-02467437a9d3-node-exporter-textfile\") pod \"node-exporter-275zk\" (UID: \"e10e4c9c-5a7b-4a00-8515-02467437a9d3\") " pod="openshift-monitoring/node-exporter-275zk"
Oct 30 12:52:02.537686 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:02.537590    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e10e4c9c-5a7b-4a00-8515-02467437a9d3-metrics-client-ca\") pod \"node-exporter-275zk\" (UID: \"e10e4c9c-5a7b-4a00-8515-02467437a9d3\") " pod="openshift-monitoring/node-exporter-275zk"
Oct 30 12:52:02.537686 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:02.537672    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e10e4c9c-5a7b-4a00-8515-02467437a9d3-node-exporter-wtmp\") pod \"node-exporter-275zk\" (UID: \"e10e4c9c-5a7b-4a00-8515-02467437a9d3\") " pod="openshift-monitoring/node-exporter-275zk"
Oct 30 12:52:02.537779 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:02.537725    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e10e4c9c-5a7b-4a00-8515-02467437a9d3-sys\") pod \"node-exporter-275zk\" (UID: \"e10e4c9c-5a7b-4a00-8515-02467437a9d3\") " pod="openshift-monitoring/node-exporter-275zk"
Oct 30 12:52:02.537779 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:02.537770    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e10e4c9c-5a7b-4a00-8515-02467437a9d3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-275zk\" (UID: \"e10e4c9c-5a7b-4a00-8515-02467437a9d3\") " pod="openshift-monitoring/node-exporter-275zk"
Oct 30 12:52:02.537872 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:02.537790    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e10e4c9c-5a7b-4a00-8515-02467437a9d3-node-exporter-accelerators-collector-config\") pod \"node-exporter-275zk\" (UID: \"e10e4c9c-5a7b-4a00-8515-02467437a9d3\") " pod="openshift-monitoring/node-exporter-275zk"
Oct 30 12:52:02.537929 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:02.537870    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e10e4c9c-5a7b-4a00-8515-02467437a9d3-root\") pod \"node-exporter-275zk\" (UID: \"e10e4c9c-5a7b-4a00-8515-02467437a9d3\") " pod="openshift-monitoring/node-exporter-275zk"
Oct 30 12:52:02.537929 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:02.537913    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99mrm\" (UniqueName: \"kubernetes.io/projected/e10e4c9c-5a7b-4a00-8515-02467437a9d3-kube-api-access-99mrm\") pod \"node-exporter-275zk\" (UID: \"e10e4c9c-5a7b-4a00-8515-02467437a9d3\") " pod="openshift-monitoring/node-exporter-275zk"
Oct 30 12:52:02.538029 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:02.537985    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e10e4c9c-5a7b-4a00-8515-02467437a9d3-node-exporter-tls\") pod \"node-exporter-275zk\" (UID: \"e10e4c9c-5a7b-4a00-8515-02467437a9d3\") " pod="openshift-monitoring/node-exporter-275zk"
Oct 30 12:52:02.638935 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:02.638905    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e10e4c9c-5a7b-4a00-8515-02467437a9d3-metrics-client-ca\") pod \"node-exporter-275zk\" (UID: \"e10e4c9c-5a7b-4a00-8515-02467437a9d3\") " pod="openshift-monitoring/node-exporter-275zk"
Oct 30 12:52:02.639127 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:02.638957    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e10e4c9c-5a7b-4a00-8515-02467437a9d3-node-exporter-wtmp\") pod \"node-exporter-275zk\" (UID: \"e10e4c9c-5a7b-4a00-8515-02467437a9d3\") " pod="openshift-monitoring/node-exporter-275zk"
Oct 30 12:52:02.639127 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:02.638982    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e10e4c9c-5a7b-4a00-8515-02467437a9d3-sys\") pod \"node-exporter-275zk\" (UID: \"e10e4c9c-5a7b-4a00-8515-02467437a9d3\") " pod="openshift-monitoring/node-exporter-275zk"
Oct 30 12:52:02.639127 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:02.639005    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e10e4c9c-5a7b-4a00-8515-02467437a9d3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-275zk\" (UID: \"e10e4c9c-5a7b-4a00-8515-02467437a9d3\") " pod="openshift-monitoring/node-exporter-275zk"
Oct 30 12:52:02.639127 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:02.639037    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e10e4c9c-5a7b-4a00-8515-02467437a9d3-node-exporter-accelerators-collector-config\") pod \"node-exporter-275zk\" (UID: \"e10e4c9c-5a7b-4a00-8515-02467437a9d3\") " pod="openshift-monitoring/node-exporter-275zk"
Oct 30 12:52:02.639127 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:02.639064    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e10e4c9c-5a7b-4a00-8515-02467437a9d3-root\") pod \"node-exporter-275zk\" (UID: \"e10e4c9c-5a7b-4a00-8515-02467437a9d3\") " pod="openshift-monitoring/node-exporter-275zk"
Oct 30 12:52:02.639127 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:02.639080    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99mrm\" (UniqueName: \"kubernetes.io/projected/e10e4c9c-5a7b-4a00-8515-02467437a9d3-kube-api-access-99mrm\") pod \"node-exporter-275zk\" (UID: \"e10e4c9c-5a7b-4a00-8515-02467437a9d3\") " pod="openshift-monitoring/node-exporter-275zk"
Oct 30 12:52:02.639127 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:02.639099    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e10e4c9c-5a7b-4a00-8515-02467437a9d3-node-exporter-tls\") pod \"node-exporter-275zk\" (UID: \"e10e4c9c-5a7b-4a00-8515-02467437a9d3\") " pod="openshift-monitoring/node-exporter-275zk"
Oct 30 12:52:02.639127 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:02.639097    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e10e4c9c-5a7b-4a00-8515-02467437a9d3-sys\") pod \"node-exporter-275zk\" (UID: \"e10e4c9c-5a7b-4a00-8515-02467437a9d3\") " pod="openshift-monitoring/node-exporter-275zk"
Oct 30 12:52:02.639127 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:02.639132    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e10e4c9c-5a7b-4a00-8515-02467437a9d3-node-exporter-textfile\") pod \"node-exporter-275zk\" (UID: \"e10e4c9c-5a7b-4a00-8515-02467437a9d3\") " pod="openshift-monitoring/node-exporter-275zk"
Oct 30 12:52:02.639661 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:02.639240    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e10e4c9c-5a7b-4a00-8515-02467437a9d3-node-exporter-wtmp\") pod \"node-exporter-275zk\" (UID: \"e10e4c9c-5a7b-4a00-8515-02467437a9d3\") " pod="openshift-monitoring/node-exporter-275zk"
Oct 30 12:52:02.639661 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:02.639270    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e10e4c9c-5a7b-4a00-8515-02467437a9d3-root\") pod \"node-exporter-275zk\" (UID: \"e10e4c9c-5a7b-4a00-8515-02467437a9d3\") " pod="openshift-monitoring/node-exporter-275zk"
Oct 30 12:52:02.639661 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:02.639371    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e10e4c9c-5a7b-4a00-8515-02467437a9d3-node-exporter-textfile\") pod \"node-exporter-275zk\" (UID: \"e10e4c9c-5a7b-4a00-8515-02467437a9d3\") " pod="openshift-monitoring/node-exporter-275zk"
Oct 30 12:52:02.639661 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:02.639618    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e10e4c9c-5a7b-4a00-8515-02467437a9d3-metrics-client-ca\") pod \"node-exporter-275zk\" (UID: \"e10e4c9c-5a7b-4a00-8515-02467437a9d3\") " pod="openshift-monitoring/node-exporter-275zk"
Oct 30 12:52:02.639999 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:02.639974    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e10e4c9c-5a7b-4a00-8515-02467437a9d3-node-exporter-accelerators-collector-config\") pod \"node-exporter-275zk\" (UID: \"e10e4c9c-5a7b-4a00-8515-02467437a9d3\") " pod="openshift-monitoring/node-exporter-275zk"
Oct 30 12:52:02.641828 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:02.641802    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e10e4c9c-5a7b-4a00-8515-02467437a9d3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-275zk\" (UID: \"e10e4c9c-5a7b-4a00-8515-02467437a9d3\") " pod="openshift-monitoring/node-exporter-275zk"
Oct 30 12:52:02.642115 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:02.642100    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e10e4c9c-5a7b-4a00-8515-02467437a9d3-node-exporter-tls\") pod \"node-exporter-275zk\" (UID: \"e10e4c9c-5a7b-4a00-8515-02467437a9d3\") " pod="openshift-monitoring/node-exporter-275zk"
Oct 30 12:52:02.685053 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:02.685004    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99mrm\" (UniqueName: \"kubernetes.io/projected/e10e4c9c-5a7b-4a00-8515-02467437a9d3-kube-api-access-99mrm\") pod \"node-exporter-275zk\" (UID: \"e10e4c9c-5a7b-4a00-8515-02467437a9d3\") " pod="openshift-monitoring/node-exporter-275zk"
Oct 30 12:52:02.718830 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:02.718805    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-275zk"
Oct 30 12:52:02.730137 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:52:02.730110    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode10e4c9c_5a7b_4a00_8515_02467437a9d3.slice/crio-4f592d8db4b433dce03fccc80fea31dfbae1438f322fbe243cf6110df068b741 WatchSource:0}: Error finding container 4f592d8db4b433dce03fccc80fea31dfbae1438f322fbe243cf6110df068b741: Status 404 returned error can't find the container with id 4f592d8db4b433dce03fccc80fea31dfbae1438f322fbe243cf6110df068b741
Oct 30 12:52:03.105828 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:03.105747    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-275zk" event={"ID":"e10e4c9c-5a7b-4a00-8515-02467437a9d3","Type":"ContainerStarted","Data":"4f592d8db4b433dce03fccc80fea31dfbae1438f322fbe243cf6110df068b741"}
Oct 30 12:52:04.111384 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:04.111304    4364 generic.go:356] "Generic (PLEG): container finished" podID="e10e4c9c-5a7b-4a00-8515-02467437a9d3" containerID="8f7eac18aaf429b68f2e8e742e6d81864644de287ad8f2f93a768e4be4534cdb" exitCode=0
Oct 30 12:52:04.111384 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:04.111349    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-275zk" event={"ID":"e10e4c9c-5a7b-4a00-8515-02467437a9d3","Type":"ContainerDied","Data":"8f7eac18aaf429b68f2e8e742e6d81864644de287ad8f2f93a768e4be4534cdb"}
Oct 30 12:52:04.527715 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:04.527680    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-64f798964-m7xhn"
Oct 30 12:52:04.527900 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:04.527746    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-64f798964-m7xhn"
Oct 30 12:52:04.533636 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:04.533614    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-64f798964-m7xhn"
Oct 30 12:52:05.117939 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:05.117905    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-275zk" event={"ID":"e10e4c9c-5a7b-4a00-8515-02467437a9d3","Type":"ContainerStarted","Data":"1c02a9cd6c8be164c58c7812106c032382416aca224367f54ad6fd377d363299"}
Oct 30 12:52:05.118405 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:05.117962    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-275zk" event={"ID":"e10e4c9c-5a7b-4a00-8515-02467437a9d3","Type":"ContainerStarted","Data":"f85e51a45a46a539ef29ad57fc68f91dca2dd4f27736ae7d508feae916a5f0a1"}
Oct 30 12:52:05.122311 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:05.122289    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-64f798964-m7xhn"
Oct 30 12:52:05.192392 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:05.192343    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-275zk" podStartSLOduration=2.032867534 podStartE2EDuration="3.192326607s" totalImagesPullingTime="1.081302463s" totalInitContainerRuntime="78.15661ms" isStatefulPod=true podCreationTimestamp="2026-10-30 12:52:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:05.178975383 +0000 UTC m=+24.345340031" watchObservedRunningTime="2026-10-30 12:52:05.192326607 +0000 UTC m=+24.358691254"
Oct 30 12:52:08.087826 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:08.087792    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-ffvhz"
Oct 30 12:52:09.321167 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:09.321143    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wpkrj"
Oct 30 12:52:09.474265 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:09.474223    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d4d8455c7-25km7"]
Oct 30 12:52:09.541624 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:09.541579    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-bb77db499-ntjp7"]
Oct 30 12:52:09.545433 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:09.545409    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bb77db499-ntjp7"
Oct 30 12:52:09.580056 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:09.579959    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bb77db499-ntjp7"]
Oct 30 12:52:09.588348 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:09.588324    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8e0c4f0e-91f5-4b63-81ac-26dc34c5d023-tmp\") pod \"console-bb77db499-ntjp7\" (UID: \"8e0c4f0e-91f5-4b63-81ac-26dc34c5d023\") " pod="openshift-console/console-bb77db499-ntjp7"
Oct 30 12:52:09.588489 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:09.588442    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8e0c4f0e-91f5-4b63-81ac-26dc34c5d023-oauth-serving-cert\") pod \"console-bb77db499-ntjp7\" (UID: \"8e0c4f0e-91f5-4b63-81ac-26dc34c5d023\") " pod="openshift-console/console-bb77db499-ntjp7"
Oct 30 12:52:09.588550 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:09.588500    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8e0c4f0e-91f5-4b63-81ac-26dc34c5d023-console-serving-cert\") pod \"console-bb77db499-ntjp7\" (UID: \"8e0c4f0e-91f5-4b63-81ac-26dc34c5d023\") " pod="openshift-console/console-bb77db499-ntjp7"
Oct 30 12:52:09.588550 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:09.588529    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8e0c4f0e-91f5-4b63-81ac-26dc34c5d023-console-config\") pod \"console-bb77db499-ntjp7\" (UID: \"8e0c4f0e-91f5-4b63-81ac-26dc34c5d023\") " pod="openshift-console/console-bb77db499-ntjp7"
Oct 30 12:52:09.588683 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:09.588565    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8e0c4f0e-91f5-4b63-81ac-26dc34c5d023-service-ca\") pod \"console-bb77db499-ntjp7\" (UID: \"8e0c4f0e-91f5-4b63-81ac-26dc34c5d023\") " pod="openshift-console/console-bb77db499-ntjp7"
Oct 30 12:52:09.588683 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:09.588619    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e0c4f0e-91f5-4b63-81ac-26dc34c5d023-trusted-ca-bundle\") pod \"console-bb77db499-ntjp7\" (UID: \"8e0c4f0e-91f5-4b63-81ac-26dc34c5d023\") " pod="openshift-console/console-bb77db499-ntjp7"
Oct 30 12:52:09.588683 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:09.588645    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8e0c4f0e-91f5-4b63-81ac-26dc34c5d023-console-oauth-config\") pod \"console-bb77db499-ntjp7\" (UID: \"8e0c4f0e-91f5-4b63-81ac-26dc34c5d023\") " pod="openshift-console/console-bb77db499-ntjp7"
Oct 30 12:52:09.588683 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:09.588674    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzll4\" (UniqueName: \"kubernetes.io/projected/8e0c4f0e-91f5-4b63-81ac-26dc34c5d023-kube-api-access-fzll4\") pod \"console-bb77db499-ntjp7\" (UID: \"8e0c4f0e-91f5-4b63-81ac-26dc34c5d023\") " pod="openshift-console/console-bb77db499-ntjp7"
Oct 30 12:52:09.689095 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:09.689058    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8e0c4f0e-91f5-4b63-81ac-26dc34c5d023-tmp\") pod \"console-bb77db499-ntjp7\" (UID: \"8e0c4f0e-91f5-4b63-81ac-26dc34c5d023\") " pod="openshift-console/console-bb77db499-ntjp7"
Oct 30 12:52:09.689263 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:09.689128    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8e0c4f0e-91f5-4b63-81ac-26dc34c5d023-oauth-serving-cert\") pod \"console-bb77db499-ntjp7\" (UID: \"8e0c4f0e-91f5-4b63-81ac-26dc34c5d023\") " pod="openshift-console/console-bb77db499-ntjp7"
Oct 30 12:52:09.689263 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:09.689175    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8e0c4f0e-91f5-4b63-81ac-26dc34c5d023-console-serving-cert\") pod \"console-bb77db499-ntjp7\" (UID: \"8e0c4f0e-91f5-4b63-81ac-26dc34c5d023\") " pod="openshift-console/console-bb77db499-ntjp7"
Oct 30 12:52:09.689263 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:09.689196    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8e0c4f0e-91f5-4b63-81ac-26dc34c5d023-console-config\") pod \"console-bb77db499-ntjp7\" (UID: \"8e0c4f0e-91f5-4b63-81ac-26dc34c5d023\") " pod="openshift-console/console-bb77db499-ntjp7"
Oct 30 12:52:09.689263 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:09.689230    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8e0c4f0e-91f5-4b63-81ac-26dc34c5d023-service-ca\") pod \"console-bb77db499-ntjp7\" (UID: \"8e0c4f0e-91f5-4b63-81ac-26dc34c5d023\") " pod="openshift-console/console-bb77db499-ntjp7"
Oct 30 12:52:09.689465 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:09.689264    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e0c4f0e-91f5-4b63-81ac-26dc34c5d023-trusted-ca-bundle\") pod \"console-bb77db499-ntjp7\" (UID: \"8e0c4f0e-91f5-4b63-81ac-26dc34c5d023\") " pod="openshift-console/console-bb77db499-ntjp7"
Oct 30 12:52:09.689465 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:09.689292    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8e0c4f0e-91f5-4b63-81ac-26dc34c5d023-console-oauth-config\") pod \"console-bb77db499-ntjp7\" (UID: \"8e0c4f0e-91f5-4b63-81ac-26dc34c5d023\") " pod="openshift-console/console-bb77db499-ntjp7"
Oct 30 12:52:09.689465 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:09.689325    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzll4\" (UniqueName: \"kubernetes.io/projected/8e0c4f0e-91f5-4b63-81ac-26dc34c5d023-kube-api-access-fzll4\") pod \"console-bb77db499-ntjp7\" (UID: \"8e0c4f0e-91f5-4b63-81ac-26dc34c5d023\") " pod="openshift-console/console-bb77db499-ntjp7"
Oct 30 12:52:09.689622 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:09.689474    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8e0c4f0e-91f5-4b63-81ac-26dc34c5d023-tmp\") pod \"console-bb77db499-ntjp7\" (UID: \"8e0c4f0e-91f5-4b63-81ac-26dc34c5d023\") " pod="openshift-console/console-bb77db499-ntjp7"
Oct 30 12:52:09.690082 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:09.690056    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8e0c4f0e-91f5-4b63-81ac-26dc34c5d023-console-config\") pod \"console-bb77db499-ntjp7\" (UID: \"8e0c4f0e-91f5-4b63-81ac-26dc34c5d023\") " pod="openshift-console/console-bb77db499-ntjp7"
Oct 30 12:52:09.690244 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:09.690220    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8e0c4f0e-91f5-4b63-81ac-26dc34c5d023-service-ca\") pod \"console-bb77db499-ntjp7\" (UID: \"8e0c4f0e-91f5-4b63-81ac-26dc34c5d023\") " pod="openshift-console/console-bb77db499-ntjp7"
Oct 30 12:52:09.690621 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:09.690586    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8e0c4f0e-91f5-4b63-81ac-26dc34c5d023-oauth-serving-cert\") pod \"console-bb77db499-ntjp7\" (UID: \"8e0c4f0e-91f5-4b63-81ac-26dc34c5d023\") " pod="openshift-console/console-bb77db499-ntjp7"
Oct 30 12:52:09.691067 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:09.691037    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e0c4f0e-91f5-4b63-81ac-26dc34c5d023-trusted-ca-bundle\") pod \"console-bb77db499-ntjp7\" (UID: \"8e0c4f0e-91f5-4b63-81ac-26dc34c5d023\") " pod="openshift-console/console-bb77db499-ntjp7"
Oct 30 12:52:09.693114 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:09.693094    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8e0c4f0e-91f5-4b63-81ac-26dc34c5d023-console-oauth-config\") pod \"console-bb77db499-ntjp7\" (UID: \"8e0c4f0e-91f5-4b63-81ac-26dc34c5d023\") " pod="openshift-console/console-bb77db499-ntjp7"
Oct 30 12:52:09.696095 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:09.696072    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8e0c4f0e-91f5-4b63-81ac-26dc34c5d023-console-serving-cert\") pod \"console-bb77db499-ntjp7\" (UID: \"8e0c4f0e-91f5-4b63-81ac-26dc34c5d023\") " pod="openshift-console/console-bb77db499-ntjp7"
Oct 30 12:52:09.701122 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:09.701101    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-d4d8455c7-25km7"
Oct 30 12:52:09.733113 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:09.733073    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzll4\" (UniqueName: \"kubernetes.io/projected/8e0c4f0e-91f5-4b63-81ac-26dc34c5d023-kube-api-access-fzll4\") pod \"console-bb77db499-ntjp7\" (UID: \"8e0c4f0e-91f5-4b63-81ac-26dc34c5d023\") " pod="openshift-console/console-bb77db499-ntjp7"
Oct 30 12:52:09.858661 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:09.858571    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bb77db499-ntjp7"
Oct 30 12:52:10.167768 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:10.167731    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bb77db499-ntjp7"]
Oct 30 12:52:10.170816 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:52:10.170775    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e0c4f0e_91f5_4b63_81ac_26dc34c5d023.slice/crio-5b1965a8d5f2d311bea9de2980d0e63f84eb9b36c2fabc83f99d0473dd7448e5 WatchSource:0}: Error finding container 5b1965a8d5f2d311bea9de2980d0e63f84eb9b36c2fabc83f99d0473dd7448e5: Status 404 returned error can't find the container with id 5b1965a8d5f2d311bea9de2980d0e63f84eb9b36c2fabc83f99d0473dd7448e5
Oct 30 12:52:11.142107 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:11.142069    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bb77db499-ntjp7" event={"ID":"8e0c4f0e-91f5-4b63-81ac-26dc34c5d023","Type":"ContainerStarted","Data":"fa40428711cbb68fdd2e935a9f993f6cffb2a7527bb467fb3f49f1dd2d66c405"}
Oct 30 12:52:11.142503 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:11.142129    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bb77db499-ntjp7" event={"ID":"8e0c4f0e-91f5-4b63-81ac-26dc34c5d023","Type":"ContainerStarted","Data":"5b1965a8d5f2d311bea9de2980d0e63f84eb9b36c2fabc83f99d0473dd7448e5"}
Oct 30 12:52:11.190200 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:11.190158    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-bb77db499-ntjp7" podStartSLOduration=2.190144118 podStartE2EDuration="2.190144118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:52:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:11.189285948 +0000 UTC m=+30.355650610" watchObservedRunningTime="2026-10-30 12:52:11.190144118 +0000 UTC m=+30.356508765"
Oct 30 12:52:11.508773 ip-10-0-137-6 kubenswrapper[4364]: E1030 12:52:11.508684    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 12:52:12.012139 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:12.012103    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vlfkx"
Oct 30 12:52:12.999474 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:12.999444    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-dlxch"
Oct 30 12:52:13.150722 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:13.150686    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-p566k" event={"ID":"03052418-91d3-44c9-95f2-f0b014dcbcbe","Type":"ContainerStarted","Data":"88a06cd0a937340b2066be72586725374707d7b51a7dcc144a01140c1269a3de"}
Oct 30 12:52:13.150990 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:13.150971    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-p566k"
Oct 30 12:52:13.152169 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:13.152145    4364 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-p566k container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.133.0.8:8080/\": dial tcp 10.133.0.8:8080: connect: connection refused" start-of-body=
Oct 30 12:52:13.152255 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:13.152198    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6dd789c8b7-p566k" podUID="03052418-91d3-44c9-95f2-f0b014dcbcbe" containerName="download-server" probeResult="failure" output="Get \"http://10.133.0.8:8080/\": dial tcp 10.133.0.8:8080: connect: connection refused"
Oct 30 12:52:13.200309 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:13.200261    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-p566k" podStartSLOduration=1.22648016 podStartE2EDuration="19.200246147s" totalImagesPullingTime="17.973765987s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:51:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:13.198745146 +0000 UTC m=+32.365109795" watchObservedRunningTime="2026-10-30 12:52:13.200246147 +0000 UTC m=+32.366610799"
Oct 30 12:52:14.163842 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:14.163287    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-p566k"
Oct 30 12:52:16.050491 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:16.050463    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-54cf5557c-9c668"
Oct 30 12:52:19.859352 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:19.859318    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-bb77db499-ntjp7"
Oct 30 12:52:19.859899 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:19.859381    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-bb77db499-ntjp7"
Oct 30 12:52:19.865074 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:19.865045    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-bb77db499-ntjp7"
Oct 30 12:52:20.185687 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:20.185587    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-bb77db499-ntjp7"
Oct 30 12:52:20.267386 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:20.267357    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64f798964-m7xhn"]
Oct 30 12:52:34.499628 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:34.499516    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-d4d8455c7-25km7" podUID="4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c" containerName="console" containerID="cri-o://378179c75d50d7e6272494ca137f1e32195c260a99415c687dfbe4782558e23e" gracePeriod=15
Oct 30 12:52:34.767361 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:34.767308    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d4d8455c7-25km7"
Oct 30 12:52:34.878200 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:34.878180    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c-console-config\" (UniqueName: \"kubernetes.io/configmap/4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c-console-config\") pod \"4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c\" (UID: \"4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c\") "
Oct 30 12:52:34.878331 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:34.878222    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c-service-ca\") pod \"4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c\" (UID: \"4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c\") "
Oct 30 12:52:34.878331 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:34.878245    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c-console-oauth-config\") pod \"4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c\" (UID: \"4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c\") "
Oct 30 12:52:34.878441 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:34.878424    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c-trusted-ca-bundle\") pod \"4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c\" (UID: \"4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c\") "
Oct 30 12:52:34.878497 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:34.878466    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c-tmp\" (UniqueName: \"kubernetes.io/empty-dir/4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c-tmp\") pod \"4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c\" (UID: \"4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c\") "
Oct 30 12:52:34.878551 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:34.878494    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c-kube-api-access-drkzh\" (UniqueName: \"kubernetes.io/projected/4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c-kube-api-access-drkzh\") pod \"4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c\" (UID: \"4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c\") "
Oct 30 12:52:34.878551 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:34.878501    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c-console-config" pod "4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c" (UID: "4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:52:34.878551 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:34.878531    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c-oauth-serving-cert\") pod \"4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c\" (UID: \"4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c\") "
Oct 30 12:52:34.878730 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:34.878579    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c-console-serving-cert\") pod \"4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c\" (UID: \"4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c\") "
Oct 30 12:52:34.878730 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:34.878587    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c-service-ca" pod "4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c" (UID: "4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:52:34.878841 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:34.878815    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c-tmp" pod "4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c" (UID: "4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:52:34.878884 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:34.878843    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c-trusted-ca-bundle" pod "4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c" (UID: "4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:52:34.878922 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:34.878882    4364 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c-console-config\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 12:52:34.878922 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:34.878900    4364 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c-service-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 12:52:34.879087 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:34.879059    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c-oauth-serving-cert" pod "4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c" (UID: "4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:52:34.880427 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:34.880406    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c-console-oauth-config" pod "4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c" (UID: "4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:52:34.880518 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:34.880500    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c-kube-api-access-drkzh" pod "4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c" (UID: "4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c"). InnerVolumeSpecName "kube-api-access-drkzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:52:34.880751 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:34.880730    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c-console-serving-cert" pod "4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c" (UID: "4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:52:34.979264 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:34.979246    4364 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c-trusted-ca-bundle\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 12:52:34.979264 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:34.979264    4364 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c-tmp\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 12:52:34.979378 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:34.979274    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drkzh\" (UniqueName: \"kubernetes.io/projected/4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c-kube-api-access-drkzh\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 12:52:34.979378 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:34.979284    4364 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c-oauth-serving-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 12:52:34.979378 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:34.979292    4364 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c-console-serving-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 12:52:34.979378 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:34.979300    4364 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c-console-oauth-config\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 12:52:35.232899 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:35.232866    4364 generic.go:356] "Generic (PLEG): container finished" podID="4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c" containerID="378179c75d50d7e6272494ca137f1e32195c260a99415c687dfbe4782558e23e" exitCode=0
Oct 30 12:52:35.233023 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:35.232924    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d4d8455c7-25km7"
Oct 30 12:52:35.233023 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:35.232927    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d4d8455c7-25km7" event={"ID":"4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c","Type":"ContainerDied","Data":"378179c75d50d7e6272494ca137f1e32195c260a99415c687dfbe4782558e23e"}
Oct 30 12:52:35.233023 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:35.232974    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d4d8455c7-25km7" event={"ID":"4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c","Type":"ContainerDied","Data":"7de3d34d6d02bd0960d88da9619735ef60ac868bbb616de205a903e19b8b00dc"}
Oct 30 12:52:35.233023 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:35.233003    4364 scope.go:122] "RemoveContainer" containerID="378179c75d50d7e6272494ca137f1e32195c260a99415c687dfbe4782558e23e"
Oct 30 12:52:35.241748 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:35.241728    4364 scope.go:122] "RemoveContainer" containerID="378179c75d50d7e6272494ca137f1e32195c260a99415c687dfbe4782558e23e"
Oct 30 12:52:35.242065 ip-10-0-137-6 kubenswrapper[4364]: E1030 12:52:35.242043    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"378179c75d50d7e6272494ca137f1e32195c260a99415c687dfbe4782558e23e\": container with ID starting with 378179c75d50d7e6272494ca137f1e32195c260a99415c687dfbe4782558e23e not found: ID does not exist" containerID="378179c75d50d7e6272494ca137f1e32195c260a99415c687dfbe4782558e23e"
Oct 30 12:52:35.242128 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:35.242073    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"378179c75d50d7e6272494ca137f1e32195c260a99415c687dfbe4782558e23e"} err="failed to get container status \"378179c75d50d7e6272494ca137f1e32195c260a99415c687dfbe4782558e23e\": rpc error: code = NotFound desc = could not find container \"378179c75d50d7e6272494ca137f1e32195c260a99415c687dfbe4782558e23e\": container with ID starting with 378179c75d50d7e6272494ca137f1e32195c260a99415c687dfbe4782558e23e not found: ID does not exist"
Oct 30 12:52:35.288091 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:35.288056    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d4d8455c7-25km7"]
Oct 30 12:52:35.292194 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:35.292170    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-d4d8455c7-25km7"]
Oct 30 12:52:36.966584 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:36.966536    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c" path="/var/lib/kubelet/pods/4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c/volumes"
Oct 30 12:52:41.506542 ip-10-0-137-6 kubenswrapper[4364]: E1030 12:52:41.506498    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 12:52:45.292094 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:45.292035    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-64f798964-m7xhn" podUID="0448750b-7ffc-43c1-bdd8-198f73d940be" containerName="console" containerID="cri-o://e919605fe23c1151389421ed93967b2282b3544e56f5a37ad209b6ed71163a52" gracePeriod=15
Oct 30 12:52:45.547948 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:45.547901    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64f798964-m7xhn"
Oct 30 12:52:45.648082 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:45.648057    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0448750b-7ffc-43c1-bdd8-198f73d940be-service-ca\" (UniqueName: \"kubernetes.io/configmap/0448750b-7ffc-43c1-bdd8-198f73d940be-service-ca\") pod \"0448750b-7ffc-43c1-bdd8-198f73d940be\" (UID: \"0448750b-7ffc-43c1-bdd8-198f73d940be\") "
Oct 30 12:52:45.648191 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:45.648101    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0448750b-7ffc-43c1-bdd8-198f73d940be-tmp\" (UniqueName: \"kubernetes.io/empty-dir/0448750b-7ffc-43c1-bdd8-198f73d940be-tmp\") pod \"0448750b-7ffc-43c1-bdd8-198f73d940be\" (UID: \"0448750b-7ffc-43c1-bdd8-198f73d940be\") "
Oct 30 12:52:45.648191 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:45.648133    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0448750b-7ffc-43c1-bdd8-198f73d940be-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0448750b-7ffc-43c1-bdd8-198f73d940be-oauth-serving-cert\") pod \"0448750b-7ffc-43c1-bdd8-198f73d940be\" (UID: \"0448750b-7ffc-43c1-bdd8-198f73d940be\") "
Oct 30 12:52:45.648297 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:45.648255    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0448750b-7ffc-43c1-bdd8-198f73d940be-kube-api-access-nggsh\" (UniqueName: \"kubernetes.io/projected/0448750b-7ffc-43c1-bdd8-198f73d940be-kube-api-access-nggsh\") pod \"0448750b-7ffc-43c1-bdd8-198f73d940be\" (UID: \"0448750b-7ffc-43c1-bdd8-198f73d940be\") "
Oct 30 12:52:45.648350 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:45.648299    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0448750b-7ffc-43c1-bdd8-198f73d940be-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0448750b-7ffc-43c1-bdd8-198f73d940be-console-serving-cert\") pod \"0448750b-7ffc-43c1-bdd8-198f73d940be\" (UID: \"0448750b-7ffc-43c1-bdd8-198f73d940be\") "
Oct 30 12:52:45.648350 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:45.648308    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0448750b-7ffc-43c1-bdd8-198f73d940be-tmp" pod "0448750b-7ffc-43c1-bdd8-198f73d940be" (UID: "0448750b-7ffc-43c1-bdd8-198f73d940be"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:52:45.648451 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:45.648374    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0448750b-7ffc-43c1-bdd8-198f73d940be-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0448750b-7ffc-43c1-bdd8-198f73d940be-console-oauth-config\") pod \"0448750b-7ffc-43c1-bdd8-198f73d940be\" (UID: \"0448750b-7ffc-43c1-bdd8-198f73d940be\") "
Oct 30 12:52:45.648451 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:45.648415    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0448750b-7ffc-43c1-bdd8-198f73d940be-console-config\" (UniqueName: \"kubernetes.io/configmap/0448750b-7ffc-43c1-bdd8-198f73d940be-console-config\") pod \"0448750b-7ffc-43c1-bdd8-198f73d940be\" (UID: \"0448750b-7ffc-43c1-bdd8-198f73d940be\") "
Oct 30 12:52:45.648588 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:45.648482    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0448750b-7ffc-43c1-bdd8-198f73d940be-oauth-serving-cert" pod "0448750b-7ffc-43c1-bdd8-198f73d940be" (UID: "0448750b-7ffc-43c1-bdd8-198f73d940be"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:52:45.648588 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:45.648488    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0448750b-7ffc-43c1-bdd8-198f73d940be-service-ca" pod "0448750b-7ffc-43c1-bdd8-198f73d940be" (UID: "0448750b-7ffc-43c1-bdd8-198f73d940be"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:52:45.648683 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:45.648639    4364 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0448750b-7ffc-43c1-bdd8-198f73d940be-tmp\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 12:52:45.648776 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:45.648656    4364 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0448750b-7ffc-43c1-bdd8-198f73d940be-oauth-serving-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 12:52:45.648830 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:45.648782    4364 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0448750b-7ffc-43c1-bdd8-198f73d940be-service-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 12:52:45.648867 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:45.648842    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0448750b-7ffc-43c1-bdd8-198f73d940be-console-config" pod "0448750b-7ffc-43c1-bdd8-198f73d940be" (UID: "0448750b-7ffc-43c1-bdd8-198f73d940be"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:52:45.650323 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:45.650299    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0448750b-7ffc-43c1-bdd8-198f73d940be-kube-api-access-nggsh" pod "0448750b-7ffc-43c1-bdd8-198f73d940be" (UID: "0448750b-7ffc-43c1-bdd8-198f73d940be"). InnerVolumeSpecName "kube-api-access-nggsh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:52:45.650773 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:45.650756    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0448750b-7ffc-43c1-bdd8-198f73d940be-console-serving-cert" pod "0448750b-7ffc-43c1-bdd8-198f73d940be" (UID: "0448750b-7ffc-43c1-bdd8-198f73d940be"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:52:45.650853 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:45.650779    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0448750b-7ffc-43c1-bdd8-198f73d940be-console-oauth-config" pod "0448750b-7ffc-43c1-bdd8-198f73d940be" (UID: "0448750b-7ffc-43c1-bdd8-198f73d940be"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:52:45.749285 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:45.749259    4364 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0448750b-7ffc-43c1-bdd8-198f73d940be-console-oauth-config\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 12:52:45.749285 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:45.749283    4364 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0448750b-7ffc-43c1-bdd8-198f73d940be-console-config\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 12:52:45.749412 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:45.749297    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nggsh\" (UniqueName: \"kubernetes.io/projected/0448750b-7ffc-43c1-bdd8-198f73d940be-kube-api-access-nggsh\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 12:52:45.749412 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:45.749310    4364 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0448750b-7ffc-43c1-bdd8-198f73d940be-console-serving-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 12:52:46.276580 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:46.276543    4364 generic.go:356] "Generic (PLEG): container finished" podID="0448750b-7ffc-43c1-bdd8-198f73d940be" containerID="e919605fe23c1151389421ed93967b2282b3544e56f5a37ad209b6ed71163a52" exitCode=0
Oct 30 12:52:46.276761 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:46.276616    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64f798964-m7xhn"
Oct 30 12:52:46.276761 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:46.276634    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64f798964-m7xhn" event={"ID":"0448750b-7ffc-43c1-bdd8-198f73d940be","Type":"ContainerDied","Data":"e919605fe23c1151389421ed93967b2282b3544e56f5a37ad209b6ed71163a52"}
Oct 30 12:52:46.276761 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:46.276687    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64f798964-m7xhn" event={"ID":"0448750b-7ffc-43c1-bdd8-198f73d940be","Type":"ContainerDied","Data":"7d123d21a8b350dc7912aa521924ef0938c8c61e883fc7bd575856e96b618678"}
Oct 30 12:52:46.276761 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:46.276707    4364 scope.go:122] "RemoveContainer" containerID="e919605fe23c1151389421ed93967b2282b3544e56f5a37ad209b6ed71163a52"
Oct 30 12:52:46.291475 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:46.291462    4364 scope.go:122] "RemoveContainer" containerID="e919605fe23c1151389421ed93967b2282b3544e56f5a37ad209b6ed71163a52"
Oct 30 12:52:46.291941 ip-10-0-137-6 kubenswrapper[4364]: E1030 12:52:46.291868    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e919605fe23c1151389421ed93967b2282b3544e56f5a37ad209b6ed71163a52\": container with ID starting with e919605fe23c1151389421ed93967b2282b3544e56f5a37ad209b6ed71163a52 not found: ID does not exist" containerID="e919605fe23c1151389421ed93967b2282b3544e56f5a37ad209b6ed71163a52"
Oct 30 12:52:46.291941 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:46.291901    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e919605fe23c1151389421ed93967b2282b3544e56f5a37ad209b6ed71163a52"} err="failed to get container status \"e919605fe23c1151389421ed93967b2282b3544e56f5a37ad209b6ed71163a52\": rpc error: code = NotFound desc = could not find container \"e919605fe23c1151389421ed93967b2282b3544e56f5a37ad209b6ed71163a52\": container with ID starting with e919605fe23c1151389421ed93967b2282b3544e56f5a37ad209b6ed71163a52 not found: ID does not exist"
Oct 30 12:52:46.332810 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:46.332758    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64f798964-m7xhn"]
Oct 30 12:52:46.347049 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:46.347005    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-64f798964-m7xhn"]
Oct 30 12:52:46.967178 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:46.967144    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0448750b-7ffc-43c1-bdd8-198f73d940be" path="/var/lib/kubelet/pods/0448750b-7ffc-43c1-bdd8-198f73d940be/volumes"
Oct 30 12:52:51.245556 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:51.245525    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5fd6fc79b8-4fh6v"]
Oct 30 12:52:51.245905 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:51.245755    4364 cpu_manager.go:414] "containerMap: removing container" podUID="0448750b-7ffc-43c1-bdd8-198f73d940be" containerName="console"
Oct 30 12:52:51.245905 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:51.245766    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0448750b-7ffc-43c1-bdd8-198f73d940be" containerName="console"
Oct 30 12:52:51.245905 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:51.245774    4364 cpu_manager.go:414] "containerMap: removing container" podUID="4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c" containerName="console"
Oct 30 12:52:51.245905 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:51.245778    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c" containerName="console"
Oct 30 12:52:51.245905 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:51.245820    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="0448750b-7ffc-43c1-bdd8-198f73d940be" containerName="console"
Oct 30 12:52:51.245905 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:51.245827    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="4eb7de99-cbc7-4af4-82b7-92d3ffd73b7c" containerName="console"
Oct 30 12:52:51.278883 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:51.278858    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5fd6fc79b8-4fh6v"]
Oct 30 12:52:51.279045 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:51.278955    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5fd6fc79b8-4fh6v"
Oct 30 12:52:51.295565 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:51.295541    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Oct 30 12:52:51.295713 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:51.295571    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Oct 30 12:52:51.295713 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:51.295643    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Oct 30 12:52:51.295713 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:51.295696    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Oct 30 12:52:51.295713 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:51.295700    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-fvt7w\""
Oct 30 12:52:51.318973 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:51.318950    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Oct 30 12:52:51.333512 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:51.333492    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Oct 30 12:52:51.383910 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:51.383886    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv6xd\" (UniqueName: \"kubernetes.io/projected/8d63267a-6f3e-4021-a91e-f342dab9e3aa-kube-api-access-vv6xd\") pod \"telemeter-client-5fd6fc79b8-4fh6v\" (UID: \"8d63267a-6f3e-4021-a91e-f342dab9e3aa\") " pod="openshift-monitoring/telemeter-client-5fd6fc79b8-4fh6v"
Oct 30 12:52:51.383999 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:51.383919    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/8d63267a-6f3e-4021-a91e-f342dab9e3aa-federate-client-tls\") pod \"telemeter-client-5fd6fc79b8-4fh6v\" (UID: \"8d63267a-6f3e-4021-a91e-f342dab9e3aa\") " pod="openshift-monitoring/telemeter-client-5fd6fc79b8-4fh6v"
Oct 30 12:52:51.383999 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:51.383942    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8d63267a-6f3e-4021-a91e-f342dab9e3aa-metrics-client-ca\") pod \"telemeter-client-5fd6fc79b8-4fh6v\" (UID: \"8d63267a-6f3e-4021-a91e-f342dab9e3aa\") " pod="openshift-monitoring/telemeter-client-5fd6fc79b8-4fh6v"
Oct 30 12:52:51.383999 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:51.383986    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/8d63267a-6f3e-4021-a91e-f342dab9e3aa-secret-telemeter-client\") pod \"telemeter-client-5fd6fc79b8-4fh6v\" (UID: \"8d63267a-6f3e-4021-a91e-f342dab9e3aa\") " pod="openshift-monitoring/telemeter-client-5fd6fc79b8-4fh6v"
Oct 30 12:52:51.384096 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:51.384024    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/8d63267a-6f3e-4021-a91e-f342dab9e3aa-telemeter-client-tls\") pod \"telemeter-client-5fd6fc79b8-4fh6v\" (UID: \"8d63267a-6f3e-4021-a91e-f342dab9e3aa\") " pod="openshift-monitoring/telemeter-client-5fd6fc79b8-4fh6v"
Oct 30 12:52:51.384096 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:51.384048    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d63267a-6f3e-4021-a91e-f342dab9e3aa-serving-certs-ca-bundle\") pod \"telemeter-client-5fd6fc79b8-4fh6v\" (UID: \"8d63267a-6f3e-4021-a91e-f342dab9e3aa\") " pod="openshift-monitoring/telemeter-client-5fd6fc79b8-4fh6v"
Oct 30 12:52:51.384096 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:51.384072    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8d63267a-6f3e-4021-a91e-f342dab9e3aa-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5fd6fc79b8-4fh6v\" (UID: \"8d63267a-6f3e-4021-a91e-f342dab9e3aa\") " pod="openshift-monitoring/telemeter-client-5fd6fc79b8-4fh6v"
Oct 30 12:52:51.384182 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:51.384094    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d63267a-6f3e-4021-a91e-f342dab9e3aa-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5fd6fc79b8-4fh6v\" (UID: \"8d63267a-6f3e-4021-a91e-f342dab9e3aa\") " pod="openshift-monitoring/telemeter-client-5fd6fc79b8-4fh6v"
Oct 30 12:52:51.484888 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:51.484861    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv6xd\" (UniqueName: \"kubernetes.io/projected/8d63267a-6f3e-4021-a91e-f342dab9e3aa-kube-api-access-vv6xd\") pod \"telemeter-client-5fd6fc79b8-4fh6v\" (UID: \"8d63267a-6f3e-4021-a91e-f342dab9e3aa\") " pod="openshift-monitoring/telemeter-client-5fd6fc79b8-4fh6v"
Oct 30 12:52:51.484988 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:51.484898    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/8d63267a-6f3e-4021-a91e-f342dab9e3aa-federate-client-tls\") pod \"telemeter-client-5fd6fc79b8-4fh6v\" (UID: \"8d63267a-6f3e-4021-a91e-f342dab9e3aa\") " pod="openshift-monitoring/telemeter-client-5fd6fc79b8-4fh6v"
Oct 30 12:52:51.484988 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:51.484922    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8d63267a-6f3e-4021-a91e-f342dab9e3aa-metrics-client-ca\") pod \"telemeter-client-5fd6fc79b8-4fh6v\" (UID: \"8d63267a-6f3e-4021-a91e-f342dab9e3aa\") " pod="openshift-monitoring/telemeter-client-5fd6fc79b8-4fh6v"
Oct 30 12:52:51.484988 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:51.484938    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/8d63267a-6f3e-4021-a91e-f342dab9e3aa-secret-telemeter-client\") pod \"telemeter-client-5fd6fc79b8-4fh6v\" (UID: \"8d63267a-6f3e-4021-a91e-f342dab9e3aa\") " pod="openshift-monitoring/telemeter-client-5fd6fc79b8-4fh6v"
Oct 30 12:52:51.484988 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:51.484958    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/8d63267a-6f3e-4021-a91e-f342dab9e3aa-telemeter-client-tls\") pod \"telemeter-client-5fd6fc79b8-4fh6v\" (UID: \"8d63267a-6f3e-4021-a91e-f342dab9e3aa\") " pod="openshift-monitoring/telemeter-client-5fd6fc79b8-4fh6v"
Oct 30 12:52:51.485181 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:51.485131    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d63267a-6f3e-4021-a91e-f342dab9e3aa-serving-certs-ca-bundle\") pod \"telemeter-client-5fd6fc79b8-4fh6v\" (UID: \"8d63267a-6f3e-4021-a91e-f342dab9e3aa\") " pod="openshift-monitoring/telemeter-client-5fd6fc79b8-4fh6v"
Oct 30 12:52:51.485229 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:51.485174    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8d63267a-6f3e-4021-a91e-f342dab9e3aa-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5fd6fc79b8-4fh6v\" (UID: \"8d63267a-6f3e-4021-a91e-f342dab9e3aa\") " pod="openshift-monitoring/telemeter-client-5fd6fc79b8-4fh6v"
Oct 30 12:52:51.485229 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:51.485215    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d63267a-6f3e-4021-a91e-f342dab9e3aa-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5fd6fc79b8-4fh6v\" (UID: \"8d63267a-6f3e-4021-a91e-f342dab9e3aa\") " pod="openshift-monitoring/telemeter-client-5fd6fc79b8-4fh6v"
Oct 30 12:52:51.485717 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:51.485664    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8d63267a-6f3e-4021-a91e-f342dab9e3aa-metrics-client-ca\") pod \"telemeter-client-5fd6fc79b8-4fh6v\" (UID: \"8d63267a-6f3e-4021-a91e-f342dab9e3aa\") " pod="openshift-monitoring/telemeter-client-5fd6fc79b8-4fh6v"
Oct 30 12:52:51.486031 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:51.486008    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d63267a-6f3e-4021-a91e-f342dab9e3aa-serving-certs-ca-bundle\") pod \"telemeter-client-5fd6fc79b8-4fh6v\" (UID: \"8d63267a-6f3e-4021-a91e-f342dab9e3aa\") " pod="openshift-monitoring/telemeter-client-5fd6fc79b8-4fh6v"
Oct 30 12:52:51.486162 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:51.486143    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d63267a-6f3e-4021-a91e-f342dab9e3aa-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5fd6fc79b8-4fh6v\" (UID: \"8d63267a-6f3e-4021-a91e-f342dab9e3aa\") " pod="openshift-monitoring/telemeter-client-5fd6fc79b8-4fh6v"
Oct 30 12:52:51.487528 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:51.487507    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/8d63267a-6f3e-4021-a91e-f342dab9e3aa-telemeter-client-tls\") pod \"telemeter-client-5fd6fc79b8-4fh6v\" (UID: \"8d63267a-6f3e-4021-a91e-f342dab9e3aa\") " pod="openshift-monitoring/telemeter-client-5fd6fc79b8-4fh6v"
Oct 30 12:52:51.487645 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:51.487553    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/8d63267a-6f3e-4021-a91e-f342dab9e3aa-secret-telemeter-client\") pod \"telemeter-client-5fd6fc79b8-4fh6v\" (UID: \"8d63267a-6f3e-4021-a91e-f342dab9e3aa\") " pod="openshift-monitoring/telemeter-client-5fd6fc79b8-4fh6v"
Oct 30 12:52:51.487842 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:51.487818    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8d63267a-6f3e-4021-a91e-f342dab9e3aa-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5fd6fc79b8-4fh6v\" (UID: \"8d63267a-6f3e-4021-a91e-f342dab9e3aa\") " pod="openshift-monitoring/telemeter-client-5fd6fc79b8-4fh6v"
Oct 30 12:52:51.487903 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:51.487865    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/8d63267a-6f3e-4021-a91e-f342dab9e3aa-federate-client-tls\") pod \"telemeter-client-5fd6fc79b8-4fh6v\" (UID: \"8d63267a-6f3e-4021-a91e-f342dab9e3aa\") " pod="openshift-monitoring/telemeter-client-5fd6fc79b8-4fh6v"
Oct 30 12:52:51.509974 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:51.509902    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv6xd\" (UniqueName: \"kubernetes.io/projected/8d63267a-6f3e-4021-a91e-f342dab9e3aa-kube-api-access-vv6xd\") pod \"telemeter-client-5fd6fc79b8-4fh6v\" (UID: \"8d63267a-6f3e-4021-a91e-f342dab9e3aa\") " pod="openshift-monitoring/telemeter-client-5fd6fc79b8-4fh6v"
Oct 30 12:52:51.631190 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:51.631163    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5fd6fc79b8-4fh6v"
Oct 30 12:52:51.982391 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:51.982364    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5fd6fc79b8-4fh6v"]
Oct 30 12:52:51.984617 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:52:51.984577    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d63267a_6f3e_4021_a91e_f342dab9e3aa.slice/crio-759a2188de501e6e0249374bcef1dbbc9e2e03000352ad0d53abe97ccd24bdf2 WatchSource:0}: Error finding container 759a2188de501e6e0249374bcef1dbbc9e2e03000352ad0d53abe97ccd24bdf2: Status 404 returned error can't find the container with id 759a2188de501e6e0249374bcef1dbbc9e2e03000352ad0d53abe97ccd24bdf2
Oct 30 12:52:52.304425 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:52.304354    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5fd6fc79b8-4fh6v" event={"ID":"8d63267a-6f3e-4021-a91e-f342dab9e3aa","Type":"ContainerStarted","Data":"759a2188de501e6e0249374bcef1dbbc9e2e03000352ad0d53abe97ccd24bdf2"}
Oct 30 12:52:54.313263 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:54.313191    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5fd6fc79b8-4fh6v" event={"ID":"8d63267a-6f3e-4021-a91e-f342dab9e3aa","Type":"ContainerStarted","Data":"6ef5e39f48fec67e55ab2c5152aff1bfed934744b1b68b768a80471ecf9242b9"}
Oct 30 12:52:56.322106 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:56.322068    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5fd6fc79b8-4fh6v" event={"ID":"8d63267a-6f3e-4021-a91e-f342dab9e3aa","Type":"ContainerStarted","Data":"a38f5ddcacd25b7ca885168b0949c081a3860a0bfe28a6cfc52848ce3a2e2a5f"}
Oct 30 12:52:56.322622 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:56.322126    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5fd6fc79b8-4fh6v" event={"ID":"8d63267a-6f3e-4021-a91e-f342dab9e3aa","Type":"ContainerStarted","Data":"d70f5b692009ad5e6e86d606cefc3507f327029ed56aa23f5eb820bcd28af693"}
Oct 30 12:52:56.400284 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:56.400221    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5fd6fc79b8-4fh6v" podStartSLOduration=2.08335989 podStartE2EDuration="5.400205896s" totalImagesPullingTime="3.316846006s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:52:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:56.385775509 +0000 UTC m=+75.552140155" watchObservedRunningTime="2026-10-30 12:52:56.400205896 +0000 UTC m=+75.566570627"
Oct 30 12:52:56.545291 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:56.545261    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-564f9c97b6-vhkwx"]
Oct 30 12:52:56.559592 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:56.559560    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-564f9c97b6-vhkwx"
Oct 30 12:52:56.570334 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:56.570308    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-564f9c97b6-vhkwx"]
Oct 30 12:52:56.622687 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:56.622659    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb5rv\" (UniqueName: \"kubernetes.io/projected/f2fd49ba-6ca2-4231-a303-f89b01276e70-kube-api-access-mb5rv\") pod \"console-564f9c97b6-vhkwx\" (UID: \"f2fd49ba-6ca2-4231-a303-f89b01276e70\") " pod="openshift-console/console-564f9c97b6-vhkwx"
Oct 30 12:52:56.622804 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:56.622691    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f2fd49ba-6ca2-4231-a303-f89b01276e70-console-oauth-config\") pod \"console-564f9c97b6-vhkwx\" (UID: \"f2fd49ba-6ca2-4231-a303-f89b01276e70\") " pod="openshift-console/console-564f9c97b6-vhkwx"
Oct 30 12:52:56.622804 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:56.622712    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2fd49ba-6ca2-4231-a303-f89b01276e70-tmp\") pod \"console-564f9c97b6-vhkwx\" (UID: \"f2fd49ba-6ca2-4231-a303-f89b01276e70\") " pod="openshift-console/console-564f9c97b6-vhkwx"
Oct 30 12:52:56.622804 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:56.622729    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f2fd49ba-6ca2-4231-a303-f89b01276e70-oauth-serving-cert\") pod \"console-564f9c97b6-vhkwx\" (UID: \"f2fd49ba-6ca2-4231-a303-f89b01276e70\") " pod="openshift-console/console-564f9c97b6-vhkwx"
Oct 30 12:52:56.622917 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:56.622854    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f2fd49ba-6ca2-4231-a303-f89b01276e70-service-ca\") pod \"console-564f9c97b6-vhkwx\" (UID: \"f2fd49ba-6ca2-4231-a303-f89b01276e70\") " pod="openshift-console/console-564f9c97b6-vhkwx"
Oct 30 12:52:56.622917 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:56.622886    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f2fd49ba-6ca2-4231-a303-f89b01276e70-console-serving-cert\") pod \"console-564f9c97b6-vhkwx\" (UID: \"f2fd49ba-6ca2-4231-a303-f89b01276e70\") " pod="openshift-console/console-564f9c97b6-vhkwx"
Oct 30 12:52:56.622988 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:56.622920    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f2fd49ba-6ca2-4231-a303-f89b01276e70-console-config\") pod \"console-564f9c97b6-vhkwx\" (UID: \"f2fd49ba-6ca2-4231-a303-f89b01276e70\") " pod="openshift-console/console-564f9c97b6-vhkwx"
Oct 30 12:52:56.622988 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:56.622980    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2fd49ba-6ca2-4231-a303-f89b01276e70-trusted-ca-bundle\") pod \"console-564f9c97b6-vhkwx\" (UID: \"f2fd49ba-6ca2-4231-a303-f89b01276e70\") " pod="openshift-console/console-564f9c97b6-vhkwx"
Oct 30 12:52:56.723585 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:56.723551    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb5rv\" (UniqueName: \"kubernetes.io/projected/f2fd49ba-6ca2-4231-a303-f89b01276e70-kube-api-access-mb5rv\") pod \"console-564f9c97b6-vhkwx\" (UID: \"f2fd49ba-6ca2-4231-a303-f89b01276e70\") " pod="openshift-console/console-564f9c97b6-vhkwx"
Oct 30 12:52:56.723717 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:56.723586    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f2fd49ba-6ca2-4231-a303-f89b01276e70-console-oauth-config\") pod \"console-564f9c97b6-vhkwx\" (UID: \"f2fd49ba-6ca2-4231-a303-f89b01276e70\") " pod="openshift-console/console-564f9c97b6-vhkwx"
Oct 30 12:52:56.723717 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:56.723654    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2fd49ba-6ca2-4231-a303-f89b01276e70-tmp\") pod \"console-564f9c97b6-vhkwx\" (UID: \"f2fd49ba-6ca2-4231-a303-f89b01276e70\") " pod="openshift-console/console-564f9c97b6-vhkwx"
Oct 30 12:52:56.723717 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:56.723680    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f2fd49ba-6ca2-4231-a303-f89b01276e70-oauth-serving-cert\") pod \"console-564f9c97b6-vhkwx\" (UID: \"f2fd49ba-6ca2-4231-a303-f89b01276e70\") " pod="openshift-console/console-564f9c97b6-vhkwx"
Oct 30 12:52:56.723717 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:56.723714    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f2fd49ba-6ca2-4231-a303-f89b01276e70-service-ca\") pod \"console-564f9c97b6-vhkwx\" (UID: \"f2fd49ba-6ca2-4231-a303-f89b01276e70\") " pod="openshift-console/console-564f9c97b6-vhkwx"
Oct 30 12:52:56.723905 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:56.723742    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f2fd49ba-6ca2-4231-a303-f89b01276e70-console-serving-cert\") pod \"console-564f9c97b6-vhkwx\" (UID: \"f2fd49ba-6ca2-4231-a303-f89b01276e70\") " pod="openshift-console/console-564f9c97b6-vhkwx"
Oct 30 12:52:56.723905 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:56.723778    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f2fd49ba-6ca2-4231-a303-f89b01276e70-console-config\") pod \"console-564f9c97b6-vhkwx\" (UID: \"f2fd49ba-6ca2-4231-a303-f89b01276e70\") " pod="openshift-console/console-564f9c97b6-vhkwx"
Oct 30 12:52:56.723905 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:56.723848    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2fd49ba-6ca2-4231-a303-f89b01276e70-trusted-ca-bundle\") pod \"console-564f9c97b6-vhkwx\" (UID: \"f2fd49ba-6ca2-4231-a303-f89b01276e70\") " pod="openshift-console/console-564f9c97b6-vhkwx"
Oct 30 12:52:56.724165 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:56.724110    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2fd49ba-6ca2-4231-a303-f89b01276e70-tmp\") pod \"console-564f9c97b6-vhkwx\" (UID: \"f2fd49ba-6ca2-4231-a303-f89b01276e70\") " pod="openshift-console/console-564f9c97b6-vhkwx"
Oct 30 12:52:56.724658 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:56.724594    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f2fd49ba-6ca2-4231-a303-f89b01276e70-service-ca\") pod \"console-564f9c97b6-vhkwx\" (UID: \"f2fd49ba-6ca2-4231-a303-f89b01276e70\") " pod="openshift-console/console-564f9c97b6-vhkwx"
Oct 30 12:52:56.724783 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:56.724627    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f2fd49ba-6ca2-4231-a303-f89b01276e70-oauth-serving-cert\") pod \"console-564f9c97b6-vhkwx\" (UID: \"f2fd49ba-6ca2-4231-a303-f89b01276e70\") " pod="openshift-console/console-564f9c97b6-vhkwx"
Oct 30 12:52:56.724853 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:56.724786    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f2fd49ba-6ca2-4231-a303-f89b01276e70-console-config\") pod \"console-564f9c97b6-vhkwx\" (UID: \"f2fd49ba-6ca2-4231-a303-f89b01276e70\") " pod="openshift-console/console-564f9c97b6-vhkwx"
Oct 30 12:52:56.724913 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:56.724886    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2fd49ba-6ca2-4231-a303-f89b01276e70-trusted-ca-bundle\") pod \"console-564f9c97b6-vhkwx\" (UID: \"f2fd49ba-6ca2-4231-a303-f89b01276e70\") " pod="openshift-console/console-564f9c97b6-vhkwx"
Oct 30 12:52:56.726492 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:56.726463    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f2fd49ba-6ca2-4231-a303-f89b01276e70-console-oauth-config\") pod \"console-564f9c97b6-vhkwx\" (UID: \"f2fd49ba-6ca2-4231-a303-f89b01276e70\") " pod="openshift-console/console-564f9c97b6-vhkwx"
Oct 30 12:52:56.726724 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:56.726703    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f2fd49ba-6ca2-4231-a303-f89b01276e70-console-serving-cert\") pod \"console-564f9c97b6-vhkwx\" (UID: \"f2fd49ba-6ca2-4231-a303-f89b01276e70\") " pod="openshift-console/console-564f9c97b6-vhkwx"
Oct 30 12:52:56.748811 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:56.748785    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb5rv\" (UniqueName: \"kubernetes.io/projected/f2fd49ba-6ca2-4231-a303-f89b01276e70-kube-api-access-mb5rv\") pod \"console-564f9c97b6-vhkwx\" (UID: \"f2fd49ba-6ca2-4231-a303-f89b01276e70\") " pod="openshift-console/console-564f9c97b6-vhkwx"
Oct 30 12:52:56.882474 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:56.882413    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-564f9c97b6-vhkwx"
Oct 30 12:52:57.207880 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:57.207821    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-564f9c97b6-vhkwx"]
Oct 30 12:52:57.210661 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:52:57.210639    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2fd49ba_6ca2_4231_a303_f89b01276e70.slice/crio-20d824f89d7d3e8bfc8dc5d316dbc3feedcafc00c790c793637d8c6f80633f63 WatchSource:0}: Error finding container 20d824f89d7d3e8bfc8dc5d316dbc3feedcafc00c790c793637d8c6f80633f63: Status 404 returned error can't find the container with id 20d824f89d7d3e8bfc8dc5d316dbc3feedcafc00c790c793637d8c6f80633f63
Oct 30 12:52:57.328823 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:57.328785    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-564f9c97b6-vhkwx" event={"ID":"f2fd49ba-6ca2-4231-a303-f89b01276e70","Type":"ContainerStarted","Data":"2b7b6ba5fe7b4c656f6cb3fb8671941a21b1ffcc27a905fecb88aaf089876b13"}
Oct 30 12:52:57.329242 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:57.328869    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-564f9c97b6-vhkwx" event={"ID":"f2fd49ba-6ca2-4231-a303-f89b01276e70","Type":"ContainerStarted","Data":"20d824f89d7d3e8bfc8dc5d316dbc3feedcafc00c790c793637d8c6f80633f63"}
Oct 30 12:52:57.375428 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:52:57.375374    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-564f9c97b6-vhkwx" podStartSLOduration=1.375361455 podStartE2EDuration="1.375361455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:52:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:57.374856221 +0000 UTC m=+76.541220868" watchObservedRunningTime="2026-10-30 12:52:57.375361455 +0000 UTC m=+76.541726102"
Oct 30 12:53:05.023349 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:05.023309    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-564f9c97b6-vhkwx"]
Oct 30 12:53:05.160789 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:05.160759    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-744d9f8b78-x4qbn"]
Oct 30 12:53:05.163364 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:05.163340    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-744d9f8b78-x4qbn"
Oct 30 12:53:05.198903 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:05.198882    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-744d9f8b78-x4qbn"]
Oct 30 12:53:05.282491 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:05.282433    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f7d7ad7-a4ec-42bb-b124-8b9ac9e7b1a7-trusted-ca-bundle\") pod \"console-744d9f8b78-x4qbn\" (UID: \"4f7d7ad7-a4ec-42bb-b124-8b9ac9e7b1a7\") " pod="openshift-console/console-744d9f8b78-x4qbn"
Oct 30 12:53:05.282491 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:05.282465    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4f7d7ad7-a4ec-42bb-b124-8b9ac9e7b1a7-tmp\") pod \"console-744d9f8b78-x4qbn\" (UID: \"4f7d7ad7-a4ec-42bb-b124-8b9ac9e7b1a7\") " pod="openshift-console/console-744d9f8b78-x4qbn"
Oct 30 12:53:05.282684 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:05.282548    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4f7d7ad7-a4ec-42bb-b124-8b9ac9e7b1a7-console-config\") pod \"console-744d9f8b78-x4qbn\" (UID: \"4f7d7ad7-a4ec-42bb-b124-8b9ac9e7b1a7\") " pod="openshift-console/console-744d9f8b78-x4qbn"
Oct 30 12:53:05.282684 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:05.282573    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4f7d7ad7-a4ec-42bb-b124-8b9ac9e7b1a7-console-serving-cert\") pod \"console-744d9f8b78-x4qbn\" (UID: \"4f7d7ad7-a4ec-42bb-b124-8b9ac9e7b1a7\") " pod="openshift-console/console-744d9f8b78-x4qbn"
Oct 30 12:53:05.282684 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:05.282594    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4f7d7ad7-a4ec-42bb-b124-8b9ac9e7b1a7-service-ca\") pod \"console-744d9f8b78-x4qbn\" (UID: \"4f7d7ad7-a4ec-42bb-b124-8b9ac9e7b1a7\") " pod="openshift-console/console-744d9f8b78-x4qbn"
Oct 30 12:53:05.282793 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:05.282696    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4f7d7ad7-a4ec-42bb-b124-8b9ac9e7b1a7-console-oauth-config\") pod \"console-744d9f8b78-x4qbn\" (UID: \"4f7d7ad7-a4ec-42bb-b124-8b9ac9e7b1a7\") " pod="openshift-console/console-744d9f8b78-x4qbn"
Oct 30 12:53:05.282793 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:05.282735    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l7wh\" (UniqueName: \"kubernetes.io/projected/4f7d7ad7-a4ec-42bb-b124-8b9ac9e7b1a7-kube-api-access-4l7wh\") pod \"console-744d9f8b78-x4qbn\" (UID: \"4f7d7ad7-a4ec-42bb-b124-8b9ac9e7b1a7\") " pod="openshift-console/console-744d9f8b78-x4qbn"
Oct 30 12:53:05.282793 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:05.282763    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4f7d7ad7-a4ec-42bb-b124-8b9ac9e7b1a7-oauth-serving-cert\") pod \"console-744d9f8b78-x4qbn\" (UID: \"4f7d7ad7-a4ec-42bb-b124-8b9ac9e7b1a7\") " pod="openshift-console/console-744d9f8b78-x4qbn"
Oct 30 12:53:05.383785 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:05.383758    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l7wh\" (UniqueName: \"kubernetes.io/projected/4f7d7ad7-a4ec-42bb-b124-8b9ac9e7b1a7-kube-api-access-4l7wh\") pod \"console-744d9f8b78-x4qbn\" (UID: \"4f7d7ad7-a4ec-42bb-b124-8b9ac9e7b1a7\") " pod="openshift-console/console-744d9f8b78-x4qbn"
Oct 30 12:53:05.383901 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:05.383793    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4f7d7ad7-a4ec-42bb-b124-8b9ac9e7b1a7-oauth-serving-cert\") pod \"console-744d9f8b78-x4qbn\" (UID: \"4f7d7ad7-a4ec-42bb-b124-8b9ac9e7b1a7\") " pod="openshift-console/console-744d9f8b78-x4qbn"
Oct 30 12:53:05.383901 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:05.383825    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f7d7ad7-a4ec-42bb-b124-8b9ac9e7b1a7-trusted-ca-bundle\") pod \"console-744d9f8b78-x4qbn\" (UID: \"4f7d7ad7-a4ec-42bb-b124-8b9ac9e7b1a7\") " pod="openshift-console/console-744d9f8b78-x4qbn"
Oct 30 12:53:05.383901 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:05.383850    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4f7d7ad7-a4ec-42bb-b124-8b9ac9e7b1a7-tmp\") pod \"console-744d9f8b78-x4qbn\" (UID: \"4f7d7ad7-a4ec-42bb-b124-8b9ac9e7b1a7\") " pod="openshift-console/console-744d9f8b78-x4qbn"
Oct 30 12:53:05.383901 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:05.383894    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4f7d7ad7-a4ec-42bb-b124-8b9ac9e7b1a7-console-config\") pod \"console-744d9f8b78-x4qbn\" (UID: \"4f7d7ad7-a4ec-42bb-b124-8b9ac9e7b1a7\") " pod="openshift-console/console-744d9f8b78-x4qbn"
Oct 30 12:53:05.384112 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:05.383918    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4f7d7ad7-a4ec-42bb-b124-8b9ac9e7b1a7-console-serving-cert\") pod \"console-744d9f8b78-x4qbn\" (UID: \"4f7d7ad7-a4ec-42bb-b124-8b9ac9e7b1a7\") " pod="openshift-console/console-744d9f8b78-x4qbn"
Oct 30 12:53:05.384112 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:05.383941    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4f7d7ad7-a4ec-42bb-b124-8b9ac9e7b1a7-service-ca\") pod \"console-744d9f8b78-x4qbn\" (UID: \"4f7d7ad7-a4ec-42bb-b124-8b9ac9e7b1a7\") " pod="openshift-console/console-744d9f8b78-x4qbn"
Oct 30 12:53:05.384112 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:05.383977    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4f7d7ad7-a4ec-42bb-b124-8b9ac9e7b1a7-console-oauth-config\") pod \"console-744d9f8b78-x4qbn\" (UID: \"4f7d7ad7-a4ec-42bb-b124-8b9ac9e7b1a7\") " pod="openshift-console/console-744d9f8b78-x4qbn"
Oct 30 12:53:05.384274 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:05.384211    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4f7d7ad7-a4ec-42bb-b124-8b9ac9e7b1a7-tmp\") pod \"console-744d9f8b78-x4qbn\" (UID: \"4f7d7ad7-a4ec-42bb-b124-8b9ac9e7b1a7\") " pod="openshift-console/console-744d9f8b78-x4qbn"
Oct 30 12:53:05.384682 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:05.384656    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4f7d7ad7-a4ec-42bb-b124-8b9ac9e7b1a7-oauth-serving-cert\") pod \"console-744d9f8b78-x4qbn\" (UID: \"4f7d7ad7-a4ec-42bb-b124-8b9ac9e7b1a7\") " pod="openshift-console/console-744d9f8b78-x4qbn"
Oct 30 12:53:05.384812 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:05.384794    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f7d7ad7-a4ec-42bb-b124-8b9ac9e7b1a7-trusted-ca-bundle\") pod \"console-744d9f8b78-x4qbn\" (UID: \"4f7d7ad7-a4ec-42bb-b124-8b9ac9e7b1a7\") " pod="openshift-console/console-744d9f8b78-x4qbn"
Oct 30 12:53:05.384882 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:05.384826    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4f7d7ad7-a4ec-42bb-b124-8b9ac9e7b1a7-service-ca\") pod \"console-744d9f8b78-x4qbn\" (UID: \"4f7d7ad7-a4ec-42bb-b124-8b9ac9e7b1a7\") " pod="openshift-console/console-744d9f8b78-x4qbn"
Oct 30 12:53:05.384882 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:05.384850    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4f7d7ad7-a4ec-42bb-b124-8b9ac9e7b1a7-console-config\") pod \"console-744d9f8b78-x4qbn\" (UID: \"4f7d7ad7-a4ec-42bb-b124-8b9ac9e7b1a7\") " pod="openshift-console/console-744d9f8b78-x4qbn"
Oct 30 12:53:05.386376 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:05.386353    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4f7d7ad7-a4ec-42bb-b124-8b9ac9e7b1a7-console-serving-cert\") pod \"console-744d9f8b78-x4qbn\" (UID: \"4f7d7ad7-a4ec-42bb-b124-8b9ac9e7b1a7\") " pod="openshift-console/console-744d9f8b78-x4qbn"
Oct 30 12:53:05.386456 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:05.386397    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4f7d7ad7-a4ec-42bb-b124-8b9ac9e7b1a7-console-oauth-config\") pod \"console-744d9f8b78-x4qbn\" (UID: \"4f7d7ad7-a4ec-42bb-b124-8b9ac9e7b1a7\") " pod="openshift-console/console-744d9f8b78-x4qbn"
Oct 30 12:53:05.408907 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:05.408883    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l7wh\" (UniqueName: \"kubernetes.io/projected/4f7d7ad7-a4ec-42bb-b124-8b9ac9e7b1a7-kube-api-access-4l7wh\") pod \"console-744d9f8b78-x4qbn\" (UID: \"4f7d7ad7-a4ec-42bb-b124-8b9ac9e7b1a7\") " pod="openshift-console/console-744d9f8b78-x4qbn"
Oct 30 12:53:05.474001 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:05.473969    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-744d9f8b78-x4qbn"
Oct 30 12:53:05.774379 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:05.774355    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-744d9f8b78-x4qbn"]
Oct 30 12:53:05.776972 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:53:05.776948    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f7d7ad7_a4ec_42bb_b124_8b9ac9e7b1a7.slice/crio-7d392c24c0601f2e93a32d0b541aa1c893ea65a1a987a061a58439a6d5438622 WatchSource:0}: Error finding container 7d392c24c0601f2e93a32d0b541aa1c893ea65a1a987a061a58439a6d5438622: Status 404 returned error can't find the container with id 7d392c24c0601f2e93a32d0b541aa1c893ea65a1a987a061a58439a6d5438622
Oct 30 12:53:06.360416 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:06.360376    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-744d9f8b78-x4qbn" event={"ID":"4f7d7ad7-a4ec-42bb-b124-8b9ac9e7b1a7","Type":"ContainerStarted","Data":"7bd4b933ee28e0001472b810f059fcdd26b4f8ca90d24ffcef34638271cfdf2c"}
Oct 30 12:53:06.360843 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:06.360437    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-744d9f8b78-x4qbn" event={"ID":"4f7d7ad7-a4ec-42bb-b124-8b9ac9e7b1a7","Type":"ContainerStarted","Data":"7d392c24c0601f2e93a32d0b541aa1c893ea65a1a987a061a58439a6d5438622"}
Oct 30 12:53:06.408556 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:06.408505    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-744d9f8b78-x4qbn" podStartSLOduration=1.40849019 podStartE2EDuration="1.40849019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:53:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:53:06.406837509 +0000 UTC m=+85.573202156" watchObservedRunningTime="2026-10-30 12:53:06.40849019 +0000 UTC m=+85.574854836"
Oct 30 12:53:06.883488 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:06.883460    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-564f9c97b6-vhkwx"
Oct 30 12:53:11.503168 ip-10-0-137-6 kubenswrapper[4364]: E1030 12:53:11.503131    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 12:53:15.474847 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:15.474800    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-744d9f8b78-x4qbn"
Oct 30 12:53:15.474847 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:15.474855    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-744d9f8b78-x4qbn"
Oct 30 12:53:15.479668 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:15.479627    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-744d9f8b78-x4qbn"
Oct 30 12:53:16.397420 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:16.397392    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-744d9f8b78-x4qbn"
Oct 30 12:53:16.480393 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:16.480369    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-bb77db499-ntjp7"]
Oct 30 12:53:30.042257 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:30.042189    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-564f9c97b6-vhkwx" podUID="f2fd49ba-6ca2-4231-a303-f89b01276e70" containerName="console" containerID="cri-o://2b7b6ba5fe7b4c656f6cb3fb8671941a21b1ffcc27a905fecb88aaf089876b13" gracePeriod=15
Oct 30 12:53:30.279716 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:30.279694    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-564f9c97b6-vhkwx"
Oct 30 12:53:30.354118 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:30.354061    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2fd49ba-6ca2-4231-a303-f89b01276e70-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2fd49ba-6ca2-4231-a303-f89b01276e70-trusted-ca-bundle\") pod \"f2fd49ba-6ca2-4231-a303-f89b01276e70\" (UID: \"f2fd49ba-6ca2-4231-a303-f89b01276e70\") "
Oct 30 12:53:30.354118 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:30.354115    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2fd49ba-6ca2-4231-a303-f89b01276e70-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2fd49ba-6ca2-4231-a303-f89b01276e70-tmp\") pod \"f2fd49ba-6ca2-4231-a303-f89b01276e70\" (UID: \"f2fd49ba-6ca2-4231-a303-f89b01276e70\") "
Oct 30 12:53:30.354275 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:30.354139    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2fd49ba-6ca2-4231-a303-f89b01276e70-console-config\" (UniqueName: \"kubernetes.io/configmap/f2fd49ba-6ca2-4231-a303-f89b01276e70-console-config\") pod \"f2fd49ba-6ca2-4231-a303-f89b01276e70\" (UID: \"f2fd49ba-6ca2-4231-a303-f89b01276e70\") "
Oct 30 12:53:30.354275 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:30.354164    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2fd49ba-6ca2-4231-a303-f89b01276e70-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f2fd49ba-6ca2-4231-a303-f89b01276e70-console-serving-cert\") pod \"f2fd49ba-6ca2-4231-a303-f89b01276e70\" (UID: \"f2fd49ba-6ca2-4231-a303-f89b01276e70\") "
Oct 30 12:53:30.354275 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:30.354192    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2fd49ba-6ca2-4231-a303-f89b01276e70-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f2fd49ba-6ca2-4231-a303-f89b01276e70-console-oauth-config\") pod \"f2fd49ba-6ca2-4231-a303-f89b01276e70\" (UID: \"f2fd49ba-6ca2-4231-a303-f89b01276e70\") "
Oct 30 12:53:30.354275 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:30.354207    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2fd49ba-6ca2-4231-a303-f89b01276e70-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2fd49ba-6ca2-4231-a303-f89b01276e70-service-ca\") pod \"f2fd49ba-6ca2-4231-a303-f89b01276e70\" (UID: \"f2fd49ba-6ca2-4231-a303-f89b01276e70\") "
Oct 30 12:53:30.354275 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:30.354236    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2fd49ba-6ca2-4231-a303-f89b01276e70-kube-api-access-mb5rv\" (UniqueName: \"kubernetes.io/projected/f2fd49ba-6ca2-4231-a303-f89b01276e70-kube-api-access-mb5rv\") pod \"f2fd49ba-6ca2-4231-a303-f89b01276e70\" (UID: \"f2fd49ba-6ca2-4231-a303-f89b01276e70\") "
Oct 30 12:53:30.354275 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:30.354257    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2fd49ba-6ca2-4231-a303-f89b01276e70-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f2fd49ba-6ca2-4231-a303-f89b01276e70-oauth-serving-cert\") pod \"f2fd49ba-6ca2-4231-a303-f89b01276e70\" (UID: \"f2fd49ba-6ca2-4231-a303-f89b01276e70\") "
Oct 30 12:53:30.354551 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:30.354470    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2fd49ba-6ca2-4231-a303-f89b01276e70-tmp" pod "f2fd49ba-6ca2-4231-a303-f89b01276e70" (UID: "f2fd49ba-6ca2-4231-a303-f89b01276e70"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:53:30.354551 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:30.354537    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2fd49ba-6ca2-4231-a303-f89b01276e70-trusted-ca-bundle" pod "f2fd49ba-6ca2-4231-a303-f89b01276e70" (UID: "f2fd49ba-6ca2-4231-a303-f89b01276e70"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:53:30.354652 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:30.354563    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2fd49ba-6ca2-4231-a303-f89b01276e70-console-config" pod "f2fd49ba-6ca2-4231-a303-f89b01276e70" (UID: "f2fd49ba-6ca2-4231-a303-f89b01276e70"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:53:30.354792 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:30.354767    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2fd49ba-6ca2-4231-a303-f89b01276e70-oauth-serving-cert" pod "f2fd49ba-6ca2-4231-a303-f89b01276e70" (UID: "f2fd49ba-6ca2-4231-a303-f89b01276e70"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:53:30.354977 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:30.354959    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2fd49ba-6ca2-4231-a303-f89b01276e70-service-ca" pod "f2fd49ba-6ca2-4231-a303-f89b01276e70" (UID: "f2fd49ba-6ca2-4231-a303-f89b01276e70"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:53:30.356404 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:30.356380    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2fd49ba-6ca2-4231-a303-f89b01276e70-console-oauth-config" pod "f2fd49ba-6ca2-4231-a303-f89b01276e70" (UID: "f2fd49ba-6ca2-4231-a303-f89b01276e70"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:53:30.356718 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:30.356688    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2fd49ba-6ca2-4231-a303-f89b01276e70-console-serving-cert" pod "f2fd49ba-6ca2-4231-a303-f89b01276e70" (UID: "f2fd49ba-6ca2-4231-a303-f89b01276e70"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:53:30.356718 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:30.356700    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2fd49ba-6ca2-4231-a303-f89b01276e70-kube-api-access-mb5rv" pod "f2fd49ba-6ca2-4231-a303-f89b01276e70" (UID: "f2fd49ba-6ca2-4231-a303-f89b01276e70"). InnerVolumeSpecName "kube-api-access-mb5rv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:53:30.449923 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:30.449895    4364 generic.go:356] "Generic (PLEG): container finished" podID="f2fd49ba-6ca2-4231-a303-f89b01276e70" containerID="2b7b6ba5fe7b4c656f6cb3fb8671941a21b1ffcc27a905fecb88aaf089876b13" exitCode=0
Oct 30 12:53:30.450004 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:30.449937    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-564f9c97b6-vhkwx"
Oct 30 12:53:30.450004 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:30.449974    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-564f9c97b6-vhkwx" event={"ID":"f2fd49ba-6ca2-4231-a303-f89b01276e70","Type":"ContainerDied","Data":"2b7b6ba5fe7b4c656f6cb3fb8671941a21b1ffcc27a905fecb88aaf089876b13"}
Oct 30 12:53:30.450068 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:30.450014    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-564f9c97b6-vhkwx" event={"ID":"f2fd49ba-6ca2-4231-a303-f89b01276e70","Type":"ContainerDied","Data":"20d824f89d7d3e8bfc8dc5d316dbc3feedcafc00c790c793637d8c6f80633f63"}
Oct 30 12:53:30.450068 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:30.450037    4364 scope.go:122] "RemoveContainer" containerID="2b7b6ba5fe7b4c656f6cb3fb8671941a21b1ffcc27a905fecb88aaf089876b13"
Oct 30 12:53:30.454739 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:30.454719    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mb5rv\" (UniqueName: \"kubernetes.io/projected/f2fd49ba-6ca2-4231-a303-f89b01276e70-kube-api-access-mb5rv\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 12:53:30.454739 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:30.454738    4364 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f2fd49ba-6ca2-4231-a303-f89b01276e70-oauth-serving-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 12:53:30.454885 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:30.454749    4364 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2fd49ba-6ca2-4231-a303-f89b01276e70-trusted-ca-bundle\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 12:53:30.454885 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:30.454757    4364 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2fd49ba-6ca2-4231-a303-f89b01276e70-tmp\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 12:53:30.454885 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:30.454767    4364 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f2fd49ba-6ca2-4231-a303-f89b01276e70-console-config\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 12:53:30.454885 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:30.454775    4364 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f2fd49ba-6ca2-4231-a303-f89b01276e70-console-serving-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 12:53:30.454885 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:30.454782    4364 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f2fd49ba-6ca2-4231-a303-f89b01276e70-console-oauth-config\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 12:53:30.454885 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:30.454790    4364 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f2fd49ba-6ca2-4231-a303-f89b01276e70-service-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 12:53:30.460896 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:30.460878    4364 scope.go:122] "RemoveContainer" containerID="2b7b6ba5fe7b4c656f6cb3fb8671941a21b1ffcc27a905fecb88aaf089876b13"
Oct 30 12:53:30.461277 ip-10-0-137-6 kubenswrapper[4364]: E1030 12:53:30.461259    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b7b6ba5fe7b4c656f6cb3fb8671941a21b1ffcc27a905fecb88aaf089876b13\": container with ID starting with 2b7b6ba5fe7b4c656f6cb3fb8671941a21b1ffcc27a905fecb88aaf089876b13 not found: ID does not exist" containerID="2b7b6ba5fe7b4c656f6cb3fb8671941a21b1ffcc27a905fecb88aaf089876b13"
Oct 30 12:53:30.461337 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:30.461287    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b7b6ba5fe7b4c656f6cb3fb8671941a21b1ffcc27a905fecb88aaf089876b13"} err="failed to get container status \"2b7b6ba5fe7b4c656f6cb3fb8671941a21b1ffcc27a905fecb88aaf089876b13\": rpc error: code = NotFound desc = could not find container \"2b7b6ba5fe7b4c656f6cb3fb8671941a21b1ffcc27a905fecb88aaf089876b13\": container with ID starting with 2b7b6ba5fe7b4c656f6cb3fb8671941a21b1ffcc27a905fecb88aaf089876b13 not found: ID does not exist"
Oct 30 12:53:30.504978 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:30.504956    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-564f9c97b6-vhkwx"]
Oct 30 12:53:30.511150 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:30.511120    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-564f9c97b6-vhkwx"]
Oct 30 12:53:30.966702 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:30.966677    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2fd49ba-6ca2-4231-a303-f89b01276e70" path="/var/lib/kubelet/pods/f2fd49ba-6ca2-4231-a303-f89b01276e70/volumes"
Oct 30 12:53:41.501189 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:41.501123    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-bb77db499-ntjp7" podUID="8e0c4f0e-91f5-4b63-81ac-26dc34c5d023" containerName="console" containerID="cri-o://fa40428711cbb68fdd2e935a9f993f6cffb2a7527bb467fb3f49f1dd2d66c405" gracePeriod=15
Oct 30 12:53:41.502681 ip-10-0-137-6 kubenswrapper[4364]: E1030 12:53:41.502653    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 12:53:41.740455 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:41.740433    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bb77db499-ntjp7"
Oct 30 12:53:41.822260 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:41.822190    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e0c4f0e-91f5-4b63-81ac-26dc34c5d023-kube-api-access-fzll4\" (UniqueName: \"kubernetes.io/projected/8e0c4f0e-91f5-4b63-81ac-26dc34c5d023-kube-api-access-fzll4\") pod \"8e0c4f0e-91f5-4b63-81ac-26dc34c5d023\" (UID: \"8e0c4f0e-91f5-4b63-81ac-26dc34c5d023\") "
Oct 30 12:53:41.822260 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:41.822227    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e0c4f0e-91f5-4b63-81ac-26dc34c5d023-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8e0c4f0e-91f5-4b63-81ac-26dc34c5d023-oauth-serving-cert\") pod \"8e0c4f0e-91f5-4b63-81ac-26dc34c5d023\" (UID: \"8e0c4f0e-91f5-4b63-81ac-26dc34c5d023\") "
Oct 30 12:53:41.822457 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:41.822278    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e0c4f0e-91f5-4b63-81ac-26dc34c5d023-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8e0c4f0e-91f5-4b63-81ac-26dc34c5d023-console-oauth-config\") pod \"8e0c4f0e-91f5-4b63-81ac-26dc34c5d023\" (UID: \"8e0c4f0e-91f5-4b63-81ac-26dc34c5d023\") "
Oct 30 12:53:41.822457 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:41.822329    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e0c4f0e-91f5-4b63-81ac-26dc34c5d023-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e0c4f0e-91f5-4b63-81ac-26dc34c5d023-service-ca\") pod \"8e0c4f0e-91f5-4b63-81ac-26dc34c5d023\" (UID: \"8e0c4f0e-91f5-4b63-81ac-26dc34c5d023\") "
Oct 30 12:53:41.822457 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:41.822355    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e0c4f0e-91f5-4b63-81ac-26dc34c5d023-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e0c4f0e-91f5-4b63-81ac-26dc34c5d023-trusted-ca-bundle\") pod \"8e0c4f0e-91f5-4b63-81ac-26dc34c5d023\" (UID: \"8e0c4f0e-91f5-4b63-81ac-26dc34c5d023\") "
Oct 30 12:53:41.822457 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:41.822383    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e0c4f0e-91f5-4b63-81ac-26dc34c5d023-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8e0c4f0e-91f5-4b63-81ac-26dc34c5d023-console-serving-cert\") pod \"8e0c4f0e-91f5-4b63-81ac-26dc34c5d023\" (UID: \"8e0c4f0e-91f5-4b63-81ac-26dc34c5d023\") "
Oct 30 12:53:41.822457 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:41.822431    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e0c4f0e-91f5-4b63-81ac-26dc34c5d023-tmp\" (UniqueName: \"kubernetes.io/empty-dir/8e0c4f0e-91f5-4b63-81ac-26dc34c5d023-tmp\") pod \"8e0c4f0e-91f5-4b63-81ac-26dc34c5d023\" (UID: \"8e0c4f0e-91f5-4b63-81ac-26dc34c5d023\") "
Oct 30 12:53:41.822721 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:41.822458    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e0c4f0e-91f5-4b63-81ac-26dc34c5d023-console-config\" (UniqueName: \"kubernetes.io/configmap/8e0c4f0e-91f5-4b63-81ac-26dc34c5d023-console-config\") pod \"8e0c4f0e-91f5-4b63-81ac-26dc34c5d023\" (UID: \"8e0c4f0e-91f5-4b63-81ac-26dc34c5d023\") "
Oct 30 12:53:41.822778 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:41.822735    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e0c4f0e-91f5-4b63-81ac-26dc34c5d023-oauth-serving-cert" pod "8e0c4f0e-91f5-4b63-81ac-26dc34c5d023" (UID: "8e0c4f0e-91f5-4b63-81ac-26dc34c5d023"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:53:41.822778 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:41.822755    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e0c4f0e-91f5-4b63-81ac-26dc34c5d023-service-ca" pod "8e0c4f0e-91f5-4b63-81ac-26dc34c5d023" (UID: "8e0c4f0e-91f5-4b63-81ac-26dc34c5d023"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:53:41.822884 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:41.822767    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e0c4f0e-91f5-4b63-81ac-26dc34c5d023-trusted-ca-bundle" pod "8e0c4f0e-91f5-4b63-81ac-26dc34c5d023" (UID: "8e0c4f0e-91f5-4b63-81ac-26dc34c5d023"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:53:41.823007 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:41.822964    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e0c4f0e-91f5-4b63-81ac-26dc34c5d023-tmp" pod "8e0c4f0e-91f5-4b63-81ac-26dc34c5d023" (UID: "8e0c4f0e-91f5-4b63-81ac-26dc34c5d023"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:53:41.823125 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:41.823010    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e0c4f0e-91f5-4b63-81ac-26dc34c5d023-console-config" pod "8e0c4f0e-91f5-4b63-81ac-26dc34c5d023" (UID: "8e0c4f0e-91f5-4b63-81ac-26dc34c5d023"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:53:41.824557 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:41.824529    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e0c4f0e-91f5-4b63-81ac-26dc34c5d023-console-serving-cert" pod "8e0c4f0e-91f5-4b63-81ac-26dc34c5d023" (UID: "8e0c4f0e-91f5-4b63-81ac-26dc34c5d023"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:53:41.824840 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:41.824816    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e0c4f0e-91f5-4b63-81ac-26dc34c5d023-console-oauth-config" pod "8e0c4f0e-91f5-4b63-81ac-26dc34c5d023" (UID: "8e0c4f0e-91f5-4b63-81ac-26dc34c5d023"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:53:41.824913 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:41.824839    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e0c4f0e-91f5-4b63-81ac-26dc34c5d023-kube-api-access-fzll4" pod "8e0c4f0e-91f5-4b63-81ac-26dc34c5d023" (UID: "8e0c4f0e-91f5-4b63-81ac-26dc34c5d023"). InnerVolumeSpecName "kube-api-access-fzll4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:53:41.923925 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:41.923901    4364 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8e0c4f0e-91f5-4b63-81ac-26dc34c5d023-service-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 12:53:41.923925 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:41.923922    4364 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e0c4f0e-91f5-4b63-81ac-26dc34c5d023-trusted-ca-bundle\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 12:53:41.924070 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:41.923931    4364 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8e0c4f0e-91f5-4b63-81ac-26dc34c5d023-console-serving-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 12:53:41.924070 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:41.923942    4364 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8e0c4f0e-91f5-4b63-81ac-26dc34c5d023-tmp\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 12:53:41.924070 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:41.923951    4364 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8e0c4f0e-91f5-4b63-81ac-26dc34c5d023-console-config\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 12:53:41.924070 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:41.923960    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzll4\" (UniqueName: \"kubernetes.io/projected/8e0c4f0e-91f5-4b63-81ac-26dc34c5d023-kube-api-access-fzll4\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 12:53:41.924070 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:41.923968    4364 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8e0c4f0e-91f5-4b63-81ac-26dc34c5d023-oauth-serving-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 12:53:41.924070 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:41.923976    4364 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8e0c4f0e-91f5-4b63-81ac-26dc34c5d023-console-oauth-config\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 12:53:42.490361 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:42.490327    4364 generic.go:356] "Generic (PLEG): container finished" podID="8e0c4f0e-91f5-4b63-81ac-26dc34c5d023" containerID="fa40428711cbb68fdd2e935a9f993f6cffb2a7527bb467fb3f49f1dd2d66c405" exitCode=0
Oct 30 12:53:42.490643 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:42.490376    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bb77db499-ntjp7" event={"ID":"8e0c4f0e-91f5-4b63-81ac-26dc34c5d023","Type":"ContainerDied","Data":"fa40428711cbb68fdd2e935a9f993f6cffb2a7527bb467fb3f49f1dd2d66c405"}
Oct 30 12:53:42.490643 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:42.490379    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bb77db499-ntjp7"
Oct 30 12:53:42.490643 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:42.490419    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bb77db499-ntjp7" event={"ID":"8e0c4f0e-91f5-4b63-81ac-26dc34c5d023","Type":"ContainerDied","Data":"5b1965a8d5f2d311bea9de2980d0e63f84eb9b36c2fabc83f99d0473dd7448e5"}
Oct 30 12:53:42.490643 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:42.490440    4364 scope.go:122] "RemoveContainer" containerID="fa40428711cbb68fdd2e935a9f993f6cffb2a7527bb467fb3f49f1dd2d66c405"
Oct 30 12:53:42.499061 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:42.498887    4364 scope.go:122] "RemoveContainer" containerID="fa40428711cbb68fdd2e935a9f993f6cffb2a7527bb467fb3f49f1dd2d66c405"
Oct 30 12:53:42.499222 ip-10-0-137-6 kubenswrapper[4364]: E1030 12:53:42.499201    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa40428711cbb68fdd2e935a9f993f6cffb2a7527bb467fb3f49f1dd2d66c405\": container with ID starting with fa40428711cbb68fdd2e935a9f993f6cffb2a7527bb467fb3f49f1dd2d66c405 not found: ID does not exist" containerID="fa40428711cbb68fdd2e935a9f993f6cffb2a7527bb467fb3f49f1dd2d66c405"
Oct 30 12:53:42.499290 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:42.499229    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa40428711cbb68fdd2e935a9f993f6cffb2a7527bb467fb3f49f1dd2d66c405"} err="failed to get container status \"fa40428711cbb68fdd2e935a9f993f6cffb2a7527bb467fb3f49f1dd2d66c405\": rpc error: code = NotFound desc = could not find container \"fa40428711cbb68fdd2e935a9f993f6cffb2a7527bb467fb3f49f1dd2d66c405\": container with ID starting with fa40428711cbb68fdd2e935a9f993f6cffb2a7527bb467fb3f49f1dd2d66c405 not found: ID does not exist"
Oct 30 12:53:42.545910 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:42.545888    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-bb77db499-ntjp7"]
Oct 30 12:53:42.559646 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:42.559623    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-bb77db499-ntjp7"]
Oct 30 12:53:42.966489 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:53:42.966460    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e0c4f0e-91f5-4b63-81ac-26dc34c5d023" path="/var/lib/kubelet/pods/8e0c4f0e-91f5-4b63-81ac-26dc34c5d023/volumes"
Oct 30 12:54:11.505323 ip-10-0-137-6 kubenswrapper[4364]: E1030 12:54:11.505276    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 12:54:41.504101 ip-10-0-137-6 kubenswrapper[4364]: E1030 12:54:41.504062    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 12:54:53.320936 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:54:53.320904    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccjfm6"]
Oct 30 12:54:53.321336 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:54:53.321139    4364 cpu_manager.go:414] "containerMap: removing container" podUID="f2fd49ba-6ca2-4231-a303-f89b01276e70" containerName="console"
Oct 30 12:54:53.321336 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:54:53.321149    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2fd49ba-6ca2-4231-a303-f89b01276e70" containerName="console"
Oct 30 12:54:53.321336 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:54:53.321159    4364 cpu_manager.go:414] "containerMap: removing container" podUID="8e0c4f0e-91f5-4b63-81ac-26dc34c5d023" containerName="console"
Oct 30 12:54:53.321336 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:54:53.321163    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e0c4f0e-91f5-4b63-81ac-26dc34c5d023" containerName="console"
Oct 30 12:54:53.321336 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:54:53.321219    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2fd49ba-6ca2-4231-a303-f89b01276e70" containerName="console"
Oct 30 12:54:53.321336 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:54:53.321227    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e0c4f0e-91f5-4b63-81ac-26dc34c5d023" containerName="console"
Oct 30 12:54:53.325087 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:54:53.325071    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccjfm6"
Oct 30 12:54:53.355628 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:54:53.355585    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 12:54:53.355738 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:54:53.355708    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-hqz7g\""
Oct 30 12:54:53.355793 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:54:53.355759    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 12:54:53.375727 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:54:53.375706    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvf77\" (UniqueName: \"kubernetes.io/projected/14f7c9cc-4223-4821-8737-b80d9d5921c7-kube-api-access-fvf77\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccjfm6\" (UID: \"14f7c9cc-4223-4821-8737-b80d9d5921c7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccjfm6"
Oct 30 12:54:53.375849 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:54:53.375739    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/14f7c9cc-4223-4821-8737-b80d9d5921c7-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccjfm6\" (UID: \"14f7c9cc-4223-4821-8737-b80d9d5921c7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccjfm6"
Oct 30 12:54:53.375849 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:54:53.375758    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/14f7c9cc-4223-4821-8737-b80d9d5921c7-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccjfm6\" (UID: \"14f7c9cc-4223-4821-8737-b80d9d5921c7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccjfm6"
Oct 30 12:54:53.427869 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:54:53.427849    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccjfm6"]
Oct 30 12:54:53.476194 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:54:53.476174    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvf77\" (UniqueName: \"kubernetes.io/projected/14f7c9cc-4223-4821-8737-b80d9d5921c7-kube-api-access-fvf77\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccjfm6\" (UID: \"14f7c9cc-4223-4821-8737-b80d9d5921c7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccjfm6"
Oct 30 12:54:53.476306 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:54:53.476207    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/14f7c9cc-4223-4821-8737-b80d9d5921c7-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccjfm6\" (UID: \"14f7c9cc-4223-4821-8737-b80d9d5921c7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccjfm6"
Oct 30 12:54:53.476306 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:54:53.476226    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/14f7c9cc-4223-4821-8737-b80d9d5921c7-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccjfm6\" (UID: \"14f7c9cc-4223-4821-8737-b80d9d5921c7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccjfm6"
Oct 30 12:54:53.476535 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:54:53.476517    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/14f7c9cc-4223-4821-8737-b80d9d5921c7-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccjfm6\" (UID: \"14f7c9cc-4223-4821-8737-b80d9d5921c7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccjfm6"
Oct 30 12:54:53.476580 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:54:53.476564    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/14f7c9cc-4223-4821-8737-b80d9d5921c7-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccjfm6\" (UID: \"14f7c9cc-4223-4821-8737-b80d9d5921c7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccjfm6"
Oct 30 12:54:53.503690 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:54:53.503665    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvf77\" (UniqueName: \"kubernetes.io/projected/14f7c9cc-4223-4821-8737-b80d9d5921c7-kube-api-access-fvf77\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccjfm6\" (UID: \"14f7c9cc-4223-4821-8737-b80d9d5921c7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccjfm6"
Oct 30 12:54:53.641308 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:54:53.641286    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccjfm6"
Oct 30 12:54:53.938180 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:54:53.938078    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccjfm6"]
Oct 30 12:54:53.940626 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:54:53.940583    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14f7c9cc_4223_4821_8737_b80d9d5921c7.slice/crio-07bac66a5583b649aa2470b22f3a69d2b3607ef543310db536b925a7de494580 WatchSource:0}: Error finding container 07bac66a5583b649aa2470b22f3a69d2b3607ef543310db536b925a7de494580: Status 404 returned error can't find the container with id 07bac66a5583b649aa2470b22f3a69d2b3607ef543310db536b925a7de494580
Oct 30 12:54:54.729017 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:54:54.728976    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccjfm6" event={"ID":"14f7c9cc-4223-4821-8737-b80d9d5921c7","Type":"ContainerStarted","Data":"07bac66a5583b649aa2470b22f3a69d2b3607ef543310db536b925a7de494580"}
Oct 30 12:54:59.748468 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:54:59.748434    4364 generic.go:356] "Generic (PLEG): container finished" podID="14f7c9cc-4223-4821-8737-b80d9d5921c7" containerID="ad14360ba75fca8dcfe9443a9f0b0301bda67e3b30dc182a8ac53c743e01e786" exitCode=0
Oct 30 12:54:59.749148 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:54:59.748512    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccjfm6" event={"ID":"14f7c9cc-4223-4821-8737-b80d9d5921c7","Type":"ContainerDied","Data":"ad14360ba75fca8dcfe9443a9f0b0301bda67e3b30dc182a8ac53c743e01e786"}
Oct 30 12:55:01.756875 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:01.756845    4364 generic.go:356] "Generic (PLEG): container finished" podID="14f7c9cc-4223-4821-8737-b80d9d5921c7" containerID="be3660f044830f713d0a9534f15ac17e94afe804f7f4c2a873d318e5c46b7603" exitCode=0
Oct 30 12:55:01.757166 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:01.756896    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccjfm6" event={"ID":"14f7c9cc-4223-4821-8737-b80d9d5921c7","Type":"ContainerDied","Data":"be3660f044830f713d0a9534f15ac17e94afe804f7f4c2a873d318e5c46b7603"}
Oct 30 12:55:07.779477 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:07.779439    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccjfm6" event={"ID":"14f7c9cc-4223-4821-8737-b80d9d5921c7","Type":"ContainerStarted","Data":"4fe92fb1d3a21d288e5e8b51d07787b6ffcabfe4d681ce2c2bba819e02b7166f"}
Oct 30 12:55:07.828407 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:07.828300    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccjfm6" podStartSLOduration=2.037028764 podStartE2EDuration="14.828286755s" totalImagesPullingTime="12.791257991s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:54:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:55:07.82756622 +0000 UTC m=+206.993930870" watchObservedRunningTime="2026-10-30 12:55:07.828286755 +0000 UTC m=+206.994651402"
Oct 30 12:55:08.789274 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:08.789235    4364 generic.go:356] "Generic (PLEG): container finished" podID="14f7c9cc-4223-4821-8737-b80d9d5921c7" containerID="4fe92fb1d3a21d288e5e8b51d07787b6ffcabfe4d681ce2c2bba819e02b7166f" exitCode=0
Oct 30 12:55:08.789718 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:08.789302    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccjfm6" event={"ID":"14f7c9cc-4223-4821-8737-b80d9d5921c7","Type":"ContainerDied","Data":"4fe92fb1d3a21d288e5e8b51d07787b6ffcabfe4d681ce2c2bba819e02b7166f"}
Oct 30 12:55:09.910631 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:09.910608    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccjfm6"
Oct 30 12:55:10.000952 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:10.000922    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14f7c9cc-4223-4821-8737-b80d9d5921c7-bundle\" (UniqueName: \"kubernetes.io/empty-dir/14f7c9cc-4223-4821-8737-b80d9d5921c7-bundle\") pod \"14f7c9cc-4223-4821-8737-b80d9d5921c7\" (UID: \"14f7c9cc-4223-4821-8737-b80d9d5921c7\") "
Oct 30 12:55:10.001106 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:10.000977    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14f7c9cc-4223-4821-8737-b80d9d5921c7-util\" (UniqueName: \"kubernetes.io/empty-dir/14f7c9cc-4223-4821-8737-b80d9d5921c7-util\") pod \"14f7c9cc-4223-4821-8737-b80d9d5921c7\" (UID: \"14f7c9cc-4223-4821-8737-b80d9d5921c7\") "
Oct 30 12:55:10.001106 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:10.001029    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14f7c9cc-4223-4821-8737-b80d9d5921c7-kube-api-access-fvf77\" (UniqueName: \"kubernetes.io/projected/14f7c9cc-4223-4821-8737-b80d9d5921c7-kube-api-access-fvf77\") pod \"14f7c9cc-4223-4821-8737-b80d9d5921c7\" (UID: \"14f7c9cc-4223-4821-8737-b80d9d5921c7\") "
Oct 30 12:55:10.001736 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:10.001701    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14f7c9cc-4223-4821-8737-b80d9d5921c7-bundle" pod "14f7c9cc-4223-4821-8737-b80d9d5921c7" (UID: "14f7c9cc-4223-4821-8737-b80d9d5921c7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:55:10.003296 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:10.003274    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14f7c9cc-4223-4821-8737-b80d9d5921c7-kube-api-access-fvf77" pod "14f7c9cc-4223-4821-8737-b80d9d5921c7" (UID: "14f7c9cc-4223-4821-8737-b80d9d5921c7"). InnerVolumeSpecName "kube-api-access-fvf77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:55:10.005367 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:10.005346    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14f7c9cc-4223-4821-8737-b80d9d5921c7-util" pod "14f7c9cc-4223-4821-8737-b80d9d5921c7" (UID: "14f7c9cc-4223-4821-8737-b80d9d5921c7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:55:10.101999 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:10.101915    4364 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/14f7c9cc-4223-4821-8737-b80d9d5921c7-util\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 12:55:10.101999 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:10.101948    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvf77\" (UniqueName: \"kubernetes.io/projected/14f7c9cc-4223-4821-8737-b80d9d5921c7-kube-api-access-fvf77\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 12:55:10.101999 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:10.101961    4364 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/14f7c9cc-4223-4821-8737-b80d9d5921c7-bundle\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 12:55:10.797729 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:10.797692    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccjfm6" event={"ID":"14f7c9cc-4223-4821-8737-b80d9d5921c7","Type":"ContainerDied","Data":"07bac66a5583b649aa2470b22f3a69d2b3607ef543310db536b925a7de494580"}
Oct 30 12:55:10.797894 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:10.797747    4364 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07bac66a5583b649aa2470b22f3a69d2b3607ef543310db536b925a7de494580"
Oct 30 12:55:10.797894 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:10.797710    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccjfm6"
Oct 30 12:55:11.502697 ip-10-0-137-6 kubenswrapper[4364]: E1030 12:55:11.502664    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 12:55:16.324296 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:16.324266    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-v5st9"]
Oct 30 12:55:16.324690 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:16.324516    4364 cpu_manager.go:414] "containerMap: removing container" podUID="14f7c9cc-4223-4821-8737-b80d9d5921c7" containerName="extract"
Oct 30 12:55:16.324690 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:16.324528    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14f7c9cc-4223-4821-8737-b80d9d5921c7" containerName="extract"
Oct 30 12:55:16.324690 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:16.324547    4364 cpu_manager.go:414] "containerMap: removing container" podUID="14f7c9cc-4223-4821-8737-b80d9d5921c7" containerName="util"
Oct 30 12:55:16.324690 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:16.324552    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14f7c9cc-4223-4821-8737-b80d9d5921c7" containerName="util"
Oct 30 12:55:16.324690 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:16.324562    4364 cpu_manager.go:414] "containerMap: removing container" podUID="14f7c9cc-4223-4821-8737-b80d9d5921c7" containerName="pull"
Oct 30 12:55:16.324690 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:16.324570    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14f7c9cc-4223-4821-8737-b80d9d5921c7" containerName="pull"
Oct 30 12:55:16.324690 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:16.324636    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="14f7c9cc-4223-4821-8737-b80d9d5921c7" containerName="extract"
Oct 30 12:55:16.365489 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:16.365459    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-v5st9"]
Oct 30 12:55:16.365652 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:16.365562    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-v5st9"
Oct 30 12:55:16.382446 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:16.382424    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Oct 30 12:55:16.382640 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:16.382624    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Oct 30 12:55:16.395631 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:16.395614    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-btgjv\""
Oct 30 12:55:16.446671 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:16.446642    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aa4e8e4c-0f09-4837-bf6b-6f9854592436-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-v5st9\" (UID: \"aa4e8e4c-0f09-4837-bf6b-6f9854592436\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-v5st9"
Oct 30 12:55:16.446775 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:16.446681    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd4th\" (UniqueName: \"kubernetes.io/projected/aa4e8e4c-0f09-4837-bf6b-6f9854592436-kube-api-access-nd4th\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-v5st9\" (UID: \"aa4e8e4c-0f09-4837-bf6b-6f9854592436\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-v5st9"
Oct 30 12:55:16.547884 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:16.547857    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd4th\" (UniqueName: \"kubernetes.io/projected/aa4e8e4c-0f09-4837-bf6b-6f9854592436-kube-api-access-nd4th\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-v5st9\" (UID: \"aa4e8e4c-0f09-4837-bf6b-6f9854592436\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-v5st9"
Oct 30 12:55:16.547973 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:16.547915    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aa4e8e4c-0f09-4837-bf6b-6f9854592436-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-v5st9\" (UID: \"aa4e8e4c-0f09-4837-bf6b-6f9854592436\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-v5st9"
Oct 30 12:55:16.548223 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:16.548208    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aa4e8e4c-0f09-4837-bf6b-6f9854592436-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-v5st9\" (UID: \"aa4e8e4c-0f09-4837-bf6b-6f9854592436\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-v5st9"
Oct 30 12:55:16.574627 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:16.574553    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd4th\" (UniqueName: \"kubernetes.io/projected/aa4e8e4c-0f09-4837-bf6b-6f9854592436-kube-api-access-nd4th\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-v5st9\" (UID: \"aa4e8e4c-0f09-4837-bf6b-6f9854592436\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-v5st9"
Oct 30 12:55:16.687641 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:16.687619    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-v5st9"
Oct 30 12:55:17.033612 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:17.033580    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-v5st9"]
Oct 30 12:55:17.035196 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:55:17.035172    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa4e8e4c_0f09_4837_bf6b_6f9854592436.slice/crio-76973d4bc8a29f297cdd1a5563e8f422b4c1c56ecf5a27d32e2b9637e5220941 WatchSource:0}: Error finding container 76973d4bc8a29f297cdd1a5563e8f422b4c1c56ecf5a27d32e2b9637e5220941: Status 404 returned error can't find the container with id 76973d4bc8a29f297cdd1a5563e8f422b4c1c56ecf5a27d32e2b9637e5220941
Oct 30 12:55:17.821755 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:17.821720    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-v5st9" event={"ID":"aa4e8e4c-0f09-4837-bf6b-6f9854592436","Type":"ContainerStarted","Data":"76973d4bc8a29f297cdd1a5563e8f422b4c1c56ecf5a27d32e2b9637e5220941"}
Oct 30 12:55:19.830821 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:19.830788    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-v5st9" event={"ID":"aa4e8e4c-0f09-4837-bf6b-6f9854592436","Type":"ContainerStarted","Data":"87a0d4fab659d4382a7b6a2d65408ab79ac3c7fa775219ca08e9284a4cacd70b"}
Oct 30 12:55:19.908281 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:19.908229    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-v5st9" podStartSLOduration=1.856787206 podStartE2EDuration="3.908213488s" totalImagesPullingTime="2.051426282s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:55:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:55:19.893781683 +0000 UTC m=+219.060146332" watchObservedRunningTime="2026-10-30 12:55:19.908213488 +0000 UTC m=+219.074578137"
Oct 30 12:55:20.972490 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:20.972459    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85whpd5"]
Oct 30 12:55:20.974575 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:20.974559    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85whpd5"
Oct 30 12:55:20.994258 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:20.994239    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 12:55:20.998552 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:20.998532    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 12:55:21.009305 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:21.009276    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85whpd5"]
Oct 30 12:55:21.012659 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:21.012640    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"61b1ff68c7255ee5c5329fe19a5d96293a9578-ce91a85d-dockercfg-2xhq8\""
Oct 30 12:55:21.080375 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:21.080347    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/edcc6376-b33d-4105-8872-d64e5151582b-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85whpd5\" (UID: \"edcc6376-b33d-4105-8872-d64e5151582b\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85whpd5"
Oct 30 12:55:21.080477 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:21.080409    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/edcc6376-b33d-4105-8872-d64e5151582b-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85whpd5\" (UID: \"edcc6376-b33d-4105-8872-d64e5151582b\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85whpd5"
Oct 30 12:55:21.080526 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:21.080505    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmvs5\" (UniqueName: \"kubernetes.io/projected/edcc6376-b33d-4105-8872-d64e5151582b-kube-api-access-hmvs5\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85whpd5\" (UID: \"edcc6376-b33d-4105-8872-d64e5151582b\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85whpd5"
Oct 30 12:55:21.157792 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:21.157770    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-t4p44"]
Oct 30 12:55:21.159554 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:21.159540    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-t4p44"
Oct 30 12:55:21.176936 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:21.176915    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 30 12:55:21.176936 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:21.176924    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 30 12:55:21.180965 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:21.180945    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/edcc6376-b33d-4105-8872-d64e5151582b-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85whpd5\" (UID: \"edcc6376-b33d-4105-8872-d64e5151582b\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85whpd5"
Oct 30 12:55:21.181065 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:21.180971    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmvs5\" (UniqueName: \"kubernetes.io/projected/edcc6376-b33d-4105-8872-d64e5151582b-kube-api-access-hmvs5\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85whpd5\" (UID: \"edcc6376-b33d-4105-8872-d64e5151582b\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85whpd5"
Oct 30 12:55:21.181065 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:21.180994    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/edcc6376-b33d-4105-8872-d64e5151582b-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85whpd5\" (UID: \"edcc6376-b33d-4105-8872-d64e5151582b\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85whpd5"
Oct 30 12:55:21.181318 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:21.181299    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/edcc6376-b33d-4105-8872-d64e5151582b-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85whpd5\" (UID: \"edcc6376-b33d-4105-8872-d64e5151582b\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85whpd5"
Oct 30 12:55:21.181392 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:21.181324    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-klkr6\""
Oct 30 12:55:21.181435 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:21.181394    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/edcc6376-b33d-4105-8872-d64e5151582b-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85whpd5\" (UID: \"edcc6376-b33d-4105-8872-d64e5151582b\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85whpd5"
Oct 30 12:55:21.184345 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:21.184324    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-t4p44"]
Oct 30 12:55:21.214998 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:21.214963    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmvs5\" (UniqueName: \"kubernetes.io/projected/edcc6376-b33d-4105-8872-d64e5151582b-kube-api-access-hmvs5\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85whpd5\" (UID: \"edcc6376-b33d-4105-8872-d64e5151582b\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85whpd5"
Oct 30 12:55:21.281905 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:21.281859    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/723ed38b-9866-4716-b18f-d6c8e7e68a39-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-t4p44\" (UID: \"723ed38b-9866-4716-b18f-d6c8e7e68a39\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-t4p44"
Oct 30 12:55:21.281905 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:21.281897    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xvs5\" (UniqueName: \"kubernetes.io/projected/723ed38b-9866-4716-b18f-d6c8e7e68a39-kube-api-access-8xvs5\") pod \"cert-manager-webhook-5dbbbf75cc-t4p44\" (UID: \"723ed38b-9866-4716-b18f-d6c8e7e68a39\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-t4p44"
Oct 30 12:55:21.285846 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:21.285830    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85whpd5"
Oct 30 12:55:21.383191 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:21.383163    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/723ed38b-9866-4716-b18f-d6c8e7e68a39-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-t4p44\" (UID: \"723ed38b-9866-4716-b18f-d6c8e7e68a39\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-t4p44"
Oct 30 12:55:21.383307 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:21.383208    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xvs5\" (UniqueName: \"kubernetes.io/projected/723ed38b-9866-4716-b18f-d6c8e7e68a39-kube-api-access-8xvs5\") pod \"cert-manager-webhook-5dbbbf75cc-t4p44\" (UID: \"723ed38b-9866-4716-b18f-d6c8e7e68a39\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-t4p44"
Oct 30 12:55:21.408022 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:21.407998    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/723ed38b-9866-4716-b18f-d6c8e7e68a39-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-t4p44\" (UID: \"723ed38b-9866-4716-b18f-d6c8e7e68a39\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-t4p44"
Oct 30 12:55:21.408258 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:21.408240    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xvs5\" (UniqueName: \"kubernetes.io/projected/723ed38b-9866-4716-b18f-d6c8e7e68a39-kube-api-access-8xvs5\") pod \"cert-manager-webhook-5dbbbf75cc-t4p44\" (UID: \"723ed38b-9866-4716-b18f-d6c8e7e68a39\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-t4p44"
Oct 30 12:55:21.482465 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:21.482441    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-t4p44"
Oct 30 12:55:21.601897 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:21.601834    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85whpd5"]
Oct 30 12:55:21.604322 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:55:21.604300    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedcc6376_b33d_4105_8872_d64e5151582b.slice/crio-f73d174cddda7c0801f26e3c48d834afea95d86c657ef3035d0f05d3cabc89aa WatchSource:0}: Error finding container f73d174cddda7c0801f26e3c48d834afea95d86c657ef3035d0f05d3cabc89aa: Status 404 returned error can't find the container with id f73d174cddda7c0801f26e3c48d834afea95d86c657ef3035d0f05d3cabc89aa
Oct 30 12:55:21.778236 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:21.778208    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-t4p44"]
Oct 30 12:55:21.780516 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:55:21.780492    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod723ed38b_9866_4716_b18f_d6c8e7e68a39.slice/crio-72c96af8d4147b9a9cee633a5a6a176dc106893ef0c9147a9ece5a2576094e77 WatchSource:0}: Error finding container 72c96af8d4147b9a9cee633a5a6a176dc106893ef0c9147a9ece5a2576094e77: Status 404 returned error can't find the container with id 72c96af8d4147b9a9cee633a5a6a176dc106893ef0c9147a9ece5a2576094e77
Oct 30 12:55:21.839199 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:21.839171    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-t4p44" event={"ID":"723ed38b-9866-4716-b18f-d6c8e7e68a39","Type":"ContainerStarted","Data":"72c96af8d4147b9a9cee633a5a6a176dc106893ef0c9147a9ece5a2576094e77"}
Oct 30 12:55:21.840703 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:21.840682    4364 generic.go:356] "Generic (PLEG): container finished" podID="edcc6376-b33d-4105-8872-d64e5151582b" containerID="a33ceb140be5298200ff5cecc871fb5fdfb9bf5471ef7037e18f2ba884bce40e" exitCode=0
Oct 30 12:55:21.840794 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:21.840732    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85whpd5" event={"ID":"edcc6376-b33d-4105-8872-d64e5151582b","Type":"ContainerDied","Data":"a33ceb140be5298200ff5cecc871fb5fdfb9bf5471ef7037e18f2ba884bce40e"}
Oct 30 12:55:21.840794 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:21.840785    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85whpd5" event={"ID":"edcc6376-b33d-4105-8872-d64e5151582b","Type":"ContainerStarted","Data":"f73d174cddda7c0801f26e3c48d834afea95d86c657ef3035d0f05d3cabc89aa"}
Oct 30 12:55:23.850861 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:23.850786    4364 generic.go:356] "Generic (PLEG): container finished" podID="edcc6376-b33d-4105-8872-d64e5151582b" containerID="9626db618984393b12674703634744e2ce59eef04bc3d43bddc4458c03c83f6e" exitCode=0
Oct 30 12:55:23.851255 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:23.850877    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85whpd5" event={"ID":"edcc6376-b33d-4105-8872-d64e5151582b","Type":"ContainerDied","Data":"9626db618984393b12674703634744e2ce59eef04bc3d43bddc4458c03c83f6e"}
Oct 30 12:55:24.856621 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:24.856570    4364 generic.go:356] "Generic (PLEG): container finished" podID="edcc6376-b33d-4105-8872-d64e5151582b" containerID="5c9b451bb60e1574f3bc898f019b8e49cdf1ca01e2e619d220007a383e816dc1" exitCode=0
Oct 30 12:55:24.856985 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:24.856650    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85whpd5" event={"ID":"edcc6376-b33d-4105-8872-d64e5151582b","Type":"ContainerDied","Data":"5c9b451bb60e1574f3bc898f019b8e49cdf1ca01e2e619d220007a383e816dc1"}
Oct 30 12:55:25.861807 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:25.861766    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-t4p44" event={"ID":"723ed38b-9866-4716-b18f-d6c8e7e68a39","Type":"ContainerStarted","Data":"76bb56cdb3a7b95c396eb3fb4c08f6f2a865163b5abe35b79c9960e7edc6fcd2"}
Oct 30 12:55:25.862264 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:25.862011    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-t4p44"
Oct 30 12:55:25.911495 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:25.911446    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-t4p44" podStartSLOduration=1.788210151 podStartE2EDuration="4.911429546s" totalImagesPullingTime="3.123219395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:55:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:55:25.909703605 +0000 UTC m=+225.076068253" watchObservedRunningTime="2026-10-30 12:55:25.911429546 +0000 UTC m=+225.077794194"
Oct 30 12:55:25.981195 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:25.981174    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85whpd5"
Oct 30 12:55:26.121047 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:26.121020    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edcc6376-b33d-4105-8872-d64e5151582b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/edcc6376-b33d-4105-8872-d64e5151582b-bundle\") pod \"edcc6376-b33d-4105-8872-d64e5151582b\" (UID: \"edcc6376-b33d-4105-8872-d64e5151582b\") "
Oct 30 12:55:26.121190 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:26.121061    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edcc6376-b33d-4105-8872-d64e5151582b-kube-api-access-hmvs5\" (UniqueName: \"kubernetes.io/projected/edcc6376-b33d-4105-8872-d64e5151582b-kube-api-access-hmvs5\") pod \"edcc6376-b33d-4105-8872-d64e5151582b\" (UID: \"edcc6376-b33d-4105-8872-d64e5151582b\") "
Oct 30 12:55:26.121190 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:26.121150    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edcc6376-b33d-4105-8872-d64e5151582b-util\" (UniqueName: \"kubernetes.io/empty-dir/edcc6376-b33d-4105-8872-d64e5151582b-util\") pod \"edcc6376-b33d-4105-8872-d64e5151582b\" (UID: \"edcc6376-b33d-4105-8872-d64e5151582b\") "
Oct 30 12:55:26.121526 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:26.121502    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edcc6376-b33d-4105-8872-d64e5151582b-bundle" pod "edcc6376-b33d-4105-8872-d64e5151582b" (UID: "edcc6376-b33d-4105-8872-d64e5151582b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:55:26.123145 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:26.123120    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edcc6376-b33d-4105-8872-d64e5151582b-kube-api-access-hmvs5" pod "edcc6376-b33d-4105-8872-d64e5151582b" (UID: "edcc6376-b33d-4105-8872-d64e5151582b"). InnerVolumeSpecName "kube-api-access-hmvs5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:55:26.126246 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:26.126211    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edcc6376-b33d-4105-8872-d64e5151582b-util" pod "edcc6376-b33d-4105-8872-d64e5151582b" (UID: "edcc6376-b33d-4105-8872-d64e5151582b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:55:26.221763 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:26.221727    4364 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/edcc6376-b33d-4105-8872-d64e5151582b-util\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 12:55:26.221763 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:26.221759    4364 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/edcc6376-b33d-4105-8872-d64e5151582b-bundle\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 12:55:26.221916 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:26.221774    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmvs5\" (UniqueName: \"kubernetes.io/projected/edcc6376-b33d-4105-8872-d64e5151582b-kube-api-access-hmvs5\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 12:55:26.867053 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:26.867016    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85whpd5"
Oct 30 12:55:26.867438 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:26.867011    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85whpd5" event={"ID":"edcc6376-b33d-4105-8872-d64e5151582b","Type":"ContainerDied","Data":"f73d174cddda7c0801f26e3c48d834afea95d86c657ef3035d0f05d3cabc89aa"}
Oct 30 12:55:26.867438 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:26.867187    4364 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f73d174cddda7c0801f26e3c48d834afea95d86c657ef3035d0f05d3cabc89aa"
Oct 30 12:55:27.942645 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:27.942590    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-jgwj4"]
Oct 30 12:55:27.943058 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:27.942887    4364 cpu_manager.go:414] "containerMap: removing container" podUID="edcc6376-b33d-4105-8872-d64e5151582b" containerName="util"
Oct 30 12:55:27.943058 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:27.942898    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edcc6376-b33d-4105-8872-d64e5151582b" containerName="util"
Oct 30 12:55:27.943058 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:27.942916    4364 cpu_manager.go:414] "containerMap: removing container" podUID="edcc6376-b33d-4105-8872-d64e5151582b" containerName="extract"
Oct 30 12:55:27.943058 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:27.942923    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edcc6376-b33d-4105-8872-d64e5151582b" containerName="extract"
Oct 30 12:55:27.943058 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:27.942932    4364 cpu_manager.go:414] "containerMap: removing container" podUID="edcc6376-b33d-4105-8872-d64e5151582b" containerName="pull"
Oct 30 12:55:27.943058 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:27.942937    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edcc6376-b33d-4105-8872-d64e5151582b" containerName="pull"
Oct 30 12:55:27.943058 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:27.943001    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="edcc6376-b33d-4105-8872-d64e5151582b" containerName="extract"
Oct 30 12:55:27.945943 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:27.945923    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jgwj4"
Oct 30 12:55:27.967400 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:27.967372    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-jgwj4"]
Oct 30 12:55:27.979530 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:27.979508    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-khj44\""
Oct 30 12:55:28.035971 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:28.035945    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a844dbdb-e175-4739-bc27-d15241277273-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-jgwj4\" (UID: \"a844dbdb-e175-4739-bc27-d15241277273\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jgwj4"
Oct 30 12:55:28.036070 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:28.035974    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r8dw\" (UniqueName: \"kubernetes.io/projected/a844dbdb-e175-4739-bc27-d15241277273-kube-api-access-8r8dw\") pod \"cert-manager-cainjector-5c57bbd4fc-jgwj4\" (UID: \"a844dbdb-e175-4739-bc27-d15241277273\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jgwj4"
Oct 30 12:55:28.136242 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:28.136211    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a844dbdb-e175-4739-bc27-d15241277273-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-jgwj4\" (UID: \"a844dbdb-e175-4739-bc27-d15241277273\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jgwj4"
Oct 30 12:55:28.136242 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:28.136244    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r8dw\" (UniqueName: \"kubernetes.io/projected/a844dbdb-e175-4739-bc27-d15241277273-kube-api-access-8r8dw\") pod \"cert-manager-cainjector-5c57bbd4fc-jgwj4\" (UID: \"a844dbdb-e175-4739-bc27-d15241277273\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jgwj4"
Oct 30 12:55:28.165021 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:28.164995    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a844dbdb-e175-4739-bc27-d15241277273-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-jgwj4\" (UID: \"a844dbdb-e175-4739-bc27-d15241277273\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jgwj4"
Oct 30 12:55:28.165103 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:28.165054    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r8dw\" (UniqueName: \"kubernetes.io/projected/a844dbdb-e175-4739-bc27-d15241277273-kube-api-access-8r8dw\") pod \"cert-manager-cainjector-5c57bbd4fc-jgwj4\" (UID: \"a844dbdb-e175-4739-bc27-d15241277273\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jgwj4"
Oct 30 12:55:28.257581 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:28.257534    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jgwj4"
Oct 30 12:55:28.555498 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:28.555473    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-jgwj4"]
Oct 30 12:55:28.557643 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:55:28.557612    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda844dbdb_e175_4739_bc27_d15241277273.slice/crio-a7fdb234c0e493cfd3f08bff6d3dc23aeeff85c646195999551f802f15796f7e WatchSource:0}: Error finding container a7fdb234c0e493cfd3f08bff6d3dc23aeeff85c646195999551f802f15796f7e: Status 404 returned error can't find the container with id a7fdb234c0e493cfd3f08bff6d3dc23aeeff85c646195999551f802f15796f7e
Oct 30 12:55:28.875954 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:28.875922    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jgwj4" event={"ID":"a844dbdb-e175-4739-bc27-d15241277273","Type":"ContainerStarted","Data":"a2214428d65ef5fc648e653cd5c2c214581e683f1629d1b59f200997f69f24e7"}
Oct 30 12:55:28.876082 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:28.875960    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jgwj4" event={"ID":"a844dbdb-e175-4739-bc27-d15241277273","Type":"ContainerStarted","Data":"a7fdb234c0e493cfd3f08bff6d3dc23aeeff85c646195999551f802f15796f7e"}
Oct 30 12:55:28.949946 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:28.949901    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jgwj4" podStartSLOduration=1.949887408 podStartE2EDuration="1.949887408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:55:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:55:28.94949673 +0000 UTC m=+228.115861380" watchObservedRunningTime="2026-10-30 12:55:28.949887408 +0000 UTC m=+228.116252056"
Oct 30 12:55:31.870121 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:31.870091    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-t4p44"
Oct 30 12:55:41.470923 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:41.470889    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wjm85"]
Oct 30 12:55:41.475362 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:41.475346    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wjm85"
Oct 30 12:55:41.493921 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:41.493897    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wjm85"]
Oct 30 12:55:41.504696 ip-10-0-137-6 kubenswrapper[4364]: E1030 12:55:41.504662    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 12:55:41.506389 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:41.506370    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 12:55:41.506489 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:41.506407    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 12:55:41.506528 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:41.506490    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-vt7j7\""
Oct 30 12:55:41.626425 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:41.626396    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx4kb\" (UniqueName: \"kubernetes.io/projected/652a4af3-3f85-4744-82ed-f2664ea28b25-kube-api-access-nx4kb\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wjm85\" (UID: \"652a4af3-3f85-4744-82ed-f2664ea28b25\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wjm85"
Oct 30 12:55:41.626552 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:41.626450    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/652a4af3-3f85-4744-82ed-f2664ea28b25-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wjm85\" (UID: \"652a4af3-3f85-4744-82ed-f2664ea28b25\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wjm85"
Oct 30 12:55:41.626552 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:41.626536    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/652a4af3-3f85-4744-82ed-f2664ea28b25-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wjm85\" (UID: \"652a4af3-3f85-4744-82ed-f2664ea28b25\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wjm85"
Oct 30 12:55:41.727404 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:41.727348    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/652a4af3-3f85-4744-82ed-f2664ea28b25-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wjm85\" (UID: \"652a4af3-3f85-4744-82ed-f2664ea28b25\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wjm85"
Oct 30 12:55:41.727404 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:41.727387    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/652a4af3-3f85-4744-82ed-f2664ea28b25-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wjm85\" (UID: \"652a4af3-3f85-4744-82ed-f2664ea28b25\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wjm85"
Oct 30 12:55:41.727545 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:41.727413    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx4kb\" (UniqueName: \"kubernetes.io/projected/652a4af3-3f85-4744-82ed-f2664ea28b25-kube-api-access-nx4kb\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wjm85\" (UID: \"652a4af3-3f85-4744-82ed-f2664ea28b25\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wjm85"
Oct 30 12:55:41.727771 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:41.727754    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/652a4af3-3f85-4744-82ed-f2664ea28b25-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wjm85\" (UID: \"652a4af3-3f85-4744-82ed-f2664ea28b25\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wjm85"
Oct 30 12:55:41.727814 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:41.727781    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/652a4af3-3f85-4744-82ed-f2664ea28b25-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wjm85\" (UID: \"652a4af3-3f85-4744-82ed-f2664ea28b25\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wjm85"
Oct 30 12:55:41.752675 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:41.752652    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx4kb\" (UniqueName: \"kubernetes.io/projected/652a4af3-3f85-4744-82ed-f2664ea28b25-kube-api-access-nx4kb\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wjm85\" (UID: \"652a4af3-3f85-4744-82ed-f2664ea28b25\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wjm85"
Oct 30 12:55:41.796339 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:41.796320    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wjm85"
Oct 30 12:55:42.092645 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:42.092552    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wjm85"]
Oct 30 12:55:42.095773 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:55:42.095735    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod652a4af3_3f85_4744_82ed_f2664ea28b25.slice/crio-75afb0526dde5e765d6d3636b73bfc6078d8e974e2c2367b7da8e3d35c26f980 WatchSource:0}: Error finding container 75afb0526dde5e765d6d3636b73bfc6078d8e974e2c2367b7da8e3d35c26f980: Status 404 returned error can't find the container with id 75afb0526dde5e765d6d3636b73bfc6078d8e974e2c2367b7da8e3d35c26f980
Oct 30 12:55:42.929649 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:42.929616    4364 generic.go:356] "Generic (PLEG): container finished" podID="652a4af3-3f85-4744-82ed-f2664ea28b25" containerID="cb5c5d788780c81930e36fa0e517f3a3af2de053c82e0c496f351d301ef53993" exitCode=0
Oct 30 12:55:42.930023 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:42.929690    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wjm85" event={"ID":"652a4af3-3f85-4744-82ed-f2664ea28b25","Type":"ContainerDied","Data":"cb5c5d788780c81930e36fa0e517f3a3af2de053c82e0c496f351d301ef53993"}
Oct 30 12:55:42.930023 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:42.929743    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wjm85" event={"ID":"652a4af3-3f85-4744-82ed-f2664ea28b25","Type":"ContainerStarted","Data":"75afb0526dde5e765d6d3636b73bfc6078d8e974e2c2367b7da8e3d35c26f980"}
Oct 30 12:55:43.935131 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:43.935101    4364 generic.go:356] "Generic (PLEG): container finished" podID="652a4af3-3f85-4744-82ed-f2664ea28b25" containerID="fbd8deb01355576feb364298ef974b1a2cc20354f472bdf8a8e9261893bc8e2b" exitCode=0
Oct 30 12:55:43.935568 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:43.935192    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wjm85" event={"ID":"652a4af3-3f85-4744-82ed-f2664ea28b25","Type":"ContainerDied","Data":"fbd8deb01355576feb364298ef974b1a2cc20354f472bdf8a8e9261893bc8e2b"}
Oct 30 12:55:44.941092 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:44.941057    4364 generic.go:356] "Generic (PLEG): container finished" podID="652a4af3-3f85-4744-82ed-f2664ea28b25" containerID="0ca80fb4433d37de80f337553831e1dceb5efe6e92d1deefd596b7cd22b33769" exitCode=0
Oct 30 12:55:44.941451 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:44.941124    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wjm85" event={"ID":"652a4af3-3f85-4744-82ed-f2664ea28b25","Type":"ContainerDied","Data":"0ca80fb4433d37de80f337553831e1dceb5efe6e92d1deefd596b7cd22b33769"}
Oct 30 12:55:46.063023 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:46.062999    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wjm85"
Oct 30 12:55:46.156515 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:46.156488    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/652a4af3-3f85-4744-82ed-f2664ea28b25-kube-api-access-nx4kb\" (UniqueName: \"kubernetes.io/projected/652a4af3-3f85-4744-82ed-f2664ea28b25-kube-api-access-nx4kb\") pod \"652a4af3-3f85-4744-82ed-f2664ea28b25\" (UID: \"652a4af3-3f85-4744-82ed-f2664ea28b25\") "
Oct 30 12:55:46.156672 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:46.156539    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/652a4af3-3f85-4744-82ed-f2664ea28b25-bundle\" (UniqueName: \"kubernetes.io/empty-dir/652a4af3-3f85-4744-82ed-f2664ea28b25-bundle\") pod \"652a4af3-3f85-4744-82ed-f2664ea28b25\" (UID: \"652a4af3-3f85-4744-82ed-f2664ea28b25\") "
Oct 30 12:55:46.156672 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:46.156568    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/652a4af3-3f85-4744-82ed-f2664ea28b25-util\" (UniqueName: \"kubernetes.io/empty-dir/652a4af3-3f85-4744-82ed-f2664ea28b25-util\") pod \"652a4af3-3f85-4744-82ed-f2664ea28b25\" (UID: \"652a4af3-3f85-4744-82ed-f2664ea28b25\") "
Oct 30 12:55:46.157188 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:46.157158    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/652a4af3-3f85-4744-82ed-f2664ea28b25-bundle" pod "652a4af3-3f85-4744-82ed-f2664ea28b25" (UID: "652a4af3-3f85-4744-82ed-f2664ea28b25"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:55:46.158433 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:46.158410    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/652a4af3-3f85-4744-82ed-f2664ea28b25-kube-api-access-nx4kb" pod "652a4af3-3f85-4744-82ed-f2664ea28b25" (UID: "652a4af3-3f85-4744-82ed-f2664ea28b25"). InnerVolumeSpecName "kube-api-access-nx4kb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:55:46.162161 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:46.162140    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/652a4af3-3f85-4744-82ed-f2664ea28b25-util" pod "652a4af3-3f85-4744-82ed-f2664ea28b25" (UID: "652a4af3-3f85-4744-82ed-f2664ea28b25"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:55:46.257888 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:46.257842    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nx4kb\" (UniqueName: \"kubernetes.io/projected/652a4af3-3f85-4744-82ed-f2664ea28b25-kube-api-access-nx4kb\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 12:55:46.257888 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:46.257863    4364 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/652a4af3-3f85-4744-82ed-f2664ea28b25-bundle\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 12:55:46.257888 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:46.257872    4364 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/652a4af3-3f85-4744-82ed-f2664ea28b25-util\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 12:55:46.951168 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:46.951138    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wjm85" event={"ID":"652a4af3-3f85-4744-82ed-f2664ea28b25","Type":"ContainerDied","Data":"75afb0526dde5e765d6d3636b73bfc6078d8e974e2c2367b7da8e3d35c26f980"}
Oct 30 12:55:46.951335 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:46.951183    4364 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75afb0526dde5e765d6d3636b73bfc6078d8e974e2c2367b7da8e3d35c26f980"
Oct 30 12:55:46.951335 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:46.951186    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wjm85"
Oct 30 12:55:52.567769 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:52.567728    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-678d69fff5-rlqml"]
Oct 30 12:55:52.568369 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:52.568112    4364 cpu_manager.go:414] "containerMap: removing container" podUID="652a4af3-3f85-4744-82ed-f2664ea28b25" containerName="extract"
Oct 30 12:55:52.568369 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:52.568129    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="652a4af3-3f85-4744-82ed-f2664ea28b25" containerName="extract"
Oct 30 12:55:52.568369 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:52.568151    4364 cpu_manager.go:414] "containerMap: removing container" podUID="652a4af3-3f85-4744-82ed-f2664ea28b25" containerName="util"
Oct 30 12:55:52.568369 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:52.568158    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="652a4af3-3f85-4744-82ed-f2664ea28b25" containerName="util"
Oct 30 12:55:52.568369 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:52.568170    4364 cpu_manager.go:414] "containerMap: removing container" podUID="652a4af3-3f85-4744-82ed-f2664ea28b25" containerName="pull"
Oct 30 12:55:52.568369 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:52.568177    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="652a4af3-3f85-4744-82ed-f2664ea28b25" containerName="pull"
Oct 30 12:55:52.568369 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:52.568258    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="652a4af3-3f85-4744-82ed-f2664ea28b25" containerName="extract"
Oct 30 12:55:52.574056 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:52.574034    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-rlqml"
Oct 30 12:55:52.628877 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:52.628851    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-678d69fff5-rlqml"]
Oct 30 12:55:52.665829 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:52.665801    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 12:55:52.665829 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:52.665819    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-lvqkv\""
Oct 30 12:55:52.666152 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:52.666139    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 12:55:52.666312 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:52.666299    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\""
Oct 30 12:55:52.666585 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:52.666571    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 12:55:52.667001 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:52.666988    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 12:55:52.706930 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:52.706907    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3afcec39-f8ad-46ba-b002-cf1465de9072-metrics-cert\") pod \"opendatahub-operator-controller-manager-678d69fff5-rlqml\" (UID: \"3afcec39-f8ad-46ba-b002-cf1465de9072\") " pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-rlqml"
Oct 30 12:55:52.707061 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:52.706949    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3afcec39-f8ad-46ba-b002-cf1465de9072-manifests\") pod \"opendatahub-operator-controller-manager-678d69fff5-rlqml\" (UID: \"3afcec39-f8ad-46ba-b002-cf1465de9072\") " pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-rlqml"
Oct 30 12:55:52.707061 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:52.706984    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3afcec39-f8ad-46ba-b002-cf1465de9072-apiservice-cert\") pod \"opendatahub-operator-controller-manager-678d69fff5-rlqml\" (UID: \"3afcec39-f8ad-46ba-b002-cf1465de9072\") " pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-rlqml"
Oct 30 12:55:52.707132 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:52.707074    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3afcec39-f8ad-46ba-b002-cf1465de9072-webhook-cert\") pod \"opendatahub-operator-controller-manager-678d69fff5-rlqml\" (UID: \"3afcec39-f8ad-46ba-b002-cf1465de9072\") " pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-rlqml"
Oct 30 12:55:52.707132 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:52.707114    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3afcec39-f8ad-46ba-b002-cf1465de9072-tmp\") pod \"opendatahub-operator-controller-manager-678d69fff5-rlqml\" (UID: \"3afcec39-f8ad-46ba-b002-cf1465de9072\") " pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-rlqml"
Oct 30 12:55:52.707197 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:52.707148    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml5px\" (UniqueName: \"kubernetes.io/projected/3afcec39-f8ad-46ba-b002-cf1465de9072-kube-api-access-ml5px\") pod \"opendatahub-operator-controller-manager-678d69fff5-rlqml\" (UID: \"3afcec39-f8ad-46ba-b002-cf1465de9072\") " pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-rlqml"
Oct 30 12:55:52.808084 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:52.808059    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml5px\" (UniqueName: \"kubernetes.io/projected/3afcec39-f8ad-46ba-b002-cf1465de9072-kube-api-access-ml5px\") pod \"opendatahub-operator-controller-manager-678d69fff5-rlqml\" (UID: \"3afcec39-f8ad-46ba-b002-cf1465de9072\") " pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-rlqml"
Oct 30 12:55:52.808181 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:52.808092    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3afcec39-f8ad-46ba-b002-cf1465de9072-metrics-cert\") pod \"opendatahub-operator-controller-manager-678d69fff5-rlqml\" (UID: \"3afcec39-f8ad-46ba-b002-cf1465de9072\") " pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-rlqml"
Oct 30 12:55:52.808181 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:52.808114    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3afcec39-f8ad-46ba-b002-cf1465de9072-manifests\") pod \"opendatahub-operator-controller-manager-678d69fff5-rlqml\" (UID: \"3afcec39-f8ad-46ba-b002-cf1465de9072\") " pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-rlqml"
Oct 30 12:55:52.808181 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:52.808137    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3afcec39-f8ad-46ba-b002-cf1465de9072-apiservice-cert\") pod \"opendatahub-operator-controller-manager-678d69fff5-rlqml\" (UID: \"3afcec39-f8ad-46ba-b002-cf1465de9072\") " pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-rlqml"
Oct 30 12:55:52.808315 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:52.808178    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3afcec39-f8ad-46ba-b002-cf1465de9072-webhook-cert\") pod \"opendatahub-operator-controller-manager-678d69fff5-rlqml\" (UID: \"3afcec39-f8ad-46ba-b002-cf1465de9072\") " pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-rlqml"
Oct 30 12:55:52.808315 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:52.808207    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3afcec39-f8ad-46ba-b002-cf1465de9072-tmp\") pod \"opendatahub-operator-controller-manager-678d69fff5-rlqml\" (UID: \"3afcec39-f8ad-46ba-b002-cf1465de9072\") " pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-rlqml"
Oct 30 12:55:52.808579 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:52.808553    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3afcec39-f8ad-46ba-b002-cf1465de9072-manifests\") pod \"opendatahub-operator-controller-manager-678d69fff5-rlqml\" (UID: \"3afcec39-f8ad-46ba-b002-cf1465de9072\") " pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-rlqml"
Oct 30 12:55:52.810654 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:52.810627    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3afcec39-f8ad-46ba-b002-cf1465de9072-tmp\") pod \"opendatahub-operator-controller-manager-678d69fff5-rlqml\" (UID: \"3afcec39-f8ad-46ba-b002-cf1465de9072\") " pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-rlqml"
Oct 30 12:55:52.810654 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:52.810647    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3afcec39-f8ad-46ba-b002-cf1465de9072-metrics-cert\") pod \"opendatahub-operator-controller-manager-678d69fff5-rlqml\" (UID: \"3afcec39-f8ad-46ba-b002-cf1465de9072\") " pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-rlqml"
Oct 30 12:55:52.810815 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:52.810755    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3afcec39-f8ad-46ba-b002-cf1465de9072-apiservice-cert\") pod \"opendatahub-operator-controller-manager-678d69fff5-rlqml\" (UID: \"3afcec39-f8ad-46ba-b002-cf1465de9072\") " pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-rlqml"
Oct 30 12:55:52.810815 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:52.810768    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3afcec39-f8ad-46ba-b002-cf1465de9072-webhook-cert\") pod \"opendatahub-operator-controller-manager-678d69fff5-rlqml\" (UID: \"3afcec39-f8ad-46ba-b002-cf1465de9072\") " pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-rlqml"
Oct 30 12:55:52.832685 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:52.832632    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml5px\" (UniqueName: \"kubernetes.io/projected/3afcec39-f8ad-46ba-b002-cf1465de9072-kube-api-access-ml5px\") pod \"opendatahub-operator-controller-manager-678d69fff5-rlqml\" (UID: \"3afcec39-f8ad-46ba-b002-cf1465de9072\") " pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-rlqml"
Oct 30 12:55:52.889779 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:52.889762    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-rlqml"
Oct 30 12:55:53.272323 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:53.272301    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-678d69fff5-rlqml"]
Oct 30 12:55:53.274969 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:55:53.274940    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3afcec39_f8ad_46ba_b002_cf1465de9072.slice/crio-108d60504b795673e3045e6013a96b64e7c29028cff2ffbf7c584f61f62b7621 WatchSource:0}: Error finding container 108d60504b795673e3045e6013a96b64e7c29028cff2ffbf7c584f61f62b7621: Status 404 returned error can't find the container with id 108d60504b795673e3045e6013a96b64e7c29028cff2ffbf7c584f61f62b7621
Oct 30 12:55:53.978495 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:53.978459    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-rlqml" event={"ID":"3afcec39-f8ad-46ba-b002-cf1465de9072","Type":"ContainerStarted","Data":"108d60504b795673e3045e6013a96b64e7c29028cff2ffbf7c584f61f62b7621"}
Oct 30 12:55:55.988344 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:55.988312    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-rlqml" event={"ID":"3afcec39-f8ad-46ba-b002-cf1465de9072","Type":"ContainerStarted","Data":"b531ed488cecdacab999bf05d9e08420e11073c19bf180fa8c4588b240c2f648"}
Oct 30 12:55:56.994185 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:56.994152    4364 generic.go:356] "Generic (PLEG): container finished" podID="3afcec39-f8ad-46ba-b002-cf1465de9072" containerID="b531ed488cecdacab999bf05d9e08420e11073c19bf180fa8c4588b240c2f648" exitCode=0
Oct 30 12:55:56.994541 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:56.994234    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-rlqml" event={"ID":"3afcec39-f8ad-46ba-b002-cf1465de9072","Type":"ContainerDied","Data":"b531ed488cecdacab999bf05d9e08420e11073c19bf180fa8c4588b240c2f648"}
Oct 30 12:55:57.999869 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:57.999835    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-rlqml" event={"ID":"3afcec39-f8ad-46ba-b002-cf1465de9072","Type":"ContainerStarted","Data":"0fcf325f8f465d29b5c8312ce470badcb415d183e8a533d9e0763c3aad35e69a"}
Oct 30 12:55:58.000273 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:57.999962    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-rlqml"
Oct 30 12:55:58.094154 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:55:58.094114    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-rlqml" podStartSLOduration=2.5568098519999998 podStartE2EDuration="6.094102488s" totalImagesPullingTime="3.037716889s" totalInitContainerRuntime="499.575747ms" isStatefulPod=false podCreationTimestamp="2026-10-30 12:55:52 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:55:58.079931402 +0000 UTC m=+257.246296051" watchObservedRunningTime="2026-10-30 12:55:58.094102488 +0000 UTC m=+257.260467135"
Oct 30 12:56:09.007194 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:09.007164    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-rlqml"
Oct 30 12:56:11.505800 ip-10-0-137-6 kubenswrapper[4364]: E1030 12:56:11.505760    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 12:56:12.447475 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:12.447435    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-65dc4d65b9-59szd"]
Oct 30 12:56:12.451014 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:12.450980    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-65dc4d65b9-59szd"
Oct 30 12:56:12.482980 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:12.482946    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 12:56:12.482980 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:12.482946    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 30 12:56:12.483194 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:12.483119    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 12:56:12.483194 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:12.483143    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 30 12:56:12.485555 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:12.485488    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-65dc4d65b9-59szd"]
Oct 30 12:56:12.485960 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:12.485868    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-vz8qn\""
Oct 30 12:56:12.548296 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:12.548273    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ac493c83-00bc-43fa-b0d1-ec05f21fcd92-tmp\") pod \"kube-auth-proxy-65dc4d65b9-59szd\" (UID: \"ac493c83-00bc-43fa-b0d1-ec05f21fcd92\") " pod="openshift-ingress/kube-auth-proxy-65dc4d65b9-59szd"
Oct 30 12:56:12.548688 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:12.548307    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac493c83-00bc-43fa-b0d1-ec05f21fcd92-tls-certs\") pod \"kube-auth-proxy-65dc4d65b9-59szd\" (UID: \"ac493c83-00bc-43fa-b0d1-ec05f21fcd92\") " pod="openshift-ingress/kube-auth-proxy-65dc4d65b9-59szd"
Oct 30 12:56:12.548688 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:12.548430    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcwnp\" (UniqueName: \"kubernetes.io/projected/ac493c83-00bc-43fa-b0d1-ec05f21fcd92-kube-api-access-fcwnp\") pod \"kube-auth-proxy-65dc4d65b9-59szd\" (UID: \"ac493c83-00bc-43fa-b0d1-ec05f21fcd92\") " pod="openshift-ingress/kube-auth-proxy-65dc4d65b9-59szd"
Oct 30 12:56:12.649595 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:12.649559    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcwnp\" (UniqueName: \"kubernetes.io/projected/ac493c83-00bc-43fa-b0d1-ec05f21fcd92-kube-api-access-fcwnp\") pod \"kube-auth-proxy-65dc4d65b9-59szd\" (UID: \"ac493c83-00bc-43fa-b0d1-ec05f21fcd92\") " pod="openshift-ingress/kube-auth-proxy-65dc4d65b9-59szd"
Oct 30 12:56:12.649744 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:12.649628    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ac493c83-00bc-43fa-b0d1-ec05f21fcd92-tmp\") pod \"kube-auth-proxy-65dc4d65b9-59szd\" (UID: \"ac493c83-00bc-43fa-b0d1-ec05f21fcd92\") " pod="openshift-ingress/kube-auth-proxy-65dc4d65b9-59szd"
Oct 30 12:56:12.649744 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:12.649668    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac493c83-00bc-43fa-b0d1-ec05f21fcd92-tls-certs\") pod \"kube-auth-proxy-65dc4d65b9-59szd\" (UID: \"ac493c83-00bc-43fa-b0d1-ec05f21fcd92\") " pod="openshift-ingress/kube-auth-proxy-65dc4d65b9-59szd"
Oct 30 12:56:12.651904 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:12.651878    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ac493c83-00bc-43fa-b0d1-ec05f21fcd92-tmp\") pod \"kube-auth-proxy-65dc4d65b9-59szd\" (UID: \"ac493c83-00bc-43fa-b0d1-ec05f21fcd92\") " pod="openshift-ingress/kube-auth-proxy-65dc4d65b9-59szd"
Oct 30 12:56:12.652122 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:12.652103    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac493c83-00bc-43fa-b0d1-ec05f21fcd92-tls-certs\") pod \"kube-auth-proxy-65dc4d65b9-59szd\" (UID: \"ac493c83-00bc-43fa-b0d1-ec05f21fcd92\") " pod="openshift-ingress/kube-auth-proxy-65dc4d65b9-59szd"
Oct 30 12:56:12.674856 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:12.674837    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcwnp\" (UniqueName: \"kubernetes.io/projected/ac493c83-00bc-43fa-b0d1-ec05f21fcd92-kube-api-access-fcwnp\") pod \"kube-auth-proxy-65dc4d65b9-59szd\" (UID: \"ac493c83-00bc-43fa-b0d1-ec05f21fcd92\") " pod="openshift-ingress/kube-auth-proxy-65dc4d65b9-59szd"
Oct 30 12:56:12.764156 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:12.764108    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-65dc4d65b9-59szd"
Oct 30 12:56:13.064255 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:13.064230    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-65dc4d65b9-59szd"]
Oct 30 12:56:13.066638 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:56:13.066616    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac493c83_00bc_43fa_b0d1_ec05f21fcd92.slice/crio-8b6231ca9b2ccf89aa65138f00df05733515074a20dc1514af4551e498e9b9d1 WatchSource:0}: Error finding container 8b6231ca9b2ccf89aa65138f00df05733515074a20dc1514af4551e498e9b9d1: Status 404 returned error can't find the container with id 8b6231ca9b2ccf89aa65138f00df05733515074a20dc1514af4551e498e9b9d1
Oct 30 12:56:14.062876 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:14.062837    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-65dc4d65b9-59szd" event={"ID":"ac493c83-00bc-43fa-b0d1-ec05f21fcd92","Type":"ContainerStarted","Data":"8b6231ca9b2ccf89aa65138f00df05733515074a20dc1514af4551e498e9b9d1"}
Oct 30 12:56:15.941205 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:15.941167    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-observability-584f99c788-w8fvw"]
Oct 30 12:56:15.944534 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:15.944513    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-observability-584f99c788-w8fvw"
Oct 30 12:56:15.964276 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:15.964250    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-observability-webhook-cert\""
Oct 30 12:56:15.977962 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:15.977930    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-467jk\" (UniqueName: \"kubernetes.io/projected/a83c7ff7-e667-4c9a-b13b-ac37eb14f3b6-kube-api-access-467jk\") pod \"odh-observability-584f99c788-w8fvw\" (UID: \"a83c7ff7-e667-4c9a-b13b-ac37eb14f3b6\") " pod="opendatahub/odh-observability-584f99c788-w8fvw"
Oct 30 12:56:15.978097 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:15.978027    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-observability-dockercfg-4ktdr\""
Oct 30 12:56:15.978097 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:15.978032    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/a83c7ff7-e667-4c9a-b13b-ac37eb14f3b6-webhook-certs\") pod \"odh-observability-584f99c788-w8fvw\" (UID: \"a83c7ff7-e667-4c9a-b13b-ac37eb14f3b6\") " pod="opendatahub/odh-observability-584f99c788-w8fvw"
Oct 30 12:56:15.979470 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:15.979440    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-observability-584f99c788-w8fvw"]
Oct 30 12:56:16.071820 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:16.071787    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-65dc4d65b9-59szd" event={"ID":"ac493c83-00bc-43fa-b0d1-ec05f21fcd92","Type":"ContainerStarted","Data":"66fcfec6ba0754c00c39bc2f1c4bc4ed9db560849b33c48ed0ee2e51466d4bea"}
Oct 30 12:56:16.078277 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:16.078259    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-467jk\" (UniqueName: \"kubernetes.io/projected/a83c7ff7-e667-4c9a-b13b-ac37eb14f3b6-kube-api-access-467jk\") pod \"odh-observability-584f99c788-w8fvw\" (UID: \"a83c7ff7-e667-4c9a-b13b-ac37eb14f3b6\") " pod="opendatahub/odh-observability-584f99c788-w8fvw"
Oct 30 12:56:16.078335 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:16.078291    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/a83c7ff7-e667-4c9a-b13b-ac37eb14f3b6-webhook-certs\") pod \"odh-observability-584f99c788-w8fvw\" (UID: \"a83c7ff7-e667-4c9a-b13b-ac37eb14f3b6\") " pod="opendatahub/odh-observability-584f99c788-w8fvw"
Oct 30 12:56:16.078442 ip-10-0-137-6 kubenswrapper[4364]: E1030 12:56:16.078425    4364 secret.go:189] Couldn't get secret opendatahub/odh-observability-webhook-cert: secret "odh-observability-webhook-cert" not found
Oct 30 12:56:16.078510 ip-10-0-137-6 kubenswrapper[4364]: E1030 12:56:16.078499    4364 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a83c7ff7-e667-4c9a-b13b-ac37eb14f3b6-webhook-certs podName:a83c7ff7-e667-4c9a-b13b-ac37eb14f3b6 nodeName:}" failed. No retries permitted until 2026-10-30 12:56:16.57847692 +0000 UTC m=+275.744841562 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-certs" (UniqueName: "kubernetes.io/secret/a83c7ff7-e667-4c9a-b13b-ac37eb14f3b6-webhook-certs") pod "odh-observability-584f99c788-w8fvw" (UID: "a83c7ff7-e667-4c9a-b13b-ac37eb14f3b6") : secret "odh-observability-webhook-cert" not found
Oct 30 12:56:16.103769 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:16.103743    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-467jk\" (UniqueName: \"kubernetes.io/projected/a83c7ff7-e667-4c9a-b13b-ac37eb14f3b6-kube-api-access-467jk\") pod \"odh-observability-584f99c788-w8fvw\" (UID: \"a83c7ff7-e667-4c9a-b13b-ac37eb14f3b6\") " pod="opendatahub/odh-observability-584f99c788-w8fvw"
Oct 30 12:56:16.116289 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:16.116230    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-65dc4d65b9-59szd" podStartSLOduration=2.043305699 podStartE2EDuration="4.116216703s" totalImagesPullingTime="2.072911004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:56:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:56:16.115542061 +0000 UTC m=+275.281906710" watchObservedRunningTime="2026-10-30 12:56:16.116216703 +0000 UTC m=+275.282581353"
Oct 30 12:56:16.583052 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:16.583018    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/a83c7ff7-e667-4c9a-b13b-ac37eb14f3b6-webhook-certs\") pod \"odh-observability-584f99c788-w8fvw\" (UID: \"a83c7ff7-e667-4c9a-b13b-ac37eb14f3b6\") " pod="opendatahub/odh-observability-584f99c788-w8fvw"
Oct 30 12:56:16.585621 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:16.585586    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/a83c7ff7-e667-4c9a-b13b-ac37eb14f3b6-webhook-certs\") pod \"odh-observability-584f99c788-w8fvw\" (UID: \"a83c7ff7-e667-4c9a-b13b-ac37eb14f3b6\") " pod="opendatahub/odh-observability-584f99c788-w8fvw"
Oct 30 12:56:16.857105 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:16.857036    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-observability-584f99c788-w8fvw"
Oct 30 12:56:17.156263 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:17.156241    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-observability-584f99c788-w8fvw"]
Oct 30 12:56:17.158193 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:56:17.158168    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda83c7ff7_e667_4c9a_b13b_ac37eb14f3b6.slice/crio-3d8572b4f122992695abf606151b427b6ba556fe18155ae1fedec471b950e673 WatchSource:0}: Error finding container 3d8572b4f122992695abf606151b427b6ba556fe18155ae1fedec471b950e673: Status 404 returned error can't find the container with id 3d8572b4f122992695abf606151b427b6ba556fe18155ae1fedec471b950e673
Oct 30 12:56:18.081400 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:18.081364    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-observability-584f99c788-w8fvw" event={"ID":"a83c7ff7-e667-4c9a-b13b-ac37eb14f3b6","Type":"ContainerStarted","Data":"3d8572b4f122992695abf606151b427b6ba556fe18155ae1fedec471b950e673"}
Oct 30 12:56:19.819336 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:19.818781    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6bc88ffd5-5pkjs"]
Oct 30 12:56:19.825073 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:19.825042    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6bc88ffd5-5pkjs"
Oct 30 12:56:19.842228 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:19.842207    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 12:56:19.842960 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:19.842943    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-xdn6x\""
Oct 30 12:56:19.856418 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:19.856394    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6bc88ffd5-5pkjs"]
Oct 30 12:56:19.908914 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:19.908880    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f540848-0b4f-4398-8a1f-6d1b3d7a379d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6bc88ffd5-5pkjs\" (UID: \"5f540848-0b4f-4398-8a1f-6d1b3d7a379d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6bc88ffd5-5pkjs"
Oct 30 12:56:19.909043 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:19.908941    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f540848-0b4f-4398-8a1f-6d1b3d7a379d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6bc88ffd5-5pkjs\" (UID: \"5f540848-0b4f-4398-8a1f-6d1b3d7a379d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6bc88ffd5-5pkjs"
Oct 30 12:56:19.909043 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:19.909023    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f540848-0b4f-4398-8a1f-6d1b3d7a379d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6bc88ffd5-5pkjs\" (UID: \"5f540848-0b4f-4398-8a1f-6d1b3d7a379d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6bc88ffd5-5pkjs"
Oct 30 12:56:19.909135 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:19.909056    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f540848-0b4f-4398-8a1f-6d1b3d7a379d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6bc88ffd5-5pkjs\" (UID: \"5f540848-0b4f-4398-8a1f-6d1b3d7a379d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6bc88ffd5-5pkjs"
Oct 30 12:56:19.909135 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:19.909107    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f540848-0b4f-4398-8a1f-6d1b3d7a379d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6bc88ffd5-5pkjs\" (UID: \"5f540848-0b4f-4398-8a1f-6d1b3d7a379d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6bc88ffd5-5pkjs"
Oct 30 12:56:19.909257 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:19.909239    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f540848-0b4f-4398-8a1f-6d1b3d7a379d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6bc88ffd5-5pkjs\" (UID: \"5f540848-0b4f-4398-8a1f-6d1b3d7a379d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6bc88ffd5-5pkjs"
Oct 30 12:56:19.909316 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:19.909270    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds5ps\" (UniqueName: \"kubernetes.io/projected/5f540848-0b4f-4398-8a1f-6d1b3d7a379d-kube-api-access-ds5ps\") pod \"data-science-gateway-data-science-gateway-class-6bc88ffd5-5pkjs\" (UID: \"5f540848-0b4f-4398-8a1f-6d1b3d7a379d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6bc88ffd5-5pkjs"
Oct 30 12:56:19.909316 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:19.909304    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f540848-0b4f-4398-8a1f-6d1b3d7a379d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6bc88ffd5-5pkjs\" (UID: \"5f540848-0b4f-4398-8a1f-6d1b3d7a379d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6bc88ffd5-5pkjs"
Oct 30 12:56:19.909399 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:19.909343    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f540848-0b4f-4398-8a1f-6d1b3d7a379d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6bc88ffd5-5pkjs\" (UID: \"5f540848-0b4f-4398-8a1f-6d1b3d7a379d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6bc88ffd5-5pkjs"
Oct 30 12:56:20.009833 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:20.009804    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f540848-0b4f-4398-8a1f-6d1b3d7a379d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6bc88ffd5-5pkjs\" (UID: \"5f540848-0b4f-4398-8a1f-6d1b3d7a379d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6bc88ffd5-5pkjs"
Oct 30 12:56:20.009969 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:20.009890    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f540848-0b4f-4398-8a1f-6d1b3d7a379d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6bc88ffd5-5pkjs\" (UID: \"5f540848-0b4f-4398-8a1f-6d1b3d7a379d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6bc88ffd5-5pkjs"
Oct 30 12:56:20.009969 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:20.009921    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds5ps\" (UniqueName: \"kubernetes.io/projected/5f540848-0b4f-4398-8a1f-6d1b3d7a379d-kube-api-access-ds5ps\") pod \"data-science-gateway-data-science-gateway-class-6bc88ffd5-5pkjs\" (UID: \"5f540848-0b4f-4398-8a1f-6d1b3d7a379d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6bc88ffd5-5pkjs"
Oct 30 12:56:20.009969 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:20.009949    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f540848-0b4f-4398-8a1f-6d1b3d7a379d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6bc88ffd5-5pkjs\" (UID: \"5f540848-0b4f-4398-8a1f-6d1b3d7a379d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6bc88ffd5-5pkjs"
Oct 30 12:56:20.010140 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:20.009972    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f540848-0b4f-4398-8a1f-6d1b3d7a379d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6bc88ffd5-5pkjs\" (UID: \"5f540848-0b4f-4398-8a1f-6d1b3d7a379d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6bc88ffd5-5pkjs"
Oct 30 12:56:20.010140 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:20.010079    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f540848-0b4f-4398-8a1f-6d1b3d7a379d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6bc88ffd5-5pkjs\" (UID: \"5f540848-0b4f-4398-8a1f-6d1b3d7a379d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6bc88ffd5-5pkjs"
Oct 30 12:56:20.010312 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:20.010253    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f540848-0b4f-4398-8a1f-6d1b3d7a379d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6bc88ffd5-5pkjs\" (UID: \"5f540848-0b4f-4398-8a1f-6d1b3d7a379d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6bc88ffd5-5pkjs"
Oct 30 12:56:20.010379 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:20.010346    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f540848-0b4f-4398-8a1f-6d1b3d7a379d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6bc88ffd5-5pkjs\" (UID: \"5f540848-0b4f-4398-8a1f-6d1b3d7a379d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6bc88ffd5-5pkjs"
Oct 30 12:56:20.010541 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:20.010497    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f540848-0b4f-4398-8a1f-6d1b3d7a379d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6bc88ffd5-5pkjs\" (UID: \"5f540848-0b4f-4398-8a1f-6d1b3d7a379d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6bc88ffd5-5pkjs"
Oct 30 12:56:20.010678 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:20.010563    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f540848-0b4f-4398-8a1f-6d1b3d7a379d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6bc88ffd5-5pkjs\" (UID: \"5f540848-0b4f-4398-8a1f-6d1b3d7a379d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6bc88ffd5-5pkjs"
Oct 30 12:56:20.010678 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:20.010625    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f540848-0b4f-4398-8a1f-6d1b3d7a379d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6bc88ffd5-5pkjs\" (UID: \"5f540848-0b4f-4398-8a1f-6d1b3d7a379d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6bc88ffd5-5pkjs"
Oct 30 12:56:20.010678 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:20.010654    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f540848-0b4f-4398-8a1f-6d1b3d7a379d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6bc88ffd5-5pkjs\" (UID: \"5f540848-0b4f-4398-8a1f-6d1b3d7a379d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6bc88ffd5-5pkjs"
Oct 30 12:56:20.010916 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:20.010894    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f540848-0b4f-4398-8a1f-6d1b3d7a379d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6bc88ffd5-5pkjs\" (UID: \"5f540848-0b4f-4398-8a1f-6d1b3d7a379d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6bc88ffd5-5pkjs"
Oct 30 12:56:20.011070 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:20.011048    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f540848-0b4f-4398-8a1f-6d1b3d7a379d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6bc88ffd5-5pkjs\" (UID: \"5f540848-0b4f-4398-8a1f-6d1b3d7a379d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6bc88ffd5-5pkjs"
Oct 30 12:56:20.012758 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:20.012737    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f540848-0b4f-4398-8a1f-6d1b3d7a379d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6bc88ffd5-5pkjs\" (UID: \"5f540848-0b4f-4398-8a1f-6d1b3d7a379d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6bc88ffd5-5pkjs"
Oct 30 12:56:20.012922 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:20.012779    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f540848-0b4f-4398-8a1f-6d1b3d7a379d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6bc88ffd5-5pkjs\" (UID: \"5f540848-0b4f-4398-8a1f-6d1b3d7a379d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6bc88ffd5-5pkjs"
Oct 30 12:56:20.034877 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:20.034855    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds5ps\" (UniqueName: \"kubernetes.io/projected/5f540848-0b4f-4398-8a1f-6d1b3d7a379d-kube-api-access-ds5ps\") pod \"data-science-gateway-data-science-gateway-class-6bc88ffd5-5pkjs\" (UID: \"5f540848-0b4f-4398-8a1f-6d1b3d7a379d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6bc88ffd5-5pkjs"
Oct 30 12:56:20.034979 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:20.034889    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f540848-0b4f-4398-8a1f-6d1b3d7a379d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6bc88ffd5-5pkjs\" (UID: \"5f540848-0b4f-4398-8a1f-6d1b3d7a379d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6bc88ffd5-5pkjs"
Oct 30 12:56:20.091106 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:20.091053    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-observability-584f99c788-w8fvw" event={"ID":"a83c7ff7-e667-4c9a-b13b-ac37eb14f3b6","Type":"ContainerStarted","Data":"97f99067b6184f30c6aee8ea8139c31d4bbd81ac11d402a27080fc9ec5a61ed1"}
Oct 30 12:56:20.091207 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:20.091172    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-observability-584f99c788-w8fvw"
Oct 30 12:56:20.137258 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:20.137219    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-observability-584f99c788-w8fvw" podStartSLOduration=2.841486079 podStartE2EDuration="5.137206941s" totalImagesPullingTime="2.295720862s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:56:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:56:20.136062294 +0000 UTC m=+279.302426941" watchObservedRunningTime="2026-10-30 12:56:20.137206941 +0000 UTC m=+279.303571588"
Oct 30 12:56:20.140741 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:20.140724    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6bc88ffd5-5pkjs"
Oct 30 12:56:20.453995 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:20.453959    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6bc88ffd5-5pkjs"]
Oct 30 12:56:20.455877 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:56:20.455851    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f540848_0b4f_4398_8a1f_6d1b3d7a379d.slice/crio-94f0149a4485be1ff4f3dfc159dda3dfa21a98246021265a358a7f83aa46d1f4 WatchSource:0}: Error finding container 94f0149a4485be1ff4f3dfc159dda3dfa21a98246021265a358a7f83aa46d1f4: Status 404 returned error can't find the container with id 94f0149a4485be1ff4f3dfc159dda3dfa21a98246021265a358a7f83aa46d1f4
Oct 30 12:56:21.095648 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:21.095617    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6bc88ffd5-5pkjs" event={"ID":"5f540848-0b4f-4398-8a1f-6d1b3d7a379d","Type":"ContainerStarted","Data":"94f0149a4485be1ff4f3dfc159dda3dfa21a98246021265a358a7f83aa46d1f4"}
Oct 30 12:56:23.073419 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:23.073386    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:56:23.073703 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:23.073455    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:56:23.073703 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:23.073485    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:56:24.108785 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:24.108751    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6bc88ffd5-5pkjs" event={"ID":"5f540848-0b4f-4398-8a1f-6d1b3d7a379d","Type":"ContainerStarted","Data":"0372210fb42f0ed6149b555a9094eba509e69926fdf82dd7e17f8accd2e6d878"}
Oct 30 12:56:24.141613 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:24.141567    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6bc88ffd5-5pkjs"
Oct 30 12:56:24.146038 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:24.146017    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6bc88ffd5-5pkjs"
Oct 30 12:56:24.237332 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:24.237287    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6bc88ffd5-5pkjs" podStartSLOduration=2.622435184 podStartE2EDuration="5.237273647s" totalImagesPullingTime="2.614838463s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:56:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:56:24.187057104 +0000 UTC m=+283.353421751" watchObservedRunningTime="2026-10-30 12:56:24.237273647 +0000 UTC m=+283.403638294"
Oct 30 12:56:25.112741 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:25.112711    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6bc88ffd5-5pkjs"
Oct 30 12:56:25.113894 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:25.113876    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6bc88ffd5-5pkjs"
Oct 30 12:56:28.705526 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:28.705493    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-nbfjt"]
Oct 30 12:56:28.760969 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:28.760935    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-nbfjt"]
Oct 30 12:56:28.761102 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:28.761020    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-nbfjt"
Oct 30 12:56:28.824973 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:28.824948    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-8lkk5\""
Oct 30 12:56:28.825095 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:28.825080    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Oct 30 12:56:28.883191 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:28.883168    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e802a8b0-e3e0-4edd-aed0-c62e45c384e6-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-nbfjt\" (UID: \"e802a8b0-e3e0-4edd-aed0-c62e45c384e6\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-nbfjt"
Oct 30 12:56:28.883297 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:28.883206    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e802a8b0-e3e0-4edd-aed0-c62e45c384e6-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-nbfjt\" (UID: \"e802a8b0-e3e0-4edd-aed0-c62e45c384e6\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-nbfjt"
Oct 30 12:56:28.883297 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:28.883232    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e802a8b0-e3e0-4edd-aed0-c62e45c384e6-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-nbfjt\" (UID: \"e802a8b0-e3e0-4edd-aed0-c62e45c384e6\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-nbfjt"
Oct 30 12:56:28.883373 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:28.883288    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x44p\" (UniqueName: \"kubernetes.io/projected/e802a8b0-e3e0-4edd-aed0-c62e45c384e6-kube-api-access-2x44p\") pod \"kserve-module-controller-manager-5cfb77bf8c-nbfjt\" (UID: \"e802a8b0-e3e0-4edd-aed0-c62e45c384e6\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-nbfjt"
Oct 30 12:56:28.983917 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:28.983869    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e802a8b0-e3e0-4edd-aed0-c62e45c384e6-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-nbfjt\" (UID: \"e802a8b0-e3e0-4edd-aed0-c62e45c384e6\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-nbfjt"
Oct 30 12:56:28.983917 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:28.983901    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e802a8b0-e3e0-4edd-aed0-c62e45c384e6-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-nbfjt\" (UID: \"e802a8b0-e3e0-4edd-aed0-c62e45c384e6\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-nbfjt"
Oct 30 12:56:28.984053 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:28.983921    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e802a8b0-e3e0-4edd-aed0-c62e45c384e6-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-nbfjt\" (UID: \"e802a8b0-e3e0-4edd-aed0-c62e45c384e6\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-nbfjt"
Oct 30 12:56:28.984053 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:28.983938    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x44p\" (UniqueName: \"kubernetes.io/projected/e802a8b0-e3e0-4edd-aed0-c62e45c384e6-kube-api-access-2x44p\") pod \"kserve-module-controller-manager-5cfb77bf8c-nbfjt\" (UID: \"e802a8b0-e3e0-4edd-aed0-c62e45c384e6\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-nbfjt"
Oct 30 12:56:28.984257 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:28.984241    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e802a8b0-e3e0-4edd-aed0-c62e45c384e6-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-nbfjt\" (UID: \"e802a8b0-e3e0-4edd-aed0-c62e45c384e6\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-nbfjt"
Oct 30 12:56:28.984317 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:28.984302    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e802a8b0-e3e0-4edd-aed0-c62e45c384e6-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-nbfjt\" (UID: \"e802a8b0-e3e0-4edd-aed0-c62e45c384e6\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-nbfjt"
Oct 30 12:56:28.984802 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:28.984785    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e802a8b0-e3e0-4edd-aed0-c62e45c384e6-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-nbfjt\" (UID: \"e802a8b0-e3e0-4edd-aed0-c62e45c384e6\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-nbfjt"
Oct 30 12:56:29.010355 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:29.010327    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x44p\" (UniqueName: \"kubernetes.io/projected/e802a8b0-e3e0-4edd-aed0-c62e45c384e6-kube-api-access-2x44p\") pod \"kserve-module-controller-manager-5cfb77bf8c-nbfjt\" (UID: \"e802a8b0-e3e0-4edd-aed0-c62e45c384e6\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-nbfjt"
Oct 30 12:56:29.084693 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:29.084670    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-nbfjt"
Oct 30 12:56:29.418482 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:29.418458    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-nbfjt"]
Oct 30 12:56:29.420709 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:56:29.420680    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode802a8b0_e3e0_4edd_aed0_c62e45c384e6.slice/crio-a76114008914dcc7afc49c62ee0348defb132715d3d3645e793fd7b27edd6903 WatchSource:0}: Error finding container a76114008914dcc7afc49c62ee0348defb132715d3d3645e793fd7b27edd6903: Status 404 returned error can't find the container with id a76114008914dcc7afc49c62ee0348defb132715d3d3645e793fd7b27edd6903
Oct 30 12:56:30.135290 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:30.135257    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-nbfjt" event={"ID":"e802a8b0-e3e0-4edd-aed0-c62e45c384e6","Type":"ContainerStarted","Data":"a76114008914dcc7afc49c62ee0348defb132715d3d3645e793fd7b27edd6903"}
Oct 30 12:56:31.097537 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:31.097466    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-observability-584f99c788-w8fvw"
Oct 30 12:56:31.140505 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:31.140482    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-nbfjt" event={"ID":"e802a8b0-e3e0-4edd-aed0-c62e45c384e6","Type":"ContainerStarted","Data":"82e1d1866da86fa1412817e5ffc5ac48a49b08479dcb5671cea21e48ee22e87c"}
Oct 30 12:56:31.140844 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:31.140560    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-nbfjt"
Oct 30 12:56:31.200649 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:31.200583    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-nbfjt" podStartSLOduration=1.954166144 podStartE2EDuration="3.200570158s" totalImagesPullingTime="1.246404014s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:56:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:56:31.200131592 +0000 UTC m=+290.366496241" watchObservedRunningTime="2026-10-30 12:56:31.200570158 +0000 UTC m=+290.366934807"
Oct 30 12:56:41.505928 ip-10-0-137-6 kubenswrapper[4364]: E1030 12:56:41.505891    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 12:56:52.183390 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:56:52.183357    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-nbfjt"
Oct 30 12:57:11.513078 ip-10-0-137-6 kubenswrapper[4364]: E1030 12:57:11.513043    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 12:57:14.995312 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:14.995283    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8xkj9"]
Oct 30 12:57:15.008490 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:15.008471    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8xkj9"
Oct 30 12:57:15.027135 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:15.027099    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 12:57:15.040585 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:15.040552    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 12:57:15.040725 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:15.040647    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 12:57:15.040801 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:15.040751    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-glws9\""
Oct 30 12:57:15.042297 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:15.042277    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8xkj9"]
Oct 30 12:57:15.148084 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:15.148055    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/01088007-6fcf-4bab-9f3a-9e0dce4e1cf6-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8xkj9\" (UID: \"01088007-6fcf-4bab-9f3a-9e0dce4e1cf6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8xkj9"
Oct 30 12:57:15.148084 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:15.148083    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/01088007-6fcf-4bab-9f3a-9e0dce4e1cf6-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8xkj9\" (UID: \"01088007-6fcf-4bab-9f3a-9e0dce4e1cf6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8xkj9"
Oct 30 12:57:15.148268 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:15.148102    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phxx8\" (UniqueName: \"kubernetes.io/projected/01088007-6fcf-4bab-9f3a-9e0dce4e1cf6-kube-api-access-phxx8\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8xkj9\" (UID: \"01088007-6fcf-4bab-9f3a-9e0dce4e1cf6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8xkj9"
Oct 30 12:57:15.148268 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:15.148118    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/01088007-6fcf-4bab-9f3a-9e0dce4e1cf6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8xkj9\" (UID: \"01088007-6fcf-4bab-9f3a-9e0dce4e1cf6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8xkj9"
Oct 30 12:57:15.148268 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:15.148209    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/01088007-6fcf-4bab-9f3a-9e0dce4e1cf6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8xkj9\" (UID: \"01088007-6fcf-4bab-9f3a-9e0dce4e1cf6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8xkj9"
Oct 30 12:57:15.148430 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:15.148316    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01088007-6fcf-4bab-9f3a-9e0dce4e1cf6-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8xkj9\" (UID: \"01088007-6fcf-4bab-9f3a-9e0dce4e1cf6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8xkj9"
Oct 30 12:57:15.148430 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:15.148341    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/01088007-6fcf-4bab-9f3a-9e0dce4e1cf6-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8xkj9\" (UID: \"01088007-6fcf-4bab-9f3a-9e0dce4e1cf6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8xkj9"
Oct 30 12:57:15.249324 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:15.249262    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phxx8\" (UniqueName: \"kubernetes.io/projected/01088007-6fcf-4bab-9f3a-9e0dce4e1cf6-kube-api-access-phxx8\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8xkj9\" (UID: \"01088007-6fcf-4bab-9f3a-9e0dce4e1cf6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8xkj9"
Oct 30 12:57:15.249324 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:15.249291    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/01088007-6fcf-4bab-9f3a-9e0dce4e1cf6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8xkj9\" (UID: \"01088007-6fcf-4bab-9f3a-9e0dce4e1cf6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8xkj9"
Oct 30 12:57:15.249324 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:15.249317    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/01088007-6fcf-4bab-9f3a-9e0dce4e1cf6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8xkj9\" (UID: \"01088007-6fcf-4bab-9f3a-9e0dce4e1cf6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8xkj9"
Oct 30 12:57:15.249563 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:15.249353    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01088007-6fcf-4bab-9f3a-9e0dce4e1cf6-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8xkj9\" (UID: \"01088007-6fcf-4bab-9f3a-9e0dce4e1cf6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8xkj9"
Oct 30 12:57:15.249563 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:15.249375    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/01088007-6fcf-4bab-9f3a-9e0dce4e1cf6-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8xkj9\" (UID: \"01088007-6fcf-4bab-9f3a-9e0dce4e1cf6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8xkj9"
Oct 30 12:57:15.249563 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:15.249428    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/01088007-6fcf-4bab-9f3a-9e0dce4e1cf6-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8xkj9\" (UID: \"01088007-6fcf-4bab-9f3a-9e0dce4e1cf6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8xkj9"
Oct 30 12:57:15.249563 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:15.249451    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/01088007-6fcf-4bab-9f3a-9e0dce4e1cf6-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8xkj9\" (UID: \"01088007-6fcf-4bab-9f3a-9e0dce4e1cf6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8xkj9"
Oct 30 12:57:15.250200 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:15.250175    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/01088007-6fcf-4bab-9f3a-9e0dce4e1cf6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8xkj9\" (UID: \"01088007-6fcf-4bab-9f3a-9e0dce4e1cf6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8xkj9"
Oct 30 12:57:15.252205 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:15.252135    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/01088007-6fcf-4bab-9f3a-9e0dce4e1cf6-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8xkj9\" (UID: \"01088007-6fcf-4bab-9f3a-9e0dce4e1cf6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8xkj9"
Oct 30 12:57:15.252309 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:15.252217    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/01088007-6fcf-4bab-9f3a-9e0dce4e1cf6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8xkj9\" (UID: \"01088007-6fcf-4bab-9f3a-9e0dce4e1cf6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8xkj9"
Oct 30 12:57:15.252358 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:15.252320    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/01088007-6fcf-4bab-9f3a-9e0dce4e1cf6-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8xkj9\" (UID: \"01088007-6fcf-4bab-9f3a-9e0dce4e1cf6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8xkj9"
Oct 30 12:57:15.252665 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:15.252647    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/01088007-6fcf-4bab-9f3a-9e0dce4e1cf6-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8xkj9\" (UID: \"01088007-6fcf-4bab-9f3a-9e0dce4e1cf6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8xkj9"
Oct 30 12:57:15.292455 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:15.292430    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01088007-6fcf-4bab-9f3a-9e0dce4e1cf6-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8xkj9\" (UID: \"01088007-6fcf-4bab-9f3a-9e0dce4e1cf6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8xkj9"
Oct 30 12:57:15.292647 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:15.292505    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phxx8\" (UniqueName: \"kubernetes.io/projected/01088007-6fcf-4bab-9f3a-9e0dce4e1cf6-kube-api-access-phxx8\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8xkj9\" (UID: \"01088007-6fcf-4bab-9f3a-9e0dce4e1cf6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8xkj9"
Oct 30 12:57:15.319220 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:15.319202    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8xkj9"
Oct 30 12:57:15.617438 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:15.617417    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-8vv55"]
Oct 30 12:57:15.621046 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:15.621030    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-8vv55"
Oct 30 12:57:15.648391 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:15.648366    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-8vv55"]
Oct 30 12:57:15.652092 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:15.652074    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h58h\" (UniqueName: \"kubernetes.io/projected/44383748-6649-458a-a7b2-aa6c443d12f6-kube-api-access-4h58h\") pod \"kserve-controller-manager-776cc66d47-8vv55\" (UID: \"44383748-6649-458a-a7b2-aa6c443d12f6\") " pod="opendatahub/kserve-controller-manager-776cc66d47-8vv55"
Oct 30 12:57:15.652184 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:15.652110    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/44383748-6649-458a-a7b2-aa6c443d12f6-cert\") pod \"kserve-controller-manager-776cc66d47-8vv55\" (UID: \"44383748-6649-458a-a7b2-aa6c443d12f6\") " pod="opendatahub/kserve-controller-manager-776cc66d47-8vv55"
Oct 30 12:57:15.656112 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:15.656079    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-sf484\""
Oct 30 12:57:15.656232 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:15.656207    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Oct 30 12:57:15.732894 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:15.732873    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8xkj9"]
Oct 30 12:57:15.733519 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:57:15.733495    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01088007_6fcf_4bab_9f3a_9e0dce4e1cf6.slice/crio-75d0f52d2c0654143a597ff1f0bd703ad88a403dcdb5aebb14db9ac50ea9674e WatchSource:0}: Error finding container 75d0f52d2c0654143a597ff1f0bd703ad88a403dcdb5aebb14db9ac50ea9674e: Status 404 returned error can't find the container with id 75d0f52d2c0654143a597ff1f0bd703ad88a403dcdb5aebb14db9ac50ea9674e
Oct 30 12:57:15.735490 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:15.735475    4364 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 12:57:15.752516 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:15.752495    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/44383748-6649-458a-a7b2-aa6c443d12f6-cert\") pod \"kserve-controller-manager-776cc66d47-8vv55\" (UID: \"44383748-6649-458a-a7b2-aa6c443d12f6\") " pod="opendatahub/kserve-controller-manager-776cc66d47-8vv55"
Oct 30 12:57:15.752634 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:15.752562    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h58h\" (UniqueName: \"kubernetes.io/projected/44383748-6649-458a-a7b2-aa6c443d12f6-kube-api-access-4h58h\") pod \"kserve-controller-manager-776cc66d47-8vv55\" (UID: \"44383748-6649-458a-a7b2-aa6c443d12f6\") " pod="opendatahub/kserve-controller-manager-776cc66d47-8vv55"
Oct 30 12:57:15.754822 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:15.754804    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/44383748-6649-458a-a7b2-aa6c443d12f6-cert\") pod \"kserve-controller-manager-776cc66d47-8vv55\" (UID: \"44383748-6649-458a-a7b2-aa6c443d12f6\") " pod="opendatahub/kserve-controller-manager-776cc66d47-8vv55"
Oct 30 12:57:15.800278 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:15.800256    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h58h\" (UniqueName: \"kubernetes.io/projected/44383748-6649-458a-a7b2-aa6c443d12f6-kube-api-access-4h58h\") pod \"kserve-controller-manager-776cc66d47-8vv55\" (UID: \"44383748-6649-458a-a7b2-aa6c443d12f6\") " pod="opendatahub/kserve-controller-manager-776cc66d47-8vv55"
Oct 30 12:57:15.802354 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:15.802335    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-47nwj"]
Oct 30 12:57:15.807433 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:15.807418    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-47nwj"
Oct 30 12:57:15.832108 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:15.832087    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-47nwj"]
Oct 30 12:57:15.844172 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:15.844153    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-zwv52"]
Oct 30 12:57:15.847271 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:15.847258    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-f87fcf56c-zwv52"
Oct 30 12:57:15.852972 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:15.852948    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b39dfb8-d390-440c-bad4-9a51d4b00142-tls-certs\") pod \"model-serving-api-75bd89fdfc-47nwj\" (UID: \"9b39dfb8-d390-440c-bad4-9a51d4b00142\") " pod="opendatahub/model-serving-api-75bd89fdfc-47nwj"
Oct 30 12:57:15.853071 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:15.852993    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/426ba27f-a567-4c9c-a9da-fa6d7d4772da-metrics-certs\") pod \"odh-model-controller-f87fcf56c-zwv52\" (UID: \"426ba27f-a567-4c9c-a9da-fa6d7d4772da\") " pod="opendatahub/odh-model-controller-f87fcf56c-zwv52"
Oct 30 12:57:15.853143 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:15.853123    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/426ba27f-a567-4c9c-a9da-fa6d7d4772da-cert\") pod \"odh-model-controller-f87fcf56c-zwv52\" (UID: \"426ba27f-a567-4c9c-a9da-fa6d7d4772da\") " pod="opendatahub/odh-model-controller-f87fcf56c-zwv52"
Oct 30 12:57:15.853215 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:15.853176    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxln8\" (UniqueName: \"kubernetes.io/projected/426ba27f-a567-4c9c-a9da-fa6d7d4772da-kube-api-access-qxln8\") pod \"odh-model-controller-f87fcf56c-zwv52\" (UID: \"426ba27f-a567-4c9c-a9da-fa6d7d4772da\") " pod="opendatahub/odh-model-controller-f87fcf56c-zwv52"
Oct 30 12:57:15.853274 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:15.853221    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p66mn\" (UniqueName: \"kubernetes.io/projected/9b39dfb8-d390-440c-bad4-9a51d4b00142-kube-api-access-p66mn\") pod \"model-serving-api-75bd89fdfc-47nwj\" (UID: \"9b39dfb8-d390-440c-bad4-9a51d4b00142\") " pod="opendatahub/model-serving-api-75bd89fdfc-47nwj"
Oct 30 12:57:15.867255 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:15.867234    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-zwv52"]
Oct 30 12:57:15.876362 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:15.876317    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Oct 30 12:57:15.876464 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:15.876374    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-qbpsf\""
Oct 30 12:57:15.876532 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:15.876472    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-vflfr\""
Oct 30 12:57:15.876579 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:15.876562    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Oct 30 12:57:15.876648 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:15.876613    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Oct 30 12:57:15.933444 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:15.933423    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-8vv55"
Oct 30 12:57:15.953675 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:15.953650    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/426ba27f-a567-4c9c-a9da-fa6d7d4772da-cert\") pod \"odh-model-controller-f87fcf56c-zwv52\" (UID: \"426ba27f-a567-4c9c-a9da-fa6d7d4772da\") " pod="opendatahub/odh-model-controller-f87fcf56c-zwv52"
Oct 30 12:57:15.953766 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:15.953693    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxln8\" (UniqueName: \"kubernetes.io/projected/426ba27f-a567-4c9c-a9da-fa6d7d4772da-kube-api-access-qxln8\") pod \"odh-model-controller-f87fcf56c-zwv52\" (UID: \"426ba27f-a567-4c9c-a9da-fa6d7d4772da\") " pod="opendatahub/odh-model-controller-f87fcf56c-zwv52"
Oct 30 12:57:15.953877 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:15.953858    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p66mn\" (UniqueName: \"kubernetes.io/projected/9b39dfb8-d390-440c-bad4-9a51d4b00142-kube-api-access-p66mn\") pod \"model-serving-api-75bd89fdfc-47nwj\" (UID: \"9b39dfb8-d390-440c-bad4-9a51d4b00142\") " pod="opendatahub/model-serving-api-75bd89fdfc-47nwj"
Oct 30 12:57:15.953948 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:15.953930    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b39dfb8-d390-440c-bad4-9a51d4b00142-tls-certs\") pod \"model-serving-api-75bd89fdfc-47nwj\" (UID: \"9b39dfb8-d390-440c-bad4-9a51d4b00142\") " pod="opendatahub/model-serving-api-75bd89fdfc-47nwj"
Oct 30 12:57:15.954003 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:15.953988    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/426ba27f-a567-4c9c-a9da-fa6d7d4772da-metrics-certs\") pod \"odh-model-controller-f87fcf56c-zwv52\" (UID: \"426ba27f-a567-4c9c-a9da-fa6d7d4772da\") " pod="opendatahub/odh-model-controller-f87fcf56c-zwv52"
Oct 30 12:57:15.956147 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:15.956120    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/426ba27f-a567-4c9c-a9da-fa6d7d4772da-cert\") pod \"odh-model-controller-f87fcf56c-zwv52\" (UID: \"426ba27f-a567-4c9c-a9da-fa6d7d4772da\") " pod="opendatahub/odh-model-controller-f87fcf56c-zwv52"
Oct 30 12:57:15.956247 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:15.956216    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b39dfb8-d390-440c-bad4-9a51d4b00142-tls-certs\") pod \"model-serving-api-75bd89fdfc-47nwj\" (UID: \"9b39dfb8-d390-440c-bad4-9a51d4b00142\") " pod="opendatahub/model-serving-api-75bd89fdfc-47nwj"
Oct 30 12:57:15.956309 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:15.956259    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/426ba27f-a567-4c9c-a9da-fa6d7d4772da-metrics-certs\") pod \"odh-model-controller-f87fcf56c-zwv52\" (UID: \"426ba27f-a567-4c9c-a9da-fa6d7d4772da\") " pod="opendatahub/odh-model-controller-f87fcf56c-zwv52"
Oct 30 12:57:15.995414 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:15.995392    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxln8\" (UniqueName: \"kubernetes.io/projected/426ba27f-a567-4c9c-a9da-fa6d7d4772da-kube-api-access-qxln8\") pod \"odh-model-controller-f87fcf56c-zwv52\" (UID: \"426ba27f-a567-4c9c-a9da-fa6d7d4772da\") " pod="opendatahub/odh-model-controller-f87fcf56c-zwv52"
Oct 30 12:57:15.996712 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:15.996688    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p66mn\" (UniqueName: \"kubernetes.io/projected/9b39dfb8-d390-440c-bad4-9a51d4b00142-kube-api-access-p66mn\") pod \"model-serving-api-75bd89fdfc-47nwj\" (UID: \"9b39dfb8-d390-440c-bad4-9a51d4b00142\") " pod="opendatahub/model-serving-api-75bd89fdfc-47nwj"
Oct 30 12:57:16.120120 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:16.120094    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-47nwj"
Oct 30 12:57:16.159183 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:16.159158    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-f87fcf56c-zwv52"
Oct 30 12:57:16.233112 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:16.233085    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-8vv55"]
Oct 30 12:57:16.234370 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:57:16.234348    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44383748_6649_458a_a7b2_aa6c443d12f6.slice/crio-8e5347134fcf87ae3a827ed7d2c208000a15ebf8dc44a18130ecba9adba5bd7c WatchSource:0}: Error finding container 8e5347134fcf87ae3a827ed7d2c208000a15ebf8dc44a18130ecba9adba5bd7c: Status 404 returned error can't find the container with id 8e5347134fcf87ae3a827ed7d2c208000a15ebf8dc44a18130ecba9adba5bd7c
Oct 30 12:57:16.332739 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:16.332704    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-8vv55" event={"ID":"44383748-6649-458a-a7b2-aa6c443d12f6","Type":"ContainerStarted","Data":"8e5347134fcf87ae3a827ed7d2c208000a15ebf8dc44a18130ecba9adba5bd7c"}
Oct 30 12:57:16.333832 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:16.333808    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8xkj9" event={"ID":"01088007-6fcf-4bab-9f3a-9e0dce4e1cf6","Type":"ContainerStarted","Data":"75d0f52d2c0654143a597ff1f0bd703ad88a403dcdb5aebb14db9ac50ea9674e"}
Oct 30 12:57:16.417095 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:16.417075    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-47nwj"]
Oct 30 12:57:16.418724 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:57:16.418701    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b39dfb8_d390_440c_bad4_9a51d4b00142.slice/crio-00edbb585169a32389be5c8f3909152746a5c283e142de89ed41d46a634fa319 WatchSource:0}: Error finding container 00edbb585169a32389be5c8f3909152746a5c283e142de89ed41d46a634fa319: Status 404 returned error can't find the container with id 00edbb585169a32389be5c8f3909152746a5c283e142de89ed41d46a634fa319
Oct 30 12:57:16.460683 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:16.460667    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-zwv52"]
Oct 30 12:57:16.462759 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:57:16.462738    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod426ba27f_a567_4c9c_a9da_fa6d7d4772da.slice/crio-260da211f1b5df34b35fdd39ee6f618af8295da68800200f7df764fd3d00022b WatchSource:0}: Error finding container 260da211f1b5df34b35fdd39ee6f618af8295da68800200f7df764fd3d00022b: Status 404 returned error can't find the container with id 260da211f1b5df34b35fdd39ee6f618af8295da68800200f7df764fd3d00022b
Oct 30 12:57:17.347377 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:17.347336    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-f87fcf56c-zwv52" event={"ID":"426ba27f-a567-4c9c-a9da-fa6d7d4772da","Type":"ContainerStarted","Data":"260da211f1b5df34b35fdd39ee6f618af8295da68800200f7df764fd3d00022b"}
Oct 30 12:57:17.350666 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:17.350635    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-47nwj" event={"ID":"9b39dfb8-d390-440c-bad4-9a51d4b00142","Type":"ContainerStarted","Data":"00edbb585169a32389be5c8f3909152746a5c283e142de89ed41d46a634fa319"}
Oct 30 12:57:19.363813 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:19.363778    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-47nwj" event={"ID":"9b39dfb8-d390-440c-bad4-9a51d4b00142","Type":"ContainerStarted","Data":"ff85bfdb6bc238171367c393fe7376e2dde399e7664e5ad144dfd32ea94aae1a"}
Oct 30 12:57:19.364216 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:19.363845    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-47nwj"
Oct 30 12:57:19.365680 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:19.365655    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-8vv55" event={"ID":"44383748-6649-458a-a7b2-aa6c443d12f6","Type":"ContainerStarted","Data":"0c962440de312dfc93efe4caddadb819617a580be708d0a83c0bccb16e17c4f0"}
Oct 30 12:57:19.365826 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:19.365808    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-8vv55"
Oct 30 12:57:19.410376 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:19.410325    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-47nwj" podStartSLOduration=1.936785599 podStartE2EDuration="4.410308098s" totalImagesPullingTime="2.473522499s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:57:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:57:19.408953301 +0000 UTC m=+338.575317948" watchObservedRunningTime="2026-10-30 12:57:19.410308098 +0000 UTC m=+338.576672748"
Oct 30 12:57:19.424904 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:19.424876    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:57:19.425026 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:19.424960    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:57:19.453641 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:19.453583    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-8vv55" podStartSLOduration=1.478505357 podStartE2EDuration="4.453566045s" totalImagesPullingTime="2.975060688s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:57:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:57:19.452055968 +0000 UTC m=+338.618420611" watchObservedRunningTime="2026-10-30 12:57:19.453566045 +0000 UTC m=+338.619930695"
Oct 30 12:57:20.371024 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:20.370976    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-f87fcf56c-zwv52" event={"ID":"426ba27f-a567-4c9c-a9da-fa6d7d4772da","Type":"ContainerStarted","Data":"a39740fd7e364cdd09b96c001da9b798403120873ba30783cfbec30c9a9cddde"}
Oct 30 12:57:20.371440 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:20.371080    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-f87fcf56c-zwv52"
Oct 30 12:57:20.372609 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:20.372580    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8xkj9" event={"ID":"01088007-6fcf-4bab-9f3a-9e0dce4e1cf6","Type":"ContainerStarted","Data":"15f071a714a0aec14ed089363a7411622faff71531febff2ce91e066b7e1143d"}
Oct 30 12:57:20.427818 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:20.427778    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-f87fcf56c-zwv52" podStartSLOduration=2.125051067 podStartE2EDuration="5.427767133s" totalImagesPullingTime="3.302716066s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:57:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:57:20.427308263 +0000 UTC m=+339.593673101" watchObservedRunningTime="2026-10-30 12:57:20.427767133 +0000 UTC m=+339.594131780"
Oct 30 12:57:20.482938 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:20.482896    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8xkj9" podStartSLOduration=2.7945898810000003 podStartE2EDuration="6.482885575s" totalImagesPullingTime="3.688295694s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:57:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:57:20.482503679 +0000 UTC m=+339.648868329" watchObservedRunningTime="2026-10-30 12:57:20.482885575 +0000 UTC m=+339.649250223"
Oct 30 12:57:21.377098 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:21.377061    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8xkj9"
Oct 30 12:57:21.378666 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:21.378645    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8xkj9"
Oct 30 12:57:30.377822 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:30.377789    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-47nwj"
Oct 30 12:57:31.379377 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:31.379341    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-f87fcf56c-zwv52"
Oct 30 12:57:41.503391 ip-10-0-137-6 kubenswrapper[4364]: E1030 12:57:41.503355    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 12:57:50.377544 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:57:50.377513    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-8vv55"
Oct 30 12:58:08.332589 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:08.332556    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axlf64"]
Oct 30 12:58:08.335887 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:08.335867    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axlf64"
Oct 30 12:58:08.365917 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:08.365897    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-mjhjw\""
Oct 30 12:58:08.366034 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:08.365969    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 12:58:08.366034 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:08.365989    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 12:58:08.367518 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:08.367498    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axlf64"]
Oct 30 12:58:08.417678 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:08.417655    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx9mkw"]
Oct 30 12:58:08.420316 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:08.420300    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx9mkw"
Oct 30 12:58:08.441252 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:08.441232    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-8zpdd\""
Oct 30 12:58:08.442695 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:08.442674    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx9mkw"]
Oct 30 12:58:08.458725 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:08.458700    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a33734d3-9656-4f71-ab79-4d15d12a4417-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axlf64\" (UID: \"a33734d3-9656-4f71-ab79-4d15d12a4417\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axlf64"
Oct 30 12:58:08.458837 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:08.458749    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4njj5\" (UniqueName: \"kubernetes.io/projected/a33734d3-9656-4f71-ab79-4d15d12a4417-kube-api-access-4njj5\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axlf64\" (UID: \"a33734d3-9656-4f71-ab79-4d15d12a4417\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axlf64"
Oct 30 12:58:08.458903 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:08.458864    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a33734d3-9656-4f71-ab79-4d15d12a4417-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axlf64\" (UID: \"a33734d3-9656-4f71-ab79-4d15d12a4417\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axlf64"
Oct 30 12:58:08.519609 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:08.519582    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7twqjq"]
Oct 30 12:58:08.523837 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:08.523819    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7twqjq"
Oct 30 12:58:08.541011 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:08.540992    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-pn8sb\""
Oct 30 12:58:08.549454 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:08.549428    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7twqjq"]
Oct 30 12:58:08.559610 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:08.559580    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9ff25953-c619-46b5-bdd7-8f3a5ffe1e68-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx9mkw\" (UID: \"9ff25953-c619-46b5-bdd7-8f3a5ffe1e68\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx9mkw"
Oct 30 12:58:08.559710 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:08.559622    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a33734d3-9656-4f71-ab79-4d15d12a4417-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axlf64\" (UID: \"a33734d3-9656-4f71-ab79-4d15d12a4417\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axlf64"
Oct 30 12:58:08.559758 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:08.559713    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a33734d3-9656-4f71-ab79-4d15d12a4417-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axlf64\" (UID: \"a33734d3-9656-4f71-ab79-4d15d12a4417\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axlf64"
Oct 30 12:58:08.559758 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:08.559748    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9ff25953-c619-46b5-bdd7-8f3a5ffe1e68-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx9mkw\" (UID: \"9ff25953-c619-46b5-bdd7-8f3a5ffe1e68\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx9mkw"
Oct 30 12:58:08.559847 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:08.559784    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4njj5\" (UniqueName: \"kubernetes.io/projected/a33734d3-9656-4f71-ab79-4d15d12a4417-kube-api-access-4njj5\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axlf64\" (UID: \"a33734d3-9656-4f71-ab79-4d15d12a4417\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axlf64"
Oct 30 12:58:08.559885 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:08.559852    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlzl4\" (UniqueName: \"kubernetes.io/projected/9ff25953-c619-46b5-bdd7-8f3a5ffe1e68-kube-api-access-hlzl4\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx9mkw\" (UID: \"9ff25953-c619-46b5-bdd7-8f3a5ffe1e68\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx9mkw"
Oct 30 12:58:08.560014 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:08.559996    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a33734d3-9656-4f71-ab79-4d15d12a4417-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axlf64\" (UID: \"a33734d3-9656-4f71-ab79-4d15d12a4417\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axlf64"
Oct 30 12:58:08.560059 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:08.560036    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a33734d3-9656-4f71-ab79-4d15d12a4417-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axlf64\" (UID: \"a33734d3-9656-4f71-ab79-4d15d12a4417\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axlf64"
Oct 30 12:58:08.601987 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:08.601926    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4njj5\" (UniqueName: \"kubernetes.io/projected/a33734d3-9656-4f71-ab79-4d15d12a4417-kube-api-access-4njj5\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axlf64\" (UID: \"a33734d3-9656-4f71-ab79-4d15d12a4417\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axlf64"
Oct 30 12:58:08.633793 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:08.633768    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6956zs"]
Oct 30 12:58:08.636520 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:08.636504    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6956zs"
Oct 30 12:58:08.648637 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:08.648615    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axlf64"
Oct 30 12:58:08.659890 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:08.659870    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-hr9nr\""
Oct 30 12:58:08.660292 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:08.660268    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9ff25953-c619-46b5-bdd7-8f3a5ffe1e68-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx9mkw\" (UID: \"9ff25953-c619-46b5-bdd7-8f3a5ffe1e68\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx9mkw"
Oct 30 12:58:08.660369 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:08.660341    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/88034a3c-022f-483b-a649-7357840f7818-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7twqjq\" (UID: \"88034a3c-022f-483b-a649-7357840f7818\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7twqjq"
Oct 30 12:58:08.660427 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:08.660368    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6h94\" (UniqueName: \"kubernetes.io/projected/88034a3c-022f-483b-a649-7357840f7818-kube-api-access-t6h94\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7twqjq\" (UID: \"88034a3c-022f-483b-a649-7357840f7818\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7twqjq"
Oct 30 12:58:08.660427 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:08.660403    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9ff25953-c619-46b5-bdd7-8f3a5ffe1e68-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx9mkw\" (UID: \"9ff25953-c619-46b5-bdd7-8f3a5ffe1e68\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx9mkw"
Oct 30 12:58:08.660544 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:08.660442    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/88034a3c-022f-483b-a649-7357840f7818-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7twqjq\" (UID: \"88034a3c-022f-483b-a649-7357840f7818\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7twqjq"
Oct 30 12:58:08.660544 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:08.660477    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlzl4\" (UniqueName: \"kubernetes.io/projected/9ff25953-c619-46b5-bdd7-8f3a5ffe1e68-kube-api-access-hlzl4\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx9mkw\" (UID: \"9ff25953-c619-46b5-bdd7-8f3a5ffe1e68\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx9mkw"
Oct 30 12:58:08.661071 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:08.661049    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9ff25953-c619-46b5-bdd7-8f3a5ffe1e68-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx9mkw\" (UID: \"9ff25953-c619-46b5-bdd7-8f3a5ffe1e68\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx9mkw"
Oct 30 12:58:08.661445 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:08.661422    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9ff25953-c619-46b5-bdd7-8f3a5ffe1e68-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx9mkw\" (UID: \"9ff25953-c619-46b5-bdd7-8f3a5ffe1e68\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx9mkw"
Oct 30 12:58:08.661919 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:08.661900    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6956zs"]
Oct 30 12:58:08.688724 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:08.688700    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlzl4\" (UniqueName: \"kubernetes.io/projected/9ff25953-c619-46b5-bdd7-8f3a5ffe1e68-kube-api-access-hlzl4\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx9mkw\" (UID: \"9ff25953-c619-46b5-bdd7-8f3a5ffe1e68\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx9mkw"
Oct 30 12:58:08.733630 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:08.733592    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx9mkw"
Oct 30 12:58:08.761849 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:08.761816    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/88034a3c-022f-483b-a649-7357840f7818-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7twqjq\" (UID: \"88034a3c-022f-483b-a649-7357840f7818\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7twqjq"
Oct 30 12:58:08.761977 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:08.761852    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6h94\" (UniqueName: \"kubernetes.io/projected/88034a3c-022f-483b-a649-7357840f7818-kube-api-access-t6h94\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7twqjq\" (UID: \"88034a3c-022f-483b-a649-7357840f7818\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7twqjq"
Oct 30 12:58:08.761977 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:08.761890    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/88034a3c-022f-483b-a649-7357840f7818-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7twqjq\" (UID: \"88034a3c-022f-483b-a649-7357840f7818\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7twqjq"
Oct 30 12:58:08.761977 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:08.761932    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff894252-b16e-4844-887e-c97c5351abc3-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6956zs\" (UID: \"ff894252-b16e-4844-887e-c97c5351abc3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6956zs"
Oct 30 12:58:08.762136 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:08.762047    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ff894252-b16e-4844-887e-c97c5351abc3-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6956zs\" (UID: \"ff894252-b16e-4844-887e-c97c5351abc3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6956zs"
Oct 30 12:58:08.762136 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:08.762096    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rztfx\" (UniqueName: \"kubernetes.io/projected/ff894252-b16e-4844-887e-c97c5351abc3-kube-api-access-rztfx\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6956zs\" (UID: \"ff894252-b16e-4844-887e-c97c5351abc3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6956zs"
Oct 30 12:58:08.762257 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:08.762231    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/88034a3c-022f-483b-a649-7357840f7818-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7twqjq\" (UID: \"88034a3c-022f-483b-a649-7357840f7818\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7twqjq"
Oct 30 12:58:08.762314 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:08.762276    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/88034a3c-022f-483b-a649-7357840f7818-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7twqjq\" (UID: \"88034a3c-022f-483b-a649-7357840f7818\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7twqjq"
Oct 30 12:58:08.790346 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:08.790303    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6h94\" (UniqueName: \"kubernetes.io/projected/88034a3c-022f-483b-a649-7357840f7818-kube-api-access-t6h94\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7twqjq\" (UID: \"88034a3c-022f-483b-a649-7357840f7818\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7twqjq"
Oct 30 12:58:08.836089 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:08.836068    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7twqjq"
Oct 30 12:58:08.862865 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:08.862809    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rztfx\" (UniqueName: \"kubernetes.io/projected/ff894252-b16e-4844-887e-c97c5351abc3-kube-api-access-rztfx\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6956zs\" (UID: \"ff894252-b16e-4844-887e-c97c5351abc3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6956zs"
Oct 30 12:58:08.862961 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:08.862936    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff894252-b16e-4844-887e-c97c5351abc3-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6956zs\" (UID: \"ff894252-b16e-4844-887e-c97c5351abc3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6956zs"
Oct 30 12:58:08.863015 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:08.862967    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ff894252-b16e-4844-887e-c97c5351abc3-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6956zs\" (UID: \"ff894252-b16e-4844-887e-c97c5351abc3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6956zs"
Oct 30 12:58:08.863278 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:08.863258    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff894252-b16e-4844-887e-c97c5351abc3-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6956zs\" (UID: \"ff894252-b16e-4844-887e-c97c5351abc3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6956zs"
Oct 30 12:58:08.863339 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:08.863320    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ff894252-b16e-4844-887e-c97c5351abc3-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6956zs\" (UID: \"ff894252-b16e-4844-887e-c97c5351abc3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6956zs"
Oct 30 12:58:08.890704 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:08.890678    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rztfx\" (UniqueName: \"kubernetes.io/projected/ff894252-b16e-4844-887e-c97c5351abc3-kube-api-access-rztfx\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6956zs\" (UID: \"ff894252-b16e-4844-887e-c97c5351abc3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6956zs"
Oct 30 12:58:08.948813 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:08.948790    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6956zs"
Oct 30 12:58:08.949848 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:08.949827    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axlf64"]
Oct 30 12:58:08.950983 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:58:08.950952    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda33734d3_9656_4f71_ab79_4d15d12a4417.slice/crio-f4cd41c621c0122317149123b7770b7a299af4496d69cde1fe3226bf03518af9 WatchSource:0}: Error finding container f4cd41c621c0122317149123b7770b7a299af4496d69cde1fe3226bf03518af9: Status 404 returned error can't find the container with id f4cd41c621c0122317149123b7770b7a299af4496d69cde1fe3226bf03518af9
Oct 30 12:58:09.031460 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:09.031373    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx9mkw"]
Oct 30 12:58:09.034041 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:58:09.034016    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ff25953_c619_46b5_bdd7_8f3a5ffe1e68.slice/crio-f9c45894cb25eb55dd5e90e9b6b4893d18a7eb8cdcaf6c4cb4045fc1fb5a0626 WatchSource:0}: Error finding container f9c45894cb25eb55dd5e90e9b6b4893d18a7eb8cdcaf6c4cb4045fc1fb5a0626: Status 404 returned error can't find the container with id f9c45894cb25eb55dd5e90e9b6b4893d18a7eb8cdcaf6c4cb4045fc1fb5a0626
Oct 30 12:58:09.132836 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:09.132814    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7twqjq"]
Oct 30 12:58:09.135068 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:58:09.135042    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88034a3c_022f_483b_a649_7357840f7818.slice/crio-fa12386638903c7db9dedcef3f46cdb79ddebb71b5b25b23239260df5b664bbc WatchSource:0}: Error finding container fa12386638903c7db9dedcef3f46cdb79ddebb71b5b25b23239260df5b664bbc: Status 404 returned error can't find the container with id fa12386638903c7db9dedcef3f46cdb79ddebb71b5b25b23239260df5b664bbc
Oct 30 12:58:09.253422 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:09.253400    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6956zs"]
Oct 30 12:58:09.292492 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:58:09.292467    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff894252_b16e_4844_887e_c97c5351abc3.slice/crio-7f8b1ce2ce122e5d1070e6fe5eb3fffee8aef24b01a9a3cbcb9c5e6395c46eec WatchSource:0}: Error finding container 7f8b1ce2ce122e5d1070e6fe5eb3fffee8aef24b01a9a3cbcb9c5e6395c46eec: Status 404 returned error can't find the container with id 7f8b1ce2ce122e5d1070e6fe5eb3fffee8aef24b01a9a3cbcb9c5e6395c46eec
Oct 30 12:58:09.594331 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:09.594303    4364 generic.go:356] "Generic (PLEG): container finished" podID="ff894252-b16e-4844-887e-c97c5351abc3" containerID="438caffb204cd828457952f7d183d0009fe39ecbe1a0e06fd59ebe09e6d47515" exitCode=0
Oct 30 12:58:09.594681 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:09.594386    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6956zs" event={"ID":"ff894252-b16e-4844-887e-c97c5351abc3","Type":"ContainerDied","Data":"438caffb204cd828457952f7d183d0009fe39ecbe1a0e06fd59ebe09e6d47515"}
Oct 30 12:58:09.594681 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:09.594444    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6956zs" event={"ID":"ff894252-b16e-4844-887e-c97c5351abc3","Type":"ContainerStarted","Data":"7f8b1ce2ce122e5d1070e6fe5eb3fffee8aef24b01a9a3cbcb9c5e6395c46eec"}
Oct 30 12:58:09.596423 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:09.596375    4364 generic.go:356] "Generic (PLEG): container finished" podID="9ff25953-c619-46b5-bdd7-8f3a5ffe1e68" containerID="8d071711bb6e62f69e8501f63b88028a9dbe5719e3366337a3aa8472eddefe7a" exitCode=0
Oct 30 12:58:09.596499 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:09.596458    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx9mkw" event={"ID":"9ff25953-c619-46b5-bdd7-8f3a5ffe1e68","Type":"ContainerDied","Data":"8d071711bb6e62f69e8501f63b88028a9dbe5719e3366337a3aa8472eddefe7a"}
Oct 30 12:58:09.596566 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:09.596503    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx9mkw" event={"ID":"9ff25953-c619-46b5-bdd7-8f3a5ffe1e68","Type":"ContainerStarted","Data":"f9c45894cb25eb55dd5e90e9b6b4893d18a7eb8cdcaf6c4cb4045fc1fb5a0626"}
Oct 30 12:58:09.598189 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:09.598164    4364 generic.go:356] "Generic (PLEG): container finished" podID="88034a3c-022f-483b-a649-7357840f7818" containerID="2288e77d8ca7ad302911818275b1fbbb9f05223a15cae3f55dcc4308c08fa9a6" exitCode=0
Oct 30 12:58:09.598290 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:09.598235    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7twqjq" event={"ID":"88034a3c-022f-483b-a649-7357840f7818","Type":"ContainerDied","Data":"2288e77d8ca7ad302911818275b1fbbb9f05223a15cae3f55dcc4308c08fa9a6"}
Oct 30 12:58:09.598290 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:09.598279    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7twqjq" event={"ID":"88034a3c-022f-483b-a649-7357840f7818","Type":"ContainerStarted","Data":"fa12386638903c7db9dedcef3f46cdb79ddebb71b5b25b23239260df5b664bbc"}
Oct 30 12:58:09.599948 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:09.599930    4364 generic.go:356] "Generic (PLEG): container finished" podID="a33734d3-9656-4f71-ab79-4d15d12a4417" containerID="bedfed8ccb3bd9bc1c77531226d55b5bfae1dc79cdfc5127ede3712adb38ea3f" exitCode=0
Oct 30 12:58:09.600074 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:09.599991    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axlf64" event={"ID":"a33734d3-9656-4f71-ab79-4d15d12a4417","Type":"ContainerDied","Data":"bedfed8ccb3bd9bc1c77531226d55b5bfae1dc79cdfc5127ede3712adb38ea3f"}
Oct 30 12:58:09.600074 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:09.600021    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axlf64" event={"ID":"a33734d3-9656-4f71-ab79-4d15d12a4417","Type":"ContainerStarted","Data":"f4cd41c621c0122317149123b7770b7a299af4496d69cde1fe3226bf03518af9"}
Oct 30 12:58:10.605903 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:10.605836    4364 generic.go:356] "Generic (PLEG): container finished" podID="9ff25953-c619-46b5-bdd7-8f3a5ffe1e68" containerID="c8e688e6cd41a7804e1bad80150459288b22e222e6eedca51ca5bc0467835d33" exitCode=0
Oct 30 12:58:10.606256 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:10.605916    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx9mkw" event={"ID":"9ff25953-c619-46b5-bdd7-8f3a5ffe1e68","Type":"ContainerDied","Data":"c8e688e6cd41a7804e1bad80150459288b22e222e6eedca51ca5bc0467835d33"}
Oct 30 12:58:10.608444 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:10.608421    4364 generic.go:356] "Generic (PLEG): container finished" podID="88034a3c-022f-483b-a649-7357840f7818" containerID="00070c16613f6fc4bef7a876ab90bdbe2606e384d140498e129af029f2226fd6" exitCode=0
Oct 30 12:58:10.608531 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:10.608504    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7twqjq" event={"ID":"88034a3c-022f-483b-a649-7357840f7818","Type":"ContainerDied","Data":"00070c16613f6fc4bef7a876ab90bdbe2606e384d140498e129af029f2226fd6"}
Oct 30 12:58:10.610830 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:10.610809    4364 generic.go:356] "Generic (PLEG): container finished" podID="a33734d3-9656-4f71-ab79-4d15d12a4417" containerID="c78269c954a86078a6f371080669519c0818e9d9dd6c7070c812b5b723e92e74" exitCode=0
Oct 30 12:58:10.610914 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:10.610837    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axlf64" event={"ID":"a33734d3-9656-4f71-ab79-4d15d12a4417","Type":"ContainerDied","Data":"c78269c954a86078a6f371080669519c0818e9d9dd6c7070c812b5b723e92e74"}
Oct 30 12:58:10.612867 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:10.612846    4364 generic.go:356] "Generic (PLEG): container finished" podID="ff894252-b16e-4844-887e-c97c5351abc3" containerID="6deb91b3cce68f89bff10788d7d3aad7412ce37e41dc3e67222a7441eec737ef" exitCode=0
Oct 30 12:58:10.612953 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:10.612894    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6956zs" event={"ID":"ff894252-b16e-4844-887e-c97c5351abc3","Type":"ContainerDied","Data":"6deb91b3cce68f89bff10788d7d3aad7412ce37e41dc3e67222a7441eec737ef"}
Oct 30 12:58:11.506664 ip-10-0-137-6 kubenswrapper[4364]: E1030 12:58:11.506629    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 12:58:11.619619 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:11.619576    4364 generic.go:356] "Generic (PLEG): container finished" podID="9ff25953-c619-46b5-bdd7-8f3a5ffe1e68" containerID="61d13e9b52fee4ce7f38b60ad2e866221af240fbc8f2b0a094ac780c31150914" exitCode=0
Oct 30 12:58:11.619959 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:11.619696    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx9mkw" event={"ID":"9ff25953-c619-46b5-bdd7-8f3a5ffe1e68","Type":"ContainerDied","Data":"61d13e9b52fee4ce7f38b60ad2e866221af240fbc8f2b0a094ac780c31150914"}
Oct 30 12:58:11.621886 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:11.621864    4364 generic.go:356] "Generic (PLEG): container finished" podID="88034a3c-022f-483b-a649-7357840f7818" containerID="15e381df283ca2bf9b4e0399b673ee541a649df5641e60a51f35c37465cb4043" exitCode=0
Oct 30 12:58:11.621996 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:11.621931    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7twqjq" event={"ID":"88034a3c-022f-483b-a649-7357840f7818","Type":"ContainerDied","Data":"15e381df283ca2bf9b4e0399b673ee541a649df5641e60a51f35c37465cb4043"}
Oct 30 12:58:11.624239 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:11.624220    4364 generic.go:356] "Generic (PLEG): container finished" podID="a33734d3-9656-4f71-ab79-4d15d12a4417" containerID="9cd5b9f060e414e9c094af67e0466ca75b1c17f8aba2564a244199c316279dc7" exitCode=0
Oct 30 12:58:11.624330 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:11.624268    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axlf64" event={"ID":"a33734d3-9656-4f71-ab79-4d15d12a4417","Type":"ContainerDied","Data":"9cd5b9f060e414e9c094af67e0466ca75b1c17f8aba2564a244199c316279dc7"}
Oct 30 12:58:11.626345 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:11.626318    4364 generic.go:356] "Generic (PLEG): container finished" podID="ff894252-b16e-4844-887e-c97c5351abc3" containerID="f727c2811c26fe2cef4c897142afe0d4f8f2731cf61ae3a9e4396f06f950e9b8" exitCode=0
Oct 30 12:58:11.626436 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:11.626365    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6956zs" event={"ID":"ff894252-b16e-4844-887e-c97c5351abc3","Type":"ContainerDied","Data":"f727c2811c26fe2cef4c897142afe0d4f8f2731cf61ae3a9e4396f06f950e9b8"}
Oct 30 12:58:12.793428 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:12.793398    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6956zs"
Oct 30 12:58:12.811538 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:12.811517    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7twqjq"
Oct 30 12:58:12.841776 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:12.841756    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axlf64"
Oct 30 12:58:12.845034 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:12.845013    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx9mkw"
Oct 30 12:58:12.896828 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:12.896696    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff894252-b16e-4844-887e-c97c5351abc3-kube-api-access-rztfx\" (UniqueName: \"kubernetes.io/projected/ff894252-b16e-4844-887e-c97c5351abc3-kube-api-access-rztfx\") pod \"ff894252-b16e-4844-887e-c97c5351abc3\" (UID: \"ff894252-b16e-4844-887e-c97c5351abc3\") "
Oct 30 12:58:12.896951 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:12.896852    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88034a3c-022f-483b-a649-7357840f7818-util\" (UniqueName: \"kubernetes.io/empty-dir/88034a3c-022f-483b-a649-7357840f7818-util\") pod \"88034a3c-022f-483b-a649-7357840f7818\" (UID: \"88034a3c-022f-483b-a649-7357840f7818\") "
Oct 30 12:58:12.896951 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:12.896905    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff894252-b16e-4844-887e-c97c5351abc3-util\" (UniqueName: \"kubernetes.io/empty-dir/ff894252-b16e-4844-887e-c97c5351abc3-util\") pod \"ff894252-b16e-4844-887e-c97c5351abc3\" (UID: \"ff894252-b16e-4844-887e-c97c5351abc3\") "
Oct 30 12:58:12.897068 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:12.896975    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff894252-b16e-4844-887e-c97c5351abc3-bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff894252-b16e-4844-887e-c97c5351abc3-bundle\") pod \"ff894252-b16e-4844-887e-c97c5351abc3\" (UID: \"ff894252-b16e-4844-887e-c97c5351abc3\") "
Oct 30 12:58:12.897068 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:12.897010    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88034a3c-022f-483b-a649-7357840f7818-kube-api-access-t6h94\" (UniqueName: \"kubernetes.io/projected/88034a3c-022f-483b-a649-7357840f7818-kube-api-access-t6h94\") pod \"88034a3c-022f-483b-a649-7357840f7818\" (UID: \"88034a3c-022f-483b-a649-7357840f7818\") "
Oct 30 12:58:12.897068 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:12.897053    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88034a3c-022f-483b-a649-7357840f7818-bundle\" (UniqueName: \"kubernetes.io/empty-dir/88034a3c-022f-483b-a649-7357840f7818-bundle\") pod \"88034a3c-022f-483b-a649-7357840f7818\" (UID: \"88034a3c-022f-483b-a649-7357840f7818\") "
Oct 30 12:58:12.899175 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:12.898106    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff894252-b16e-4844-887e-c97c5351abc3-bundle" pod "ff894252-b16e-4844-887e-c97c5351abc3" (UID: "ff894252-b16e-4844-887e-c97c5351abc3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:58:12.899175 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:12.898232    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88034a3c-022f-483b-a649-7357840f7818-bundle" pod "88034a3c-022f-483b-a649-7357840f7818" (UID: "88034a3c-022f-483b-a649-7357840f7818"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:58:12.899355 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:12.899332    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff894252-b16e-4844-887e-c97c5351abc3-kube-api-access-rztfx" pod "ff894252-b16e-4844-887e-c97c5351abc3" (UID: "ff894252-b16e-4844-887e-c97c5351abc3"). InnerVolumeSpecName "kube-api-access-rztfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:58:12.900094 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:12.900056    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88034a3c-022f-483b-a649-7357840f7818-kube-api-access-t6h94" pod "88034a3c-022f-483b-a649-7357840f7818" (UID: "88034a3c-022f-483b-a649-7357840f7818"). InnerVolumeSpecName "kube-api-access-t6h94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:58:12.904177 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:12.904153    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff894252-b16e-4844-887e-c97c5351abc3-util" pod "ff894252-b16e-4844-887e-c97c5351abc3" (UID: "ff894252-b16e-4844-887e-c97c5351abc3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:58:12.906036 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:12.906012    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88034a3c-022f-483b-a649-7357840f7818-util" pod "88034a3c-022f-483b-a649-7357840f7818" (UID: "88034a3c-022f-483b-a649-7357840f7818"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:58:12.997572 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:12.997548    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a33734d3-9656-4f71-ab79-4d15d12a4417-bundle\" (UniqueName: \"kubernetes.io/empty-dir/a33734d3-9656-4f71-ab79-4d15d12a4417-bundle\") pod \"a33734d3-9656-4f71-ab79-4d15d12a4417\" (UID: \"a33734d3-9656-4f71-ab79-4d15d12a4417\") "
Oct 30 12:58:12.997697 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:12.997585    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ff25953-c619-46b5-bdd7-8f3a5ffe1e68-util\" (UniqueName: \"kubernetes.io/empty-dir/9ff25953-c619-46b5-bdd7-8f3a5ffe1e68-util\") pod \"9ff25953-c619-46b5-bdd7-8f3a5ffe1e68\" (UID: \"9ff25953-c619-46b5-bdd7-8f3a5ffe1e68\") "
Oct 30 12:58:12.997697 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:12.997635    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ff25953-c619-46b5-bdd7-8f3a5ffe1e68-kube-api-access-hlzl4\" (UniqueName: \"kubernetes.io/projected/9ff25953-c619-46b5-bdd7-8f3a5ffe1e68-kube-api-access-hlzl4\") pod \"9ff25953-c619-46b5-bdd7-8f3a5ffe1e68\" (UID: \"9ff25953-c619-46b5-bdd7-8f3a5ffe1e68\") "
Oct 30 12:58:12.997772 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:12.997706    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ff25953-c619-46b5-bdd7-8f3a5ffe1e68-bundle\" (UniqueName: \"kubernetes.io/empty-dir/9ff25953-c619-46b5-bdd7-8f3a5ffe1e68-bundle\") pod \"9ff25953-c619-46b5-bdd7-8f3a5ffe1e68\" (UID: \"9ff25953-c619-46b5-bdd7-8f3a5ffe1e68\") "
Oct 30 12:58:12.997772 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:12.997741    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a33734d3-9656-4f71-ab79-4d15d12a4417-util\" (UniqueName: \"kubernetes.io/empty-dir/a33734d3-9656-4f71-ab79-4d15d12a4417-util\") pod \"a33734d3-9656-4f71-ab79-4d15d12a4417\" (UID: \"a33734d3-9656-4f71-ab79-4d15d12a4417\") "
Oct 30 12:58:12.997849 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:12.997792    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a33734d3-9656-4f71-ab79-4d15d12a4417-kube-api-access-4njj5\" (UniqueName: \"kubernetes.io/projected/a33734d3-9656-4f71-ab79-4d15d12a4417-kube-api-access-4njj5\") pod \"a33734d3-9656-4f71-ab79-4d15d12a4417\" (UID: \"a33734d3-9656-4f71-ab79-4d15d12a4417\") "
Oct 30 12:58:12.998084 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:12.998063    4364 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/88034a3c-022f-483b-a649-7357840f7818-util\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 12:58:12.998141 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:12.998087    4364 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ff894252-b16e-4844-887e-c97c5351abc3-util\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 12:58:12.998141 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:12.998100    4364 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff894252-b16e-4844-887e-c97c5351abc3-bundle\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 12:58:12.998141 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:12.998112    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6h94\" (UniqueName: \"kubernetes.io/projected/88034a3c-022f-483b-a649-7357840f7818-kube-api-access-t6h94\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 12:58:12.998141 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:12.998126    4364 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/88034a3c-022f-483b-a649-7357840f7818-bundle\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 12:58:12.998141 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:12.998139    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rztfx\" (UniqueName: \"kubernetes.io/projected/ff894252-b16e-4844-887e-c97c5351abc3-kube-api-access-rztfx\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 12:58:12.998647 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:12.998475    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a33734d3-9656-4f71-ab79-4d15d12a4417-bundle" pod "a33734d3-9656-4f71-ab79-4d15d12a4417" (UID: "a33734d3-9656-4f71-ab79-4d15d12a4417"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:58:12.998851 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:12.998820    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ff25953-c619-46b5-bdd7-8f3a5ffe1e68-bundle" pod "9ff25953-c619-46b5-bdd7-8f3a5ffe1e68" (UID: "9ff25953-c619-46b5-bdd7-8f3a5ffe1e68"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:58:13.000283 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:13.000263    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a33734d3-9656-4f71-ab79-4d15d12a4417-kube-api-access-4njj5" pod "a33734d3-9656-4f71-ab79-4d15d12a4417" (UID: "a33734d3-9656-4f71-ab79-4d15d12a4417"). InnerVolumeSpecName "kube-api-access-4njj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:58:13.000823 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:13.000800    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ff25953-c619-46b5-bdd7-8f3a5ffe1e68-kube-api-access-hlzl4" pod "9ff25953-c619-46b5-bdd7-8f3a5ffe1e68" (UID: "9ff25953-c619-46b5-bdd7-8f3a5ffe1e68"). InnerVolumeSpecName "kube-api-access-hlzl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:58:13.003535 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:13.003498    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ff25953-c619-46b5-bdd7-8f3a5ffe1e68-util" pod "9ff25953-c619-46b5-bdd7-8f3a5ffe1e68" (UID: "9ff25953-c619-46b5-bdd7-8f3a5ffe1e68"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:58:13.004973 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:13.004949    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a33734d3-9656-4f71-ab79-4d15d12a4417-util" pod "a33734d3-9656-4f71-ab79-4d15d12a4417" (UID: "a33734d3-9656-4f71-ab79-4d15d12a4417"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:58:13.098890 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:13.098866    4364 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a33734d3-9656-4f71-ab79-4d15d12a4417-bundle\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 12:58:13.098890 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:13.098888    4364 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9ff25953-c619-46b5-bdd7-8f3a5ffe1e68-util\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 12:58:13.099021 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:13.098898    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlzl4\" (UniqueName: \"kubernetes.io/projected/9ff25953-c619-46b5-bdd7-8f3a5ffe1e68-kube-api-access-hlzl4\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 12:58:13.099021 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:13.098908    4364 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9ff25953-c619-46b5-bdd7-8f3a5ffe1e68-bundle\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 12:58:13.099021 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:13.098918    4364 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a33734d3-9656-4f71-ab79-4d15d12a4417-util\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 12:58:13.099021 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:13.098926    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4njj5\" (UniqueName: \"kubernetes.io/projected/a33734d3-9656-4f71-ab79-4d15d12a4417-kube-api-access-4njj5\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 12:58:13.638101 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:13.638074    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6956zs"
Oct 30 12:58:13.638287 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:13.638098    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6956zs" event={"ID":"ff894252-b16e-4844-887e-c97c5351abc3","Type":"ContainerDied","Data":"7f8b1ce2ce122e5d1070e6fe5eb3fffee8aef24b01a9a3cbcb9c5e6395c46eec"}
Oct 30 12:58:13.638287 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:13.638163    4364 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f8b1ce2ce122e5d1070e6fe5eb3fffee8aef24b01a9a3cbcb9c5e6395c46eec"
Oct 30 12:58:13.648200 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:13.648175    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx9mkw"
Oct 30 12:58:13.648342 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:13.648176    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx9mkw" event={"ID":"9ff25953-c619-46b5-bdd7-8f3a5ffe1e68","Type":"ContainerDied","Data":"f9c45894cb25eb55dd5e90e9b6b4893d18a7eb8cdcaf6c4cb4045fc1fb5a0626"}
Oct 30 12:58:13.648342 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:13.648319    4364 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9c45894cb25eb55dd5e90e9b6b4893d18a7eb8cdcaf6c4cb4045fc1fb5a0626"
Oct 30 12:58:13.650487 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:13.650455    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7twqjq" event={"ID":"88034a3c-022f-483b-a649-7357840f7818","Type":"ContainerDied","Data":"fa12386638903c7db9dedcef3f46cdb79ddebb71b5b25b23239260df5b664bbc"}
Oct 30 12:58:13.650487 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:13.650481    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7twqjq"
Oct 30 12:58:13.650712 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:13.650505    4364 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa12386638903c7db9dedcef3f46cdb79ddebb71b5b25b23239260df5b664bbc"
Oct 30 12:58:13.653100 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:13.653077    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axlf64" event={"ID":"a33734d3-9656-4f71-ab79-4d15d12a4417","Type":"ContainerDied","Data":"f4cd41c621c0122317149123b7770b7a299af4496d69cde1fe3226bf03518af9"}
Oct 30 12:58:13.653210 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:13.653102    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axlf64"
Oct 30 12:58:13.653210 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:58:13.653114    4364 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4cd41c621c0122317149123b7770b7a299af4496d69cde1fe3226bf03518af9"
Oct 30 12:58:41.502685 ip-10-0-137-6 kubenswrapper[4364]: E1030 12:58:41.502650    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 12:59:11.502458 ip-10-0-137-6 kubenswrapper[4364]: E1030 12:59:11.502418    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 12:59:16.735682 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:16.735648    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-r9ffg"]
Oct 30 12:59:16.736151 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:16.735971    4364 cpu_manager.go:414] "containerMap: removing container" podUID="9ff25953-c619-46b5-bdd7-8f3a5ffe1e68" containerName="pull"
Oct 30 12:59:16.736151 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:16.735983    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ff25953-c619-46b5-bdd7-8f3a5ffe1e68" containerName="pull"
Oct 30 12:59:16.736151 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:16.735993    4364 cpu_manager.go:414] "containerMap: removing container" podUID="ff894252-b16e-4844-887e-c97c5351abc3" containerName="pull"
Oct 30 12:59:16.736151 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:16.735998    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff894252-b16e-4844-887e-c97c5351abc3" containerName="pull"
Oct 30 12:59:16.736151 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:16.736007    4364 cpu_manager.go:414] "containerMap: removing container" podUID="88034a3c-022f-483b-a649-7357840f7818" containerName="util"
Oct 30 12:59:16.736151 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:16.736011    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88034a3c-022f-483b-a649-7357840f7818" containerName="util"
Oct 30 12:59:16.736151 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:16.736017    4364 cpu_manager.go:414] "containerMap: removing container" podUID="ff894252-b16e-4844-887e-c97c5351abc3" containerName="util"
Oct 30 12:59:16.736151 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:16.736022    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff894252-b16e-4844-887e-c97c5351abc3" containerName="util"
Oct 30 12:59:16.736151 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:16.736027    4364 cpu_manager.go:414] "containerMap: removing container" podUID="a33734d3-9656-4f71-ab79-4d15d12a4417" containerName="pull"
Oct 30 12:59:16.736151 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:16.736031    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a33734d3-9656-4f71-ab79-4d15d12a4417" containerName="pull"
Oct 30 12:59:16.736151 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:16.736039    4364 cpu_manager.go:414] "containerMap: removing container" podUID="88034a3c-022f-483b-a649-7357840f7818" containerName="extract"
Oct 30 12:59:16.736151 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:16.736043    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88034a3c-022f-483b-a649-7357840f7818" containerName="extract"
Oct 30 12:59:16.736151 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:16.736056    4364 cpu_manager.go:414] "containerMap: removing container" podUID="9ff25953-c619-46b5-bdd7-8f3a5ffe1e68" containerName="extract"
Oct 30 12:59:16.736151 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:16.736060    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ff25953-c619-46b5-bdd7-8f3a5ffe1e68" containerName="extract"
Oct 30 12:59:16.736151 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:16.736065    4364 cpu_manager.go:414] "containerMap: removing container" podUID="a33734d3-9656-4f71-ab79-4d15d12a4417" containerName="extract"
Oct 30 12:59:16.736151 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:16.736069    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a33734d3-9656-4f71-ab79-4d15d12a4417" containerName="extract"
Oct 30 12:59:16.736151 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:16.736078    4364 cpu_manager.go:414] "containerMap: removing container" podUID="9ff25953-c619-46b5-bdd7-8f3a5ffe1e68" containerName="util"
Oct 30 12:59:16.736151 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:16.736082    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ff25953-c619-46b5-bdd7-8f3a5ffe1e68" containerName="util"
Oct 30 12:59:16.736151 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:16.736087    4364 cpu_manager.go:414] "containerMap: removing container" podUID="88034a3c-022f-483b-a649-7357840f7818" containerName="pull"
Oct 30 12:59:16.736151 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:16.736091    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88034a3c-022f-483b-a649-7357840f7818" containerName="pull"
Oct 30 12:59:16.736151 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:16.736097    4364 cpu_manager.go:414] "containerMap: removing container" podUID="a33734d3-9656-4f71-ab79-4d15d12a4417" containerName="util"
Oct 30 12:59:16.736151 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:16.736101    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a33734d3-9656-4f71-ab79-4d15d12a4417" containerName="util"
Oct 30 12:59:16.736151 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:16.736107    4364 cpu_manager.go:414] "containerMap: removing container" podUID="ff894252-b16e-4844-887e-c97c5351abc3" containerName="extract"
Oct 30 12:59:16.736151 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:16.736110    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff894252-b16e-4844-887e-c97c5351abc3" containerName="extract"
Oct 30 12:59:16.736924 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:16.736168    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="88034a3c-022f-483b-a649-7357840f7818" containerName="extract"
Oct 30 12:59:16.736924 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:16.736176    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ff25953-c619-46b5-bdd7-8f3a5ffe1e68" containerName="extract"
Oct 30 12:59:16.736924 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:16.736182    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="a33734d3-9656-4f71-ab79-4d15d12a4417" containerName="extract"
Oct 30 12:59:16.736924 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:16.736188    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff894252-b16e-4844-887e-c97c5351abc3" containerName="extract"
Oct 30 12:59:16.739163 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:16.739144    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-r9ffg"
Oct 30 12:59:16.760959 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:16.760933    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-bwksz\""
Oct 30 12:59:16.775669 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:16.775647    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-r9ffg"]
Oct 30 12:59:16.905242 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:16.905210    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5afd5fcb-484f-407b-bbeb-6e7263bfac7a-istio-envoy\") pod \"maas-default-gateway-openshift-default-64d4cc998c-r9ffg\" (UID: \"5afd5fcb-484f-407b-bbeb-6e7263bfac7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-r9ffg"
Oct 30 12:59:16.905242 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:16.905243    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5afd5fcb-484f-407b-bbeb-6e7263bfac7a-istio-data\") pod \"maas-default-gateway-openshift-default-64d4cc998c-r9ffg\" (UID: \"5afd5fcb-484f-407b-bbeb-6e7263bfac7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-r9ffg"
Oct 30 12:59:16.905444 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:16.905270    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqbql\" (UniqueName: \"kubernetes.io/projected/5afd5fcb-484f-407b-bbeb-6e7263bfac7a-kube-api-access-rqbql\") pod \"maas-default-gateway-openshift-default-64d4cc998c-r9ffg\" (UID: \"5afd5fcb-484f-407b-bbeb-6e7263bfac7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-r9ffg"
Oct 30 12:59:16.905444 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:16.905329    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5afd5fcb-484f-407b-bbeb-6e7263bfac7a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-64d4cc998c-r9ffg\" (UID: \"5afd5fcb-484f-407b-bbeb-6e7263bfac7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-r9ffg"
Oct 30 12:59:16.905516 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:16.905472    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5afd5fcb-484f-407b-bbeb-6e7263bfac7a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-64d4cc998c-r9ffg\" (UID: \"5afd5fcb-484f-407b-bbeb-6e7263bfac7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-r9ffg"
Oct 30 12:59:16.905563 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:16.905514    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5afd5fcb-484f-407b-bbeb-6e7263bfac7a-credential-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-r9ffg\" (UID: \"5afd5fcb-484f-407b-bbeb-6e7263bfac7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-r9ffg"
Oct 30 12:59:16.905647 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:16.905628    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5afd5fcb-484f-407b-bbeb-6e7263bfac7a-istio-token\") pod \"maas-default-gateway-openshift-default-64d4cc998c-r9ffg\" (UID: \"5afd5fcb-484f-407b-bbeb-6e7263bfac7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-r9ffg"
Oct 30 12:59:16.905711 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:16.905673    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5afd5fcb-484f-407b-bbeb-6e7263bfac7a-workload-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-r9ffg\" (UID: \"5afd5fcb-484f-407b-bbeb-6e7263bfac7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-r9ffg"
Oct 30 12:59:16.905762 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:16.905708    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5afd5fcb-484f-407b-bbeb-6e7263bfac7a-workload-certs\") pod \"maas-default-gateway-openshift-default-64d4cc998c-r9ffg\" (UID: \"5afd5fcb-484f-407b-bbeb-6e7263bfac7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-r9ffg"
Oct 30 12:59:17.006283 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:17.006223    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5afd5fcb-484f-407b-bbeb-6e7263bfac7a-workload-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-r9ffg\" (UID: \"5afd5fcb-484f-407b-bbeb-6e7263bfac7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-r9ffg"
Oct 30 12:59:17.006283 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:17.006256    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5afd5fcb-484f-407b-bbeb-6e7263bfac7a-workload-certs\") pod \"maas-default-gateway-openshift-default-64d4cc998c-r9ffg\" (UID: \"5afd5fcb-484f-407b-bbeb-6e7263bfac7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-r9ffg"
Oct 30 12:59:17.006283 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:17.006276    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5afd5fcb-484f-407b-bbeb-6e7263bfac7a-istio-envoy\") pod \"maas-default-gateway-openshift-default-64d4cc998c-r9ffg\" (UID: \"5afd5fcb-484f-407b-bbeb-6e7263bfac7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-r9ffg"
Oct 30 12:59:17.006512 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:17.006292    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5afd5fcb-484f-407b-bbeb-6e7263bfac7a-istio-data\") pod \"maas-default-gateway-openshift-default-64d4cc998c-r9ffg\" (UID: \"5afd5fcb-484f-407b-bbeb-6e7263bfac7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-r9ffg"
Oct 30 12:59:17.006512 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:17.006320    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqbql\" (UniqueName: \"kubernetes.io/projected/5afd5fcb-484f-407b-bbeb-6e7263bfac7a-kube-api-access-rqbql\") pod \"maas-default-gateway-openshift-default-64d4cc998c-r9ffg\" (UID: \"5afd5fcb-484f-407b-bbeb-6e7263bfac7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-r9ffg"
Oct 30 12:59:17.006512 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:17.006339    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5afd5fcb-484f-407b-bbeb-6e7263bfac7a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-64d4cc998c-r9ffg\" (UID: \"5afd5fcb-484f-407b-bbeb-6e7263bfac7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-r9ffg"
Oct 30 12:59:17.006512 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:17.006362    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5afd5fcb-484f-407b-bbeb-6e7263bfac7a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-64d4cc998c-r9ffg\" (UID: \"5afd5fcb-484f-407b-bbeb-6e7263bfac7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-r9ffg"
Oct 30 12:59:17.006512 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:17.006414    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5afd5fcb-484f-407b-bbeb-6e7263bfac7a-credential-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-r9ffg\" (UID: \"5afd5fcb-484f-407b-bbeb-6e7263bfac7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-r9ffg"
Oct 30 12:59:17.006512 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:17.006458    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5afd5fcb-484f-407b-bbeb-6e7263bfac7a-istio-token\") pod \"maas-default-gateway-openshift-default-64d4cc998c-r9ffg\" (UID: \"5afd5fcb-484f-407b-bbeb-6e7263bfac7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-r9ffg"
Oct 30 12:59:17.006894 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:17.006684    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5afd5fcb-484f-407b-bbeb-6e7263bfac7a-workload-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-r9ffg\" (UID: \"5afd5fcb-484f-407b-bbeb-6e7263bfac7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-r9ffg"
Oct 30 12:59:17.006894 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:17.006748    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5afd5fcb-484f-407b-bbeb-6e7263bfac7a-workload-certs\") pod \"maas-default-gateway-openshift-default-64d4cc998c-r9ffg\" (UID: \"5afd5fcb-484f-407b-bbeb-6e7263bfac7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-r9ffg"
Oct 30 12:59:17.006894 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:17.006801    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5afd5fcb-484f-407b-bbeb-6e7263bfac7a-istio-data\") pod \"maas-default-gateway-openshift-default-64d4cc998c-r9ffg\" (UID: \"5afd5fcb-484f-407b-bbeb-6e7263bfac7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-r9ffg"
Oct 30 12:59:17.007087 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:17.006930    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5afd5fcb-484f-407b-bbeb-6e7263bfac7a-credential-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-r9ffg\" (UID: \"5afd5fcb-484f-407b-bbeb-6e7263bfac7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-r9ffg"
Oct 30 12:59:17.007202 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:17.007182    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5afd5fcb-484f-407b-bbeb-6e7263bfac7a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-64d4cc998c-r9ffg\" (UID: \"5afd5fcb-484f-407b-bbeb-6e7263bfac7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-r9ffg"
Oct 30 12:59:17.008670 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:17.008650    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5afd5fcb-484f-407b-bbeb-6e7263bfac7a-istio-envoy\") pod \"maas-default-gateway-openshift-default-64d4cc998c-r9ffg\" (UID: \"5afd5fcb-484f-407b-bbeb-6e7263bfac7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-r9ffg"
Oct 30 12:59:17.008815 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:17.008798    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5afd5fcb-484f-407b-bbeb-6e7263bfac7a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-64d4cc998c-r9ffg\" (UID: \"5afd5fcb-484f-407b-bbeb-6e7263bfac7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-r9ffg"
Oct 30 12:59:17.032079 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:17.032054    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqbql\" (UniqueName: \"kubernetes.io/projected/5afd5fcb-484f-407b-bbeb-6e7263bfac7a-kube-api-access-rqbql\") pod \"maas-default-gateway-openshift-default-64d4cc998c-r9ffg\" (UID: \"5afd5fcb-484f-407b-bbeb-6e7263bfac7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-r9ffg"
Oct 30 12:59:17.032197 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:17.032081    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5afd5fcb-484f-407b-bbeb-6e7263bfac7a-istio-token\") pod \"maas-default-gateway-openshift-default-64d4cc998c-r9ffg\" (UID: \"5afd5fcb-484f-407b-bbeb-6e7263bfac7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-r9ffg"
Oct 30 12:59:17.052030 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:17.052010    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-r9ffg"
Oct 30 12:59:17.372237 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:17.372212    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-r9ffg"]
Oct 30 12:59:17.373983 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:59:17.373945    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5afd5fcb_484f_407b_bbeb_6e7263bfac7a.slice/crio-801d4df035a7a93765e1c3182b96c87e9aa286cf739087ffde98dda994aa1af5 WatchSource:0}: Error finding container 801d4df035a7a93765e1c3182b96c87e9aa286cf739087ffde98dda994aa1af5: Status 404 returned error can't find the container with id 801d4df035a7a93765e1c3182b96c87e9aa286cf739087ffde98dda994aa1af5
Oct 30 12:59:17.375953 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:17.375924    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:59:17.376034 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:17.375984    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:59:17.376034 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:17.376013    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 12:59:17.942083 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:17.942055    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-r9ffg" event={"ID":"5afd5fcb-484f-407b-bbeb-6e7263bfac7a","Type":"ContainerStarted","Data":"2eec2874af42efd7cb01f6ae53bd78cc6a08c2cd56402b6438ca988b732b831d"}
Oct 30 12:59:17.942466 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:17.942110    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-r9ffg" event={"ID":"5afd5fcb-484f-407b-bbeb-6e7263bfac7a","Type":"ContainerStarted","Data":"801d4df035a7a93765e1c3182b96c87e9aa286cf739087ffde98dda994aa1af5"}
Oct 30 12:59:18.018133 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:18.018086    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-r9ffg" podStartSLOduration=2.018072851 podStartE2EDuration="2.018072851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:59:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:59:18.003739431 +0000 UTC m=+457.170104079" watchObservedRunningTime="2026-10-30 12:59:18.018072851 +0000 UTC m=+457.184437498"
Oct 30 12:59:18.052557 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:18.052534    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-r9ffg"
Oct 30 12:59:18.057209 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:18.057191    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-r9ffg"
Oct 30 12:59:18.946657 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:18.946630    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-r9ffg"
Oct 30 12:59:18.947426 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:18.947408    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-r9ffg"
Oct 30 12:59:32.991559 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:32.991528    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 12:59:32.995082 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:32.995062    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-szbp8"
Oct 30 12:59:33.028564 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:33.028527    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 12:59:33.036383 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:33.036361    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/dddc7c6a-09a5-496c-aa3f-cd8c50ed5a13-config-file\") pod \"limitador-limitador-7c9b87d574-szbp8\" (UID: \"dddc7c6a-09a5-496c-aa3f-cd8c50ed5a13\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-szbp8"
Oct 30 12:59:33.036495 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:33.036396    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcsb4\" (UniqueName: \"kubernetes.io/projected/dddc7c6a-09a5-496c-aa3f-cd8c50ed5a13-kube-api-access-qcsb4\") pod \"limitador-limitador-7c9b87d574-szbp8\" (UID: \"dddc7c6a-09a5-496c-aa3f-cd8c50ed5a13\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-szbp8"
Oct 30 12:59:33.037500 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:33.037485    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 30 12:59:33.037589 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:33.037516    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-hj987\""
Oct 30 12:59:33.040555 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:33.040538    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 12:59:33.040630 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:33.040589    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 12:59:33.137323 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:33.137291    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/dddc7c6a-09a5-496c-aa3f-cd8c50ed5a13-config-file\") pod \"limitador-limitador-7c9b87d574-szbp8\" (UID: \"dddc7c6a-09a5-496c-aa3f-cd8c50ed5a13\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-szbp8"
Oct 30 12:59:33.137471 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:33.137334    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcsb4\" (UniqueName: \"kubernetes.io/projected/dddc7c6a-09a5-496c-aa3f-cd8c50ed5a13-kube-api-access-qcsb4\") pod \"limitador-limitador-7c9b87d574-szbp8\" (UID: \"dddc7c6a-09a5-496c-aa3f-cd8c50ed5a13\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-szbp8"
Oct 30 12:59:33.137952 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:33.137934    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/dddc7c6a-09a5-496c-aa3f-cd8c50ed5a13-config-file\") pod \"limitador-limitador-7c9b87d574-szbp8\" (UID: \"dddc7c6a-09a5-496c-aa3f-cd8c50ed5a13\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-szbp8"
Oct 30 12:59:33.168705 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:33.168677    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcsb4\" (UniqueName: \"kubernetes.io/projected/dddc7c6a-09a5-496c-aa3f-cd8c50ed5a13-kube-api-access-qcsb4\") pod \"limitador-limitador-7c9b87d574-szbp8\" (UID: \"dddc7c6a-09a5-496c-aa3f-cd8c50ed5a13\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-szbp8"
Oct 30 12:59:33.175273 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:33.175249    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 12:59:33.308065 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:33.307994    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-szbp8"
Oct 30 12:59:33.608251 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:33.608225    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 12:59:33.609691 ip-10-0-137-6 kubenswrapper[4364]: W1030 12:59:33.609658    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddddc7c6a_09a5_496c_aa3f_cd8c50ed5a13.slice/crio-28725d3927cdeda3a6ff9eab4d66bd0e4c19850e11878bb394863ca9b2afae00 WatchSource:0}: Error finding container 28725d3927cdeda3a6ff9eab4d66bd0e4c19850e11878bb394863ca9b2afae00: Status 404 returned error can't find the container with id 28725d3927cdeda3a6ff9eab4d66bd0e4c19850e11878bb394863ca9b2afae00
Oct 30 12:59:34.019318 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:34.019283    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-szbp8" event={"ID":"dddc7c6a-09a5-496c-aa3f-cd8c50ed5a13","Type":"ContainerStarted","Data":"28725d3927cdeda3a6ff9eab4d66bd0e4c19850e11878bb394863ca9b2afae00"}
Oct 30 12:59:36.030395 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:36.030357    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-szbp8" event={"ID":"dddc7c6a-09a5-496c-aa3f-cd8c50ed5a13","Type":"ContainerStarted","Data":"2807060d797e92e23a011757704ec9b27ab09f775365d93d04bae9742afe3e6b"}
Oct 30 12:59:36.030914 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:36.030417    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-szbp8"
Oct 30 12:59:36.075890 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:36.075837    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-szbp8" podStartSLOduration=2.111525261 podStartE2EDuration="4.07582263s" totalImagesPullingTime="1.964297369s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:59:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:59:36.073971948 +0000 UTC m=+475.240336595" watchObservedRunningTime="2026-10-30 12:59:36.07582263 +0000 UTC m=+475.242187278"
Oct 30 12:59:41.503884 ip-10-0-137-6 kubenswrapper[4364]: E1030 12:59:41.503844    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 12:59:47.035823 ip-10-0-137-6 kubenswrapper[4364]: I1030 12:59:47.035787    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-szbp8"
Oct 30 13:00:11.502943 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:00:11.502904    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:00:41.505685 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:00:41.505648    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:00:58.615053 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:00:58.615019    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:01:08.479615 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:01:08.479572    4364 generic.go:356] "Generic (PLEG): container finished" podID="5afd5fcb-484f-407b-bbeb-6e7263bfac7a" containerID="2eec2874af42efd7cb01f6ae53bd78cc6a08c2cd56402b6438ca988b732b831d" exitCode=0
Oct 30 13:01:08.480005 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:01:08.479644    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-r9ffg" event={"ID":"5afd5fcb-484f-407b-bbeb-6e7263bfac7a","Type":"ContainerDied","Data":"2eec2874af42efd7cb01f6ae53bd78cc6a08c2cd56402b6438ca988b732b831d"}
Oct 30 13:01:08.480070 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:01:08.480051    4364 scope.go:122] "RemoveContainer" containerID="2eec2874af42efd7cb01f6ae53bd78cc6a08c2cd56402b6438ca988b732b831d"
Oct 30 13:01:08.481281 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:01:08.481248    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:01:08.481345 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:01:08.481332    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:01:08.481390 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:01:08.481374    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:01:09.508922 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:01:09.508888    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-r9ffg" event={"ID":"5afd5fcb-484f-407b-bbeb-6e7263bfac7a","Type":"ContainerStarted","Data":"0948198ee3d4b0f0298c79afec052f1eea723ebc73915b28e37f37fce302216b"}
Oct 30 13:01:10.052872 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:01:10.052838    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-r9ffg"
Oct 30 13:01:10.053073 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:01:10.053010    4364 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-r9ffg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 30 13:01:10.053157 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:01:10.053068    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-r9ffg" podUID="5afd5fcb-484f-407b-bbeb-6e7263bfac7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 30 13:01:11.052856 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:01:11.052822    4364 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-r9ffg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 30 13:01:11.053248 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:01:11.052873    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-r9ffg" podUID="5afd5fcb-484f-407b-bbeb-6e7263bfac7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 30 13:01:11.524519 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:01:11.524485    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:01:12.052593 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:01:12.052558    4364 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-r9ffg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 30 13:01:12.052783 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:01:12.052630    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-r9ffg" podUID="5afd5fcb-484f-407b-bbeb-6e7263bfac7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 30 13:01:13.052477 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:01:13.052450    4364 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-r9ffg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 30 13:01:13.052886 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:01:13.052499    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-r9ffg" podUID="5afd5fcb-484f-407b-bbeb-6e7263bfac7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 30 13:01:14.053203 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:01:14.053169    4364 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-r9ffg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 30 13:01:14.053622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:01:14.053218    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-r9ffg" podUID="5afd5fcb-484f-407b-bbeb-6e7263bfac7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 30 13:01:15.056539 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:01:15.056507    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-r9ffg"
Oct 30 13:01:15.057056 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:01:15.056912    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-r9ffg"
Oct 30 13:01:15.057651 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:01:15.057629    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-r9ffg"
Oct 30 13:01:41.502965 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:01:41.502932    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:01:56.232488 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:01:56.232451    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:01:56.881910 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:01:56.881876    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-n7v9l"]
Oct 30 13:01:56.886166 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:01:56.886137    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-n7v9l"
Oct 30 13:01:56.906743 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:01:56.906716    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-n7v9l"]
Oct 30 13:01:56.917726 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:01:56.917704    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Oct 30 13:01:56.918151 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:01:56.918137    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 30 13:01:56.918237 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:01:56.918148    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 30 13:01:56.918564 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:01:56.918545    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-tdk84\""
Oct 30 13:01:57.074439 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:01:57.074408    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3bdb4e6-148f-45c7-9fdd-1f0846e21699-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-n7v9l\" (UID: \"d3bdb4e6-148f-45c7-9fdd-1f0846e21699\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-n7v9l"
Oct 30 13:01:57.074656 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:01:57.074476    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3bdb4e6-148f-45c7-9fdd-1f0846e21699-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-n7v9l\" (UID: \"d3bdb4e6-148f-45c7-9fdd-1f0846e21699\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-n7v9l"
Oct 30 13:01:57.074656 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:01:57.074590    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvd8d\" (UniqueName: \"kubernetes.io/projected/d3bdb4e6-148f-45c7-9fdd-1f0846e21699-kube-api-access-bvd8d\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-n7v9l\" (UID: \"d3bdb4e6-148f-45c7-9fdd-1f0846e21699\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-n7v9l"
Oct 30 13:01:57.075662 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:01:57.074680    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3bdb4e6-148f-45c7-9fdd-1f0846e21699-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-n7v9l\" (UID: \"d3bdb4e6-148f-45c7-9fdd-1f0846e21699\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-n7v9l"
Oct 30 13:01:57.075662 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:01:57.074734    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3bdb4e6-148f-45c7-9fdd-1f0846e21699-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-n7v9l\" (UID: \"d3bdb4e6-148f-45c7-9fdd-1f0846e21699\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-n7v9l"
Oct 30 13:01:57.075662 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:01:57.074763    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3bdb4e6-148f-45c7-9fdd-1f0846e21699-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-n7v9l\" (UID: \"d3bdb4e6-148f-45c7-9fdd-1f0846e21699\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-n7v9l"
Oct 30 13:01:57.175640 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:01:57.175549    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3bdb4e6-148f-45c7-9fdd-1f0846e21699-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-n7v9l\" (UID: \"d3bdb4e6-148f-45c7-9fdd-1f0846e21699\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-n7v9l"
Oct 30 13:01:57.175640 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:01:57.175591    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3bdb4e6-148f-45c7-9fdd-1f0846e21699-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-n7v9l\" (UID: \"d3bdb4e6-148f-45c7-9fdd-1f0846e21699\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-n7v9l"
Oct 30 13:01:57.175640 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:01:57.175631    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3bdb4e6-148f-45c7-9fdd-1f0846e21699-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-n7v9l\" (UID: \"d3bdb4e6-148f-45c7-9fdd-1f0846e21699\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-n7v9l"
Oct 30 13:01:57.175883 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:01:57.175686    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3bdb4e6-148f-45c7-9fdd-1f0846e21699-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-n7v9l\" (UID: \"d3bdb4e6-148f-45c7-9fdd-1f0846e21699\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-n7v9l"
Oct 30 13:01:57.175883 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:01:57.175706    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3bdb4e6-148f-45c7-9fdd-1f0846e21699-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-n7v9l\" (UID: \"d3bdb4e6-148f-45c7-9fdd-1f0846e21699\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-n7v9l"
Oct 30 13:01:57.175883 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:01:57.175733    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvd8d\" (UniqueName: \"kubernetes.io/projected/d3bdb4e6-148f-45c7-9fdd-1f0846e21699-kube-api-access-bvd8d\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-n7v9l\" (UID: \"d3bdb4e6-148f-45c7-9fdd-1f0846e21699\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-n7v9l"
Oct 30 13:01:57.176048 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:01:57.175937    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3bdb4e6-148f-45c7-9fdd-1f0846e21699-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-n7v9l\" (UID: \"d3bdb4e6-148f-45c7-9fdd-1f0846e21699\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-n7v9l"
Oct 30 13:01:57.176048 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:01:57.176007    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3bdb4e6-148f-45c7-9fdd-1f0846e21699-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-n7v9l\" (UID: \"d3bdb4e6-148f-45c7-9fdd-1f0846e21699\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-n7v9l"
Oct 30 13:01:57.176114 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:01:57.176055    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3bdb4e6-148f-45c7-9fdd-1f0846e21699-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-n7v9l\" (UID: \"d3bdb4e6-148f-45c7-9fdd-1f0846e21699\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-n7v9l"
Oct 30 13:01:57.177871 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:01:57.177843    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3bdb4e6-148f-45c7-9fdd-1f0846e21699-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-n7v9l\" (UID: \"d3bdb4e6-148f-45c7-9fdd-1f0846e21699\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-n7v9l"
Oct 30 13:01:57.178183 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:01:57.178153    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3bdb4e6-148f-45c7-9fdd-1f0846e21699-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-n7v9l\" (UID: \"d3bdb4e6-148f-45c7-9fdd-1f0846e21699\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-n7v9l"
Oct 30 13:01:57.201614 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:01:57.201585    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvd8d\" (UniqueName: \"kubernetes.io/projected/d3bdb4e6-148f-45c7-9fdd-1f0846e21699-kube-api-access-bvd8d\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-n7v9l\" (UID: \"d3bdb4e6-148f-45c7-9fdd-1f0846e21699\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-n7v9l"
Oct 30 13:01:57.501825 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:01:57.501765    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-n7v9l"
Oct 30 13:01:57.808594 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:01:57.808519    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-n7v9l"]
Oct 30 13:01:57.811657 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:01:57.811627    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3bdb4e6_148f_45c7_9fdd_1f0846e21699.slice/crio-8299799b2e2f9e188619fadf2411267d308e3b481122e3a404504b152f5d432b WatchSource:0}: Error finding container 8299799b2e2f9e188619fadf2411267d308e3b481122e3a404504b152f5d432b: Status 404 returned error can't find the container with id 8299799b2e2f9e188619fadf2411267d308e3b481122e3a404504b152f5d432b
Oct 30 13:01:58.751967 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:01:58.751927    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-n7v9l" event={"ID":"d3bdb4e6-148f-45c7-9fdd-1f0846e21699","Type":"ContainerStarted","Data":"8299799b2e2f9e188619fadf2411267d308e3b481122e3a404504b152f5d432b"}
Oct 30 13:01:59.757879 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:01:59.757842    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-n7v9l" event={"ID":"d3bdb4e6-148f-45c7-9fdd-1f0846e21699","Type":"ContainerStarted","Data":"af8afbaa0063ce43fb88df748462d28566044ee3743c63a5d50a46666d23811a"}
Oct 30 13:01:59.808148 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:01:59.808099    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-n7v9l" podStartSLOduration=2.73461787 podStartE2EDuration="3.808085481s" totalImagesPullingTime="1.073467611s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:01:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:01:59.803989463 +0000 UTC m=+618.970354111" watchObservedRunningTime="2026-10-30 13:01:59.808085481 +0000 UTC m=+618.974450128"
Oct 30 13:02:00.428741 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:00.428713    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:02:00.502232 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:00.502204    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-n7v9l"
Oct 30 13:02:00.775212 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:00.775126    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-m6h9s"]
Oct 30 13:02:00.778893 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:00.778871    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m6h9s"
Oct 30 13:02:00.812049 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:00.812016    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Oct 30 13:02:00.814277 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:00.814254    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-m6h9s"]
Oct 30 13:02:00.907151 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:00.907112    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f73c0035-9dd8-47c5-9ea6-e1e3aba4968e-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-m6h9s\" (UID: \"f73c0035-9dd8-47c5-9ea6-e1e3aba4968e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m6h9s"
Oct 30 13:02:00.907315 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:00.907164    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f73c0035-9dd8-47c5-9ea6-e1e3aba4968e-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-m6h9s\" (UID: \"f73c0035-9dd8-47c5-9ea6-e1e3aba4968e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m6h9s"
Oct 30 13:02:00.907315 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:00.907205    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f73c0035-9dd8-47c5-9ea6-e1e3aba4968e-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-m6h9s\" (UID: \"f73c0035-9dd8-47c5-9ea6-e1e3aba4968e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m6h9s"
Oct 30 13:02:00.907394 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:00.907355    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f73c0035-9dd8-47c5-9ea6-e1e3aba4968e-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-m6h9s\" (UID: \"f73c0035-9dd8-47c5-9ea6-e1e3aba4968e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m6h9s"
Oct 30 13:02:00.907394 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:00.907385    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f73c0035-9dd8-47c5-9ea6-e1e3aba4968e-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-m6h9s\" (UID: \"f73c0035-9dd8-47c5-9ea6-e1e3aba4968e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m6h9s"
Oct 30 13:02:00.907457 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:00.907444    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ckgk\" (UniqueName: \"kubernetes.io/projected/f73c0035-9dd8-47c5-9ea6-e1e3aba4968e-kube-api-access-2ckgk\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-m6h9s\" (UID: \"f73c0035-9dd8-47c5-9ea6-e1e3aba4968e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m6h9s"
Oct 30 13:02:01.008237 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:01.008183    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f73c0035-9dd8-47c5-9ea6-e1e3aba4968e-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-m6h9s\" (UID: \"f73c0035-9dd8-47c5-9ea6-e1e3aba4968e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m6h9s"
Oct 30 13:02:01.008237 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:01.008239    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f73c0035-9dd8-47c5-9ea6-e1e3aba4968e-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-m6h9s\" (UID: \"f73c0035-9dd8-47c5-9ea6-e1e3aba4968e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m6h9s"
Oct 30 13:02:01.008514 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:01.008271    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f73c0035-9dd8-47c5-9ea6-e1e3aba4968e-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-m6h9s\" (UID: \"f73c0035-9dd8-47c5-9ea6-e1e3aba4968e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m6h9s"
Oct 30 13:02:01.008514 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:01.008341    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f73c0035-9dd8-47c5-9ea6-e1e3aba4968e-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-m6h9s\" (UID: \"f73c0035-9dd8-47c5-9ea6-e1e3aba4968e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m6h9s"
Oct 30 13:02:01.008514 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:01.008375    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f73c0035-9dd8-47c5-9ea6-e1e3aba4968e-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-m6h9s\" (UID: \"f73c0035-9dd8-47c5-9ea6-e1e3aba4968e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m6h9s"
Oct 30 13:02:01.008514 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:01.008411    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ckgk\" (UniqueName: \"kubernetes.io/projected/f73c0035-9dd8-47c5-9ea6-e1e3aba4968e-kube-api-access-2ckgk\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-m6h9s\" (UID: \"f73c0035-9dd8-47c5-9ea6-e1e3aba4968e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m6h9s"
Oct 30 13:02:01.009236 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:01.009205    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f73c0035-9dd8-47c5-9ea6-e1e3aba4968e-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-m6h9s\" (UID: \"f73c0035-9dd8-47c5-9ea6-e1e3aba4968e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m6h9s"
Oct 30 13:02:01.009364 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:01.009231    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f73c0035-9dd8-47c5-9ea6-e1e3aba4968e-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-m6h9s\" (UID: \"f73c0035-9dd8-47c5-9ea6-e1e3aba4968e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m6h9s"
Oct 30 13:02:01.009364 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:01.009291    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f73c0035-9dd8-47c5-9ea6-e1e3aba4968e-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-m6h9s\" (UID: \"f73c0035-9dd8-47c5-9ea6-e1e3aba4968e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m6h9s"
Oct 30 13:02:01.011586 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:01.011562    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f73c0035-9dd8-47c5-9ea6-e1e3aba4968e-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-m6h9s\" (UID: \"f73c0035-9dd8-47c5-9ea6-e1e3aba4968e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m6h9s"
Oct 30 13:02:01.011998 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:01.011975    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f73c0035-9dd8-47c5-9ea6-e1e3aba4968e-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-m6h9s\" (UID: \"f73c0035-9dd8-47c5-9ea6-e1e3aba4968e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m6h9s"
Oct 30 13:02:01.037433 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:01.037358    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ckgk\" (UniqueName: \"kubernetes.io/projected/f73c0035-9dd8-47c5-9ea6-e1e3aba4968e-kube-api-access-2ckgk\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-m6h9s\" (UID: \"f73c0035-9dd8-47c5-9ea6-e1e3aba4968e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m6h9s"
Oct 30 13:02:01.092263 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:01.092209    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m6h9s"
Oct 30 13:02:01.406835 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:01.406804    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-m6h9s"]
Oct 30 13:02:01.408924 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:02:01.408896    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf73c0035_9dd8_47c5_9ea6_e1e3aba4968e.slice/crio-7913a151dfc767aced01afc6d7cfcac0897f9cb0960bbfce3c0509ed8f1d0e37 WatchSource:0}: Error finding container 7913a151dfc767aced01afc6d7cfcac0897f9cb0960bbfce3c0509ed8f1d0e37: Status 404 returned error can't find the container with id 7913a151dfc767aced01afc6d7cfcac0897f9cb0960bbfce3c0509ed8f1d0e37
Oct 30 13:02:01.770973 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:01.770941    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m6h9s" event={"ID":"f73c0035-9dd8-47c5-9ea6-e1e3aba4968e","Type":"ContainerStarted","Data":"55303ff0a33dfc995a42840cdc7b3ba5a26de5b1166eda730e5be2302fd884c4"}
Oct 30 13:02:01.771110 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:01.770987    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m6h9s" event={"ID":"f73c0035-9dd8-47c5-9ea6-e1e3aba4968e","Type":"ContainerStarted","Data":"7913a151dfc767aced01afc6d7cfcac0897f9cb0960bbfce3c0509ed8f1d0e37"}
Oct 30 13:02:01.820946 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:01.820901    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m6h9s" podStartSLOduration=1.61354488 podStartE2EDuration="1.82088712s" totalImagesPullingTime="207.34224ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:01.819172817 +0000 UTC m=+620.985537468" watchObservedRunningTime="2026-10-30 13:02:01.82088712 +0000 UTC m=+620.987251767"
Oct 30 13:02:02.093111 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:02.093031    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m6h9s"
Oct 30 13:02:04.797289 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:04.797253    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:02:07.476115 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:07.476080    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-tvhvc"]
Oct 30 13:02:07.479787 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:07.479766    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-tvhvc"
Oct 30 13:02:07.501860 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:07.501828    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-tvhvc"]
Oct 30 13:02:07.502208 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:07.501996    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-n7v9l"
Oct 30 13:02:07.513555 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:07.513530    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Oct 30 13:02:07.515012 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:07.514987    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-n7v9l"
Oct 30 13:02:07.567015 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:07.566988    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fda756b1-bfd0-426a-b036-13c90355172a-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-tvhvc\" (UID: \"fda756b1-bfd0-426a-b036-13c90355172a\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-tvhvc"
Oct 30 13:02:07.567205 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:07.567032    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fda756b1-bfd0-426a-b036-13c90355172a-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-tvhvc\" (UID: \"fda756b1-bfd0-426a-b036-13c90355172a\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-tvhvc"
Oct 30 13:02:07.567205 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:07.567050    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fda756b1-bfd0-426a-b036-13c90355172a-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-tvhvc\" (UID: \"fda756b1-bfd0-426a-b036-13c90355172a\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-tvhvc"
Oct 30 13:02:07.567205 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:07.567107    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv9g2\" (UniqueName: \"kubernetes.io/projected/fda756b1-bfd0-426a-b036-13c90355172a-kube-api-access-rv9g2\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-tvhvc\" (UID: \"fda756b1-bfd0-426a-b036-13c90355172a\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-tvhvc"
Oct 30 13:02:07.567395 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:07.567310    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fda756b1-bfd0-426a-b036-13c90355172a-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-tvhvc\" (UID: \"fda756b1-bfd0-426a-b036-13c90355172a\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-tvhvc"
Oct 30 13:02:07.567395 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:07.567334    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fda756b1-bfd0-426a-b036-13c90355172a-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-tvhvc\" (UID: \"fda756b1-bfd0-426a-b036-13c90355172a\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-tvhvc"
Oct 30 13:02:07.670671 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:07.670629    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fda756b1-bfd0-426a-b036-13c90355172a-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-tvhvc\" (UID: \"fda756b1-bfd0-426a-b036-13c90355172a\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-tvhvc"
Oct 30 13:02:07.671088 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:07.671059    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fda756b1-bfd0-426a-b036-13c90355172a-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-tvhvc\" (UID: \"fda756b1-bfd0-426a-b036-13c90355172a\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-tvhvc"
Oct 30 13:02:07.671088 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:07.671102    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fda756b1-bfd0-426a-b036-13c90355172a-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-tvhvc\" (UID: \"fda756b1-bfd0-426a-b036-13c90355172a\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-tvhvc"
Oct 30 13:02:07.671088 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:07.671129    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv9g2\" (UniqueName: \"kubernetes.io/projected/fda756b1-bfd0-426a-b036-13c90355172a-kube-api-access-rv9g2\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-tvhvc\" (UID: \"fda756b1-bfd0-426a-b036-13c90355172a\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-tvhvc"
Oct 30 13:02:07.671088 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:07.671210    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fda756b1-bfd0-426a-b036-13c90355172a-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-tvhvc\" (UID: \"fda756b1-bfd0-426a-b036-13c90355172a\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-tvhvc"
Oct 30 13:02:07.671088 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:07.671240    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fda756b1-bfd0-426a-b036-13c90355172a-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-tvhvc\" (UID: \"fda756b1-bfd0-426a-b036-13c90355172a\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-tvhvc"
Oct 30 13:02:07.671905 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:07.671587    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fda756b1-bfd0-426a-b036-13c90355172a-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-tvhvc\" (UID: \"fda756b1-bfd0-426a-b036-13c90355172a\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-tvhvc"
Oct 30 13:02:07.671905 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:07.671861    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fda756b1-bfd0-426a-b036-13c90355172a-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-tvhvc\" (UID: \"fda756b1-bfd0-426a-b036-13c90355172a\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-tvhvc"
Oct 30 13:02:07.672003 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:07.671988    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fda756b1-bfd0-426a-b036-13c90355172a-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-tvhvc\" (UID: \"fda756b1-bfd0-426a-b036-13c90355172a\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-tvhvc"
Oct 30 13:02:07.674210 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:07.674183    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fda756b1-bfd0-426a-b036-13c90355172a-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-tvhvc\" (UID: \"fda756b1-bfd0-426a-b036-13c90355172a\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-tvhvc"
Oct 30 13:02:07.674752 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:07.674733    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fda756b1-bfd0-426a-b036-13c90355172a-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-tvhvc\" (UID: \"fda756b1-bfd0-426a-b036-13c90355172a\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-tvhvc"
Oct 30 13:02:07.697235 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:07.697203    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv9g2\" (UniqueName: \"kubernetes.io/projected/fda756b1-bfd0-426a-b036-13c90355172a-kube-api-access-rv9g2\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-tvhvc\" (UID: \"fda756b1-bfd0-426a-b036-13c90355172a\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-tvhvc"
Oct 30 13:02:07.793294 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:07.793205    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-tvhvc"
Oct 30 13:02:07.814307 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:07.814278    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-n7v9l"
Oct 30 13:02:08.103941 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:08.103916    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-tvhvc"]
Oct 30 13:02:08.105649 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:02:08.105590    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfda756b1_bfd0_426a_b036_13c90355172a.slice/crio-a1ed5463a3ea233efaf47b5047971a42532ba902c058760005572b5906a454f4 WatchSource:0}: Error finding container a1ed5463a3ea233efaf47b5047971a42532ba902c058760005572b5906a454f4: Status 404 returned error can't find the container with id a1ed5463a3ea233efaf47b5047971a42532ba902c058760005572b5906a454f4
Oct 30 13:02:08.808919 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:08.808879    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-tvhvc" event={"ID":"fda756b1-bfd0-426a-b036-13c90355172a","Type":"ContainerStarted","Data":"896b4a7588b465898ab1a6a6fa61c7a248a2dd1aa8d8e78276994439aeb290ec"}
Oct 30 13:02:08.809357 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:08.808950    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-tvhvc" event={"ID":"fda756b1-bfd0-426a-b036-13c90355172a","Type":"ContainerStarted","Data":"a1ed5463a3ea233efaf47b5047971a42532ba902c058760005572b5906a454f4"}
Oct 30 13:02:08.879610 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:08.879548    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-tvhvc" podStartSLOduration=1.696937079 podStartE2EDuration="1.879532905s" totalImagesPullingTime="182.595826ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:08.877982081 +0000 UTC m=+628.044346730" watchObservedRunningTime="2026-10-30 13:02:08.879532905 +0000 UTC m=+628.045897552"
Oct 30 13:02:09.793511 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:09.793469    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-tvhvc"
Oct 30 13:02:11.092556 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:11.092522    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m6h9s"
Oct 30 13:02:11.107102 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:11.107075    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m6h9s"
Oct 30 13:02:11.503313 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:02:11.503280    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:02:11.838717 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:11.838635    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m6h9s"
Oct 30 13:02:12.119901 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:12.119872    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:02:17.794320 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:17.794283    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-tvhvc"
Oct 30 13:02:17.808031 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:17.808005    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-tvhvc"
Oct 30 13:02:17.865980 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:17.865954    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-tvhvc"
Oct 30 13:02:18.180308 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:18.180279    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2gfr2"]
Oct 30 13:02:18.183677 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:18.183656    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2gfr2"
Oct 30 13:02:18.210444 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:18.210423    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2gfr2"]
Oct 30 13:02:18.268037 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:18.268003    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35d8c3ab-6289-40b1-ac0b-5e155420ee38-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-64d4cc998c-2gfr2\" (UID: \"35d8c3ab-6289-40b1-ac0b-5e155420ee38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2gfr2"
Oct 30 13:02:18.268037 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:18.268038    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35d8c3ab-6289-40b1-ac0b-5e155420ee38-istio-envoy\") pod \"maas-default-gateway-openshift-default-64d4cc998c-2gfr2\" (UID: \"35d8c3ab-6289-40b1-ac0b-5e155420ee38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2gfr2"
Oct 30 13:02:18.268249 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:18.268135    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35d8c3ab-6289-40b1-ac0b-5e155420ee38-istio-token\") pod \"maas-default-gateway-openshift-default-64d4cc998c-2gfr2\" (UID: \"35d8c3ab-6289-40b1-ac0b-5e155420ee38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2gfr2"
Oct 30 13:02:18.268249 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:18.268198    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35d8c3ab-6289-40b1-ac0b-5e155420ee38-workload-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-2gfr2\" (UID: \"35d8c3ab-6289-40b1-ac0b-5e155420ee38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2gfr2"
Oct 30 13:02:18.268249 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:18.268232    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74zcw\" (UniqueName: \"kubernetes.io/projected/35d8c3ab-6289-40b1-ac0b-5e155420ee38-kube-api-access-74zcw\") pod \"maas-default-gateway-openshift-default-64d4cc998c-2gfr2\" (UID: \"35d8c3ab-6289-40b1-ac0b-5e155420ee38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2gfr2"
Oct 30 13:02:18.268372 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:18.268261    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35d8c3ab-6289-40b1-ac0b-5e155420ee38-workload-certs\") pod \"maas-default-gateway-openshift-default-64d4cc998c-2gfr2\" (UID: \"35d8c3ab-6289-40b1-ac0b-5e155420ee38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2gfr2"
Oct 30 13:02:18.268372 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:18.268299    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35d8c3ab-6289-40b1-ac0b-5e155420ee38-istio-podinfo\") pod \"maas-default-gateway-openshift-default-64d4cc998c-2gfr2\" (UID: \"35d8c3ab-6289-40b1-ac0b-5e155420ee38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2gfr2"
Oct 30 13:02:18.268372 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:18.268361    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35d8c3ab-6289-40b1-ac0b-5e155420ee38-credential-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-2gfr2\" (UID: \"35d8c3ab-6289-40b1-ac0b-5e155420ee38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2gfr2"
Oct 30 13:02:18.268475 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:18.268387    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35d8c3ab-6289-40b1-ac0b-5e155420ee38-istio-data\") pod \"maas-default-gateway-openshift-default-64d4cc998c-2gfr2\" (UID: \"35d8c3ab-6289-40b1-ac0b-5e155420ee38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2gfr2"
Oct 30 13:02:18.369301 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:18.369259    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35d8c3ab-6289-40b1-ac0b-5e155420ee38-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-64d4cc998c-2gfr2\" (UID: \"35d8c3ab-6289-40b1-ac0b-5e155420ee38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2gfr2"
Oct 30 13:02:18.369301 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:18.369304    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35d8c3ab-6289-40b1-ac0b-5e155420ee38-istio-envoy\") pod \"maas-default-gateway-openshift-default-64d4cc998c-2gfr2\" (UID: \"35d8c3ab-6289-40b1-ac0b-5e155420ee38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2gfr2"
Oct 30 13:02:18.369558 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:18.369344    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35d8c3ab-6289-40b1-ac0b-5e155420ee38-istio-token\") pod \"maas-default-gateway-openshift-default-64d4cc998c-2gfr2\" (UID: \"35d8c3ab-6289-40b1-ac0b-5e155420ee38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2gfr2"
Oct 30 13:02:18.369558 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:18.369365    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35d8c3ab-6289-40b1-ac0b-5e155420ee38-workload-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-2gfr2\" (UID: \"35d8c3ab-6289-40b1-ac0b-5e155420ee38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2gfr2"
Oct 30 13:02:18.369558 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:18.369394    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74zcw\" (UniqueName: \"kubernetes.io/projected/35d8c3ab-6289-40b1-ac0b-5e155420ee38-kube-api-access-74zcw\") pod \"maas-default-gateway-openshift-default-64d4cc998c-2gfr2\" (UID: \"35d8c3ab-6289-40b1-ac0b-5e155420ee38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2gfr2"
Oct 30 13:02:18.369558 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:18.369433    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35d8c3ab-6289-40b1-ac0b-5e155420ee38-workload-certs\") pod \"maas-default-gateway-openshift-default-64d4cc998c-2gfr2\" (UID: \"35d8c3ab-6289-40b1-ac0b-5e155420ee38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2gfr2"
Oct 30 13:02:18.369558 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:18.369452    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35d8c3ab-6289-40b1-ac0b-5e155420ee38-istio-podinfo\") pod \"maas-default-gateway-openshift-default-64d4cc998c-2gfr2\" (UID: \"35d8c3ab-6289-40b1-ac0b-5e155420ee38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2gfr2"
Oct 30 13:02:18.369558 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:18.369490    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35d8c3ab-6289-40b1-ac0b-5e155420ee38-credential-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-2gfr2\" (UID: \"35d8c3ab-6289-40b1-ac0b-5e155420ee38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2gfr2"
Oct 30 13:02:18.369558 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:18.369551    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35d8c3ab-6289-40b1-ac0b-5e155420ee38-istio-data\") pod \"maas-default-gateway-openshift-default-64d4cc998c-2gfr2\" (UID: \"35d8c3ab-6289-40b1-ac0b-5e155420ee38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2gfr2"
Oct 30 13:02:18.370036 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:18.370011    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35d8c3ab-6289-40b1-ac0b-5e155420ee38-istio-data\") pod \"maas-default-gateway-openshift-default-64d4cc998c-2gfr2\" (UID: \"35d8c3ab-6289-40b1-ac0b-5e155420ee38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2gfr2"
Oct 30 13:02:18.370148 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:18.370018    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35d8c3ab-6289-40b1-ac0b-5e155420ee38-credential-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-2gfr2\" (UID: \"35d8c3ab-6289-40b1-ac0b-5e155420ee38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2gfr2"
Oct 30 13:02:18.370148 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:18.370086    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35d8c3ab-6289-40b1-ac0b-5e155420ee38-workload-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-2gfr2\" (UID: \"35d8c3ab-6289-40b1-ac0b-5e155420ee38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2gfr2"
Oct 30 13:02:18.370148 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:18.370139    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35d8c3ab-6289-40b1-ac0b-5e155420ee38-workload-certs\") pod \"maas-default-gateway-openshift-default-64d4cc998c-2gfr2\" (UID: \"35d8c3ab-6289-40b1-ac0b-5e155420ee38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2gfr2"
Oct 30 13:02:18.370318 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:18.370167    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35d8c3ab-6289-40b1-ac0b-5e155420ee38-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-64d4cc998c-2gfr2\" (UID: \"35d8c3ab-6289-40b1-ac0b-5e155420ee38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2gfr2"
Oct 30 13:02:18.371887 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:18.371865    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35d8c3ab-6289-40b1-ac0b-5e155420ee38-istio-envoy\") pod \"maas-default-gateway-openshift-default-64d4cc998c-2gfr2\" (UID: \"35d8c3ab-6289-40b1-ac0b-5e155420ee38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2gfr2"
Oct 30 13:02:18.371976 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:18.371956    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35d8c3ab-6289-40b1-ac0b-5e155420ee38-istio-podinfo\") pod \"maas-default-gateway-openshift-default-64d4cc998c-2gfr2\" (UID: \"35d8c3ab-6289-40b1-ac0b-5e155420ee38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2gfr2"
Oct 30 13:02:18.409080 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:18.409046    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74zcw\" (UniqueName: \"kubernetes.io/projected/35d8c3ab-6289-40b1-ac0b-5e155420ee38-kube-api-access-74zcw\") pod \"maas-default-gateway-openshift-default-64d4cc998c-2gfr2\" (UID: \"35d8c3ab-6289-40b1-ac0b-5e155420ee38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2gfr2"
Oct 30 13:02:18.409732 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:18.409711    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35d8c3ab-6289-40b1-ac0b-5e155420ee38-istio-token\") pod \"maas-default-gateway-openshift-default-64d4cc998c-2gfr2\" (UID: \"35d8c3ab-6289-40b1-ac0b-5e155420ee38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2gfr2"
Oct 30 13:02:18.499210 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:18.499125    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2gfr2"
Oct 30 13:02:18.862359 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:18.862331    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2gfr2"]
Oct 30 13:02:18.862842 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:02:18.862764    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35d8c3ab_6289_40b1_ac0b_5e155420ee38.slice/crio-88f00d746b01509005ba554d2d6da17024d39f31399fc84336be35638d5abc0d WatchSource:0}: Error finding container 88f00d746b01509005ba554d2d6da17024d39f31399fc84336be35638d5abc0d: Status 404 returned error can't find the container with id 88f00d746b01509005ba554d2d6da17024d39f31399fc84336be35638d5abc0d
Oct 30 13:02:18.864894 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:18.864861    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:02:18.865013 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:18.864933    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:02:18.865013 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:18.864973    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:02:19.874250 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:19.874211    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2gfr2" event={"ID":"35d8c3ab-6289-40b1-ac0b-5e155420ee38","Type":"ContainerStarted","Data":"4af097e9d0caa680e7a29135fee164ac03fbaffa49b66e89d92706576c5be268"}
Oct 30 13:02:19.874676 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:19.874292    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2gfr2" event={"ID":"35d8c3ab-6289-40b1-ac0b-5e155420ee38","Type":"ContainerStarted","Data":"88f00d746b01509005ba554d2d6da17024d39f31399fc84336be35638d5abc0d"}
Oct 30 13:02:19.964447 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:19.964394    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2gfr2" podStartSLOduration=1.964380603 podStartE2EDuration="1.964380603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:19.950069878 +0000 UTC m=+639.116434527" watchObservedRunningTime="2026-10-30 13:02:19.964380603 +0000 UTC m=+639.130745250"
Oct 30 13:02:20.501624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:20.501338    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2gfr2"
Oct 30 13:02:20.501624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:20.501460    4364 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-2gfr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 30 13:02:20.501624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:20.501506    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2gfr2" podUID="35d8c3ab-6289-40b1-ac0b-5e155420ee38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 30 13:02:21.503936 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:21.503898    4364 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-2gfr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 30 13:02:21.504338 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:21.503960    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2gfr2" podUID="35d8c3ab-6289-40b1-ac0b-5e155420ee38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 30 13:02:22.503625 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:22.502932    4364 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-2gfr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 30 13:02:22.503625 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:22.502993    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2gfr2" podUID="35d8c3ab-6289-40b1-ac0b-5e155420ee38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 30 13:02:23.499920 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:23.499884    4364 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-2gfr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 30 13:02:23.500500 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:23.500460    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2gfr2" podUID="35d8c3ab-6289-40b1-ac0b-5e155420ee38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 30 13:02:24.503745 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:24.503704    4364 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-2gfr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 30 13:02:24.504236 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:24.503769    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2gfr2" podUID="35d8c3ab-6289-40b1-ac0b-5e155420ee38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 30 13:02:25.160366 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:25.160329    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-697b76b867-ndcgg"]
Oct 30 13:02:25.163483 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:25.163468    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-697b76b867-ndcgg"
Oct 30 13:02:25.186461 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:25.186437    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Oct 30 13:02:25.200749 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:25.200724    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-bkjx4\""
Oct 30 13:02:25.200883 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:25.200750    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Oct 30 13:02:25.202966 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:25.202942    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-697b76b867-ndcgg"]
Oct 30 13:02:25.254057 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:25.254027    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b446\" (UniqueName: \"kubernetes.io/projected/95546275-c16e-4ac9-8842-2ef292e55fb1-kube-api-access-4b446\") pod \"ai-gateway-controller-697b76b867-ndcgg\" (UID: \"95546275-c16e-4ac9-8842-2ef292e55fb1\") " pod="opendatahub/ai-gateway-controller-697b76b867-ndcgg"
Oct 30 13:02:25.254252 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:25.254175    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95546275-c16e-4ac9-8842-2ef292e55fb1-metrics-certs\") pod \"ai-gateway-controller-697b76b867-ndcgg\" (UID: \"95546275-c16e-4ac9-8842-2ef292e55fb1\") " pod="opendatahub/ai-gateway-controller-697b76b867-ndcgg"
Oct 30 13:02:25.354670 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:25.354643    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b446\" (UniqueName: \"kubernetes.io/projected/95546275-c16e-4ac9-8842-2ef292e55fb1-kube-api-access-4b446\") pod \"ai-gateway-controller-697b76b867-ndcgg\" (UID: \"95546275-c16e-4ac9-8842-2ef292e55fb1\") " pod="opendatahub/ai-gateway-controller-697b76b867-ndcgg"
Oct 30 13:02:25.354832 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:25.354719    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95546275-c16e-4ac9-8842-2ef292e55fb1-metrics-certs\") pod \"ai-gateway-controller-697b76b867-ndcgg\" (UID: \"95546275-c16e-4ac9-8842-2ef292e55fb1\") " pod="opendatahub/ai-gateway-controller-697b76b867-ndcgg"
Oct 30 13:02:25.357120 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:25.357098    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95546275-c16e-4ac9-8842-2ef292e55fb1-metrics-certs\") pod \"ai-gateway-controller-697b76b867-ndcgg\" (UID: \"95546275-c16e-4ac9-8842-2ef292e55fb1\") " pod="opendatahub/ai-gateway-controller-697b76b867-ndcgg"
Oct 30 13:02:25.410486 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:25.410421    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b446\" (UniqueName: \"kubernetes.io/projected/95546275-c16e-4ac9-8842-2ef292e55fb1-kube-api-access-4b446\") pod \"ai-gateway-controller-697b76b867-ndcgg\" (UID: \"95546275-c16e-4ac9-8842-2ef292e55fb1\") " pod="opendatahub/ai-gateway-controller-697b76b867-ndcgg"
Oct 30 13:02:25.476972 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:25.476947    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-697b76b867-ndcgg"
Oct 30 13:02:25.500226 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:25.500193    4364 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-2gfr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 30 13:02:25.500335 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:25.500235    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2gfr2" podUID="35d8c3ab-6289-40b1-ac0b-5e155420ee38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 30 13:02:25.783579 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:25.783548    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-697b76b867-ndcgg"]
Oct 30 13:02:25.784007 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:02:25.783586    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95546275_c16e_4ac9_8842_2ef292e55fb1.slice/crio-fdd4dfc500c8e03588995eac69c7e2f8d03dae39f724f691348e6f55b94c021e WatchSource:0}: Error finding container fdd4dfc500c8e03588995eac69c7e2f8d03dae39f724f691348e6f55b94c021e: Status 404 returned error can't find the container with id fdd4dfc500c8e03588995eac69c7e2f8d03dae39f724f691348e6f55b94c021e
Oct 30 13:02:25.784936 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:25.784919    4364 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 13:02:25.925119 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:25.925092    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-697b76b867-ndcgg" event={"ID":"95546275-c16e-4ac9-8842-2ef292e55fb1","Type":"ContainerStarted","Data":"fdd4dfc500c8e03588995eac69c7e2f8d03dae39f724f691348e6f55b94c021e"}
Oct 30 13:02:27.403427 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:27.403394    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2gfr2"
Oct 30 13:02:27.404616 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:27.404573    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2gfr2"
Oct 30 13:02:27.405404 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:27.405358    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2gfr2"
Oct 30 13:02:28.953423 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:28.953390    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-697b76b867-ndcgg" event={"ID":"95546275-c16e-4ac9-8842-2ef292e55fb1","Type":"ContainerStarted","Data":"8454d7eb44985222df446219109f05f8bbc13555473daf44dde690950b294295"}
Oct 30 13:02:29.031475 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:29.031415    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-697b76b867-ndcgg" podStartSLOduration=1.7189917700000001 podStartE2EDuration="4.03140115s" totalImagesPullingTime="2.31240938s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:29.030101509 +0000 UTC m=+648.196466157" watchObservedRunningTime="2026-10-30 13:02:29.03140115 +0000 UTC m=+648.197765797"
Oct 30 13:02:29.799650 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:29.799622    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-mqpfw"]
Oct 30 13:02:29.806376 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:29.806350    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-mqpfw"
Oct 30 13:02:29.829873 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:29.829844    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-28598\""
Oct 30 13:02:29.859789 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:29.859765    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-mqpfw"]
Oct 30 13:02:29.892567 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:29.892539    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Oct 30 13:02:29.900077 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:29.900042    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tld9x\" (UniqueName: \"kubernetes.io/projected/310804ff-17ef-4a28-959c-47995bde35ca-kube-api-access-tld9x\") pod \"payload-processing-5f78687d5b-mqpfw\" (UID: \"310804ff-17ef-4a28-959c-47995bde35ca\") " pod="openshift-ingress/payload-processing-5f78687d5b-mqpfw"
Oct 30 13:02:29.903326 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:29.903301    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/310804ff-17ef-4a28-959c-47995bde35ca-plugins-config-volume\") pod \"payload-processing-5f78687d5b-mqpfw\" (UID: \"310804ff-17ef-4a28-959c-47995bde35ca\") " pod="openshift-ingress/payload-processing-5f78687d5b-mqpfw"
Oct 30 13:02:29.967772 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:29.967740    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-697b76b867-ndcgg"
Oct 30 13:02:30.004097 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:30.004064    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tld9x\" (UniqueName: \"kubernetes.io/projected/310804ff-17ef-4a28-959c-47995bde35ca-kube-api-access-tld9x\") pod \"payload-processing-5f78687d5b-mqpfw\" (UID: \"310804ff-17ef-4a28-959c-47995bde35ca\") " pod="openshift-ingress/payload-processing-5f78687d5b-mqpfw"
Oct 30 13:02:30.004277 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:30.004129    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/310804ff-17ef-4a28-959c-47995bde35ca-plugins-config-volume\") pod \"payload-processing-5f78687d5b-mqpfw\" (UID: \"310804ff-17ef-4a28-959c-47995bde35ca\") " pod="openshift-ingress/payload-processing-5f78687d5b-mqpfw"
Oct 30 13:02:30.004968 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:30.004945    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/310804ff-17ef-4a28-959c-47995bde35ca-plugins-config-volume\") pod \"payload-processing-5f78687d5b-mqpfw\" (UID: \"310804ff-17ef-4a28-959c-47995bde35ca\") " pod="openshift-ingress/payload-processing-5f78687d5b-mqpfw"
Oct 30 13:02:30.034131 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:30.034095    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tld9x\" (UniqueName: \"kubernetes.io/projected/310804ff-17ef-4a28-959c-47995bde35ca-kube-api-access-tld9x\") pod \"payload-processing-5f78687d5b-mqpfw\" (UID: \"310804ff-17ef-4a28-959c-47995bde35ca\") " pod="openshift-ingress/payload-processing-5f78687d5b-mqpfw"
Oct 30 13:02:30.120537 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:30.120512    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-mqpfw"
Oct 30 13:02:30.428787 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:30.428761    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-mqpfw"]
Oct 30 13:02:30.430868 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:02:30.430838    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod310804ff_17ef_4a28_959c_47995bde35ca.slice/crio-57df82fc8322484a686dba4211cff06b7533b3fc6e52021c205138afbf8a000f WatchSource:0}: Error finding container 57df82fc8322484a686dba4211cff06b7533b3fc6e52021c205138afbf8a000f: Status 404 returned error can't find the container with id 57df82fc8322484a686dba4211cff06b7533b3fc6e52021c205138afbf8a000f
Oct 30 13:02:30.976693 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:30.976620    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-mqpfw" event={"ID":"310804ff-17ef-4a28-959c-47995bde35ca","Type":"ContainerStarted","Data":"57df82fc8322484a686dba4211cff06b7533b3fc6e52021c205138afbf8a000f"}
Oct 30 13:02:32.988629 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:32.988586    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-mqpfw" event={"ID":"310804ff-17ef-4a28-959c-47995bde35ca","Type":"ContainerStarted","Data":"62ba8c73935a0e9ab11596a4677b2eab382832e4614af16ffdc503801a5c951b"}
Oct 30 13:02:32.988988 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:32.988656    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5f78687d5b-mqpfw"
Oct 30 13:02:33.036213 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:33.036173    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5f78687d5b-mqpfw" podStartSLOduration=1.991229586 podStartE2EDuration="4.036160686s" totalImagesPullingTime="2.0449311s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:33.034170929 +0000 UTC m=+652.200535574" watchObservedRunningTime="2026-10-30 13:02:33.036160686 +0000 UTC m=+652.202525334"
Oct 30 13:02:40.979396 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:40.979368    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-697b76b867-ndcgg"
Oct 30 13:02:41.504040 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:02:41.504004    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:02:43.996805 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:43.996777    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5f78687d5b-mqpfw"
Oct 30 13:02:55.476418 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:55.476337    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-697b76b867-ndcgg"]
Oct 30 13:02:55.476942 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:55.476572    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-697b76b867-ndcgg" podUID="95546275-c16e-4ac9-8842-2ef292e55fb1" containerName="manager" containerID="cri-o://8454d7eb44985222df446219109f05f8bbc13555473daf44dde690950b294295" gracePeriod=30
Oct 30 13:02:55.722127 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:55.722105    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-697b76b867-ndcgg"
Oct 30 13:02:55.831412 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:55.831318    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95546275-c16e-4ac9-8842-2ef292e55fb1-kube-api-access-4b446\" (UniqueName: \"kubernetes.io/projected/95546275-c16e-4ac9-8842-2ef292e55fb1-kube-api-access-4b446\") pod \"95546275-c16e-4ac9-8842-2ef292e55fb1\" (UID: \"95546275-c16e-4ac9-8842-2ef292e55fb1\") "
Oct 30 13:02:55.831552 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:55.831462    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95546275-c16e-4ac9-8842-2ef292e55fb1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/95546275-c16e-4ac9-8842-2ef292e55fb1-metrics-certs\") pod \"95546275-c16e-4ac9-8842-2ef292e55fb1\" (UID: \"95546275-c16e-4ac9-8842-2ef292e55fb1\") "
Oct 30 13:02:55.834059 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:55.834033    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95546275-c16e-4ac9-8842-2ef292e55fb1-metrics-certs" pod "95546275-c16e-4ac9-8842-2ef292e55fb1" (UID: "95546275-c16e-4ac9-8842-2ef292e55fb1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:02:55.834152 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:55.834131    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95546275-c16e-4ac9-8842-2ef292e55fb1-kube-api-access-4b446" pod "95546275-c16e-4ac9-8842-2ef292e55fb1" (UID: "95546275-c16e-4ac9-8842-2ef292e55fb1"). InnerVolumeSpecName "kube-api-access-4b446". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:02:55.932539 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:55.932512    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4b446\" (UniqueName: \"kubernetes.io/projected/95546275-c16e-4ac9-8842-2ef292e55fb1-kube-api-access-4b446\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:02:55.932539 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:55.932539    4364 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95546275-c16e-4ac9-8842-2ef292e55fb1-metrics-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:02:56.099923 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:56.099858    4364 generic.go:356] "Generic (PLEG): container finished" podID="95546275-c16e-4ac9-8842-2ef292e55fb1" containerID="8454d7eb44985222df446219109f05f8bbc13555473daf44dde690950b294295" exitCode=0
Oct 30 13:02:56.099923 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:56.099894    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-697b76b867-ndcgg"
Oct 30 13:02:56.100058 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:56.099930    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-697b76b867-ndcgg" event={"ID":"95546275-c16e-4ac9-8842-2ef292e55fb1","Type":"ContainerDied","Data":"8454d7eb44985222df446219109f05f8bbc13555473daf44dde690950b294295"}
Oct 30 13:02:56.100058 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:56.099983    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-697b76b867-ndcgg" event={"ID":"95546275-c16e-4ac9-8842-2ef292e55fb1","Type":"ContainerDied","Data":"fdd4dfc500c8e03588995eac69c7e2f8d03dae39f724f691348e6f55b94c021e"}
Oct 30 13:02:56.100058 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:56.100004    4364 scope.go:122] "RemoveContainer" containerID="8454d7eb44985222df446219109f05f8bbc13555473daf44dde690950b294295"
Oct 30 13:02:56.109367 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:56.109350    4364 scope.go:122] "RemoveContainer" containerID="8454d7eb44985222df446219109f05f8bbc13555473daf44dde690950b294295"
Oct 30 13:02:56.109647 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:02:56.109622    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8454d7eb44985222df446219109f05f8bbc13555473daf44dde690950b294295\": container with ID starting with 8454d7eb44985222df446219109f05f8bbc13555473daf44dde690950b294295 not found: ID does not exist" containerID="8454d7eb44985222df446219109f05f8bbc13555473daf44dde690950b294295"
Oct 30 13:02:56.109734 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:56.109656    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8454d7eb44985222df446219109f05f8bbc13555473daf44dde690950b294295"} err="failed to get container status \"8454d7eb44985222df446219109f05f8bbc13555473daf44dde690950b294295\": rpc error: code = NotFound desc = could not find container \"8454d7eb44985222df446219109f05f8bbc13555473daf44dde690950b294295\": container with ID starting with 8454d7eb44985222df446219109f05f8bbc13555473daf44dde690950b294295 not found: ID does not exist"
Oct 30 13:02:56.158788 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:56.158763    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-697b76b867-ndcgg"]
Oct 30 13:02:56.172801 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:56.172782    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-697b76b867-ndcgg"]
Oct 30 13:02:56.969092 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:02:56.969060    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95546275-c16e-4ac9-8842-2ef292e55fb1" path="/var/lib/kubelet/pods/95546275-c16e-4ac9-8842-2ef292e55fb1/volumes"
Oct 30 13:03:11.503121 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:03:11.503083    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:03:21.929866 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:21.929830    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc"]
Oct 30 13:03:21.930377 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:21.930241    4364 cpu_manager.go:414] "containerMap: removing container" podUID="95546275-c16e-4ac9-8842-2ef292e55fb1" containerName="manager"
Oct 30 13:03:21.930377 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:21.930254    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95546275-c16e-4ac9-8842-2ef292e55fb1" containerName="manager"
Oct 30 13:03:21.930377 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:21.930321    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="95546275-c16e-4ac9-8842-2ef292e55fb1" containerName="manager"
Oct 30 13:03:21.936450 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:21.936431    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc"
Oct 30 13:03:21.959282 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:21.959252    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc"]
Oct 30 13:03:21.970443 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:21.970419    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-ee7d65-openshift-default-dockercfg-5cfmj\""
Oct 30 13:03:22.036733 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.036708    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4710cc0d-1402-489e-826b-5c44a8b6f9f0-istio-token\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc\" (UID: \"4710cc0d-1402-489e-826b-5c44a8b6f9f0\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc"
Oct 30 13:03:22.036866 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.036757    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4710cc0d-1402-489e-826b-5c44a8b6f9f0-istiod-ca-cert\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc\" (UID: \"4710cc0d-1402-489e-826b-5c44a8b6f9f0\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc"
Oct 30 13:03:22.036866 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.036779    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4710cc0d-1402-489e-826b-5c44a8b6f9f0-istio-podinfo\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc\" (UID: \"4710cc0d-1402-489e-826b-5c44a8b6f9f0\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc"
Oct 30 13:03:22.036991 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.036941    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4710cc0d-1402-489e-826b-5c44a8b6f9f0-workload-socket\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc\" (UID: \"4710cc0d-1402-489e-826b-5c44a8b6f9f0\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc"
Oct 30 13:03:22.036991 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.036971    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4710cc0d-1402-489e-826b-5c44a8b6f9f0-istio-envoy\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc\" (UID: \"4710cc0d-1402-489e-826b-5c44a8b6f9f0\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc"
Oct 30 13:03:22.037104 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.037003    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4710cc0d-1402-489e-826b-5c44a8b6f9f0-workload-certs\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc\" (UID: \"4710cc0d-1402-489e-826b-5c44a8b6f9f0\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc"
Oct 30 13:03:22.037171 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.037135    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4710cc0d-1402-489e-826b-5c44a8b6f9f0-credential-socket\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc\" (UID: \"4710cc0d-1402-489e-826b-5c44a8b6f9f0\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc"
Oct 30 13:03:22.037266 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.037251    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4710cc0d-1402-489e-826b-5c44a8b6f9f0-istio-data\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc\" (UID: \"4710cc0d-1402-489e-826b-5c44a8b6f9f0\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc"
Oct 30 13:03:22.037315 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.037283    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr9l4\" (UniqueName: \"kubernetes.io/projected/4710cc0d-1402-489e-826b-5c44a8b6f9f0-kube-api-access-wr9l4\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc\" (UID: \"4710cc0d-1402-489e-826b-5c44a8b6f9f0\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc"
Oct 30 13:03:22.138003 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.137973    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4710cc0d-1402-489e-826b-5c44a8b6f9f0-istio-data\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc\" (UID: \"4710cc0d-1402-489e-826b-5c44a8b6f9f0\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc"
Oct 30 13:03:22.138158 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.138013    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr9l4\" (UniqueName: \"kubernetes.io/projected/4710cc0d-1402-489e-826b-5c44a8b6f9f0-kube-api-access-wr9l4\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc\" (UID: \"4710cc0d-1402-489e-826b-5c44a8b6f9f0\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc"
Oct 30 13:03:22.138158 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.138036    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4710cc0d-1402-489e-826b-5c44a8b6f9f0-istio-token\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc\" (UID: \"4710cc0d-1402-489e-826b-5c44a8b6f9f0\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc"
Oct 30 13:03:22.138158 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.138077    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4710cc0d-1402-489e-826b-5c44a8b6f9f0-istiod-ca-cert\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc\" (UID: \"4710cc0d-1402-489e-826b-5c44a8b6f9f0\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc"
Oct 30 13:03:22.138158 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.138102    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4710cc0d-1402-489e-826b-5c44a8b6f9f0-istio-podinfo\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc\" (UID: \"4710cc0d-1402-489e-826b-5c44a8b6f9f0\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc"
Oct 30 13:03:22.138158 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.138136    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4710cc0d-1402-489e-826b-5c44a8b6f9f0-workload-socket\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc\" (UID: \"4710cc0d-1402-489e-826b-5c44a8b6f9f0\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc"
Oct 30 13:03:22.138441 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.138317    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4710cc0d-1402-489e-826b-5c44a8b6f9f0-istio-envoy\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc\" (UID: \"4710cc0d-1402-489e-826b-5c44a8b6f9f0\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc"
Oct 30 13:03:22.138441 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.138365    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4710cc0d-1402-489e-826b-5c44a8b6f9f0-workload-certs\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc\" (UID: \"4710cc0d-1402-489e-826b-5c44a8b6f9f0\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc"
Oct 30 13:03:22.138441 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.138408    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4710cc0d-1402-489e-826b-5c44a8b6f9f0-credential-socket\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc\" (UID: \"4710cc0d-1402-489e-826b-5c44a8b6f9f0\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc"
Oct 30 13:03:22.138583 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.138476    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4710cc0d-1402-489e-826b-5c44a8b6f9f0-istio-data\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc\" (UID: \"4710cc0d-1402-489e-826b-5c44a8b6f9f0\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc"
Oct 30 13:03:22.138583 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.138514    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4710cc0d-1402-489e-826b-5c44a8b6f9f0-workload-socket\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc\" (UID: \"4710cc0d-1402-489e-826b-5c44a8b6f9f0\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc"
Oct 30 13:03:22.138723 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.138664    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4710cc0d-1402-489e-826b-5c44a8b6f9f0-workload-certs\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc\" (UID: \"4710cc0d-1402-489e-826b-5c44a8b6f9f0\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc"
Oct 30 13:03:22.138833 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.138812    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4710cc0d-1402-489e-826b-5c44a8b6f9f0-credential-socket\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc\" (UID: \"4710cc0d-1402-489e-826b-5c44a8b6f9f0\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc"
Oct 30 13:03:22.138944 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.138918    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4710cc0d-1402-489e-826b-5c44a8b6f9f0-istiod-ca-cert\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc\" (UID: \"4710cc0d-1402-489e-826b-5c44a8b6f9f0\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc"
Oct 30 13:03:22.140411 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.140388    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4710cc0d-1402-489e-826b-5c44a8b6f9f0-istio-envoy\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc\" (UID: \"4710cc0d-1402-489e-826b-5c44a8b6f9f0\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc"
Oct 30 13:03:22.140642 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.140627    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4710cc0d-1402-489e-826b-5c44a8b6f9f0-istio-podinfo\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc\" (UID: \"4710cc0d-1402-489e-826b-5c44a8b6f9f0\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc"
Oct 30 13:03:22.163140 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.163116    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4710cc0d-1402-489e-826b-5c44a8b6f9f0-istio-token\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc\" (UID: \"4710cc0d-1402-489e-826b-5c44a8b6f9f0\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc"
Oct 30 13:03:22.163824 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.163808    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr9l4\" (UniqueName: \"kubernetes.io/projected/4710cc0d-1402-489e-826b-5c44a8b6f9f0-kube-api-access-wr9l4\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc\" (UID: \"4710cc0d-1402-489e-826b-5c44a8b6f9f0\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc"
Oct 30 13:03:22.249621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.249547    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc"
Oct 30 13:03:22.281804 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.281777    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7"]
Oct 30 13:03:22.284408 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.284391    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7"
Oct 30 13:03:22.324634 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.324609    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-cb58b8-openshift-default-dockercfg-dftld\""
Oct 30 13:03:22.327660 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.327630    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7"]
Oct 30 13:03:22.441316 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.441287    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/134eb93c-b6ee-43e9-8977-950cb0188887-istio-data\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7\" (UID: \"134eb93c-b6ee-43e9-8977-950cb0188887\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7"
Oct 30 13:03:22.441490 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.441328    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/134eb93c-b6ee-43e9-8977-950cb0188887-istio-envoy\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7\" (UID: \"134eb93c-b6ee-43e9-8977-950cb0188887\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7"
Oct 30 13:03:22.441490 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.441359    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/134eb93c-b6ee-43e9-8977-950cb0188887-istio-token\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7\" (UID: \"134eb93c-b6ee-43e9-8977-950cb0188887\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7"
Oct 30 13:03:22.441574 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.441522    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/134eb93c-b6ee-43e9-8977-950cb0188887-credential-socket\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7\" (UID: \"134eb93c-b6ee-43e9-8977-950cb0188887\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7"
Oct 30 13:03:22.441629 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.441578    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/134eb93c-b6ee-43e9-8977-950cb0188887-workload-certs\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7\" (UID: \"134eb93c-b6ee-43e9-8977-950cb0188887\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7"
Oct 30 13:03:22.441688 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.441668    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txpk9\" (UniqueName: \"kubernetes.io/projected/134eb93c-b6ee-43e9-8977-950cb0188887-kube-api-access-txpk9\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7\" (UID: \"134eb93c-b6ee-43e9-8977-950cb0188887\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7"
Oct 30 13:03:22.441735 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.441720    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/134eb93c-b6ee-43e9-8977-950cb0188887-istiod-ca-cert\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7\" (UID: \"134eb93c-b6ee-43e9-8977-950cb0188887\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7"
Oct 30 13:03:22.441796 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.441779    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/134eb93c-b6ee-43e9-8977-950cb0188887-workload-socket\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7\" (UID: \"134eb93c-b6ee-43e9-8977-950cb0188887\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7"
Oct 30 13:03:22.441913 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.441888    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/134eb93c-b6ee-43e9-8977-950cb0188887-istio-podinfo\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7\" (UID: \"134eb93c-b6ee-43e9-8977-950cb0188887\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7"
Oct 30 13:03:22.542558 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.542488    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/134eb93c-b6ee-43e9-8977-950cb0188887-istio-envoy\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7\" (UID: \"134eb93c-b6ee-43e9-8977-950cb0188887\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7"
Oct 30 13:03:22.542558 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.542526    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/134eb93c-b6ee-43e9-8977-950cb0188887-istio-token\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7\" (UID: \"134eb93c-b6ee-43e9-8977-950cb0188887\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7"
Oct 30 13:03:22.542558 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.542546    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/134eb93c-b6ee-43e9-8977-950cb0188887-credential-socket\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7\" (UID: \"134eb93c-b6ee-43e9-8977-950cb0188887\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7"
Oct 30 13:03:22.542841 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.542577    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/134eb93c-b6ee-43e9-8977-950cb0188887-workload-certs\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7\" (UID: \"134eb93c-b6ee-43e9-8977-950cb0188887\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7"
Oct 30 13:03:22.542841 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.542643    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txpk9\" (UniqueName: \"kubernetes.io/projected/134eb93c-b6ee-43e9-8977-950cb0188887-kube-api-access-txpk9\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7\" (UID: \"134eb93c-b6ee-43e9-8977-950cb0188887\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7"
Oct 30 13:03:22.542841 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.542683    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/134eb93c-b6ee-43e9-8977-950cb0188887-istiod-ca-cert\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7\" (UID: \"134eb93c-b6ee-43e9-8977-950cb0188887\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7"
Oct 30 13:03:22.542841 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.542714    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/134eb93c-b6ee-43e9-8977-950cb0188887-workload-socket\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7\" (UID: \"134eb93c-b6ee-43e9-8977-950cb0188887\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7"
Oct 30 13:03:22.542841 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.542774    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/134eb93c-b6ee-43e9-8977-950cb0188887-istio-podinfo\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7\" (UID: \"134eb93c-b6ee-43e9-8977-950cb0188887\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7"
Oct 30 13:03:22.542841 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.542826    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/134eb93c-b6ee-43e9-8977-950cb0188887-istio-data\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7\" (UID: \"134eb93c-b6ee-43e9-8977-950cb0188887\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7"
Oct 30 13:03:22.543139 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.542998    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/134eb93c-b6ee-43e9-8977-950cb0188887-credential-socket\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7\" (UID: \"134eb93c-b6ee-43e9-8977-950cb0188887\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7"
Oct 30 13:03:22.543139 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.543066    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/134eb93c-b6ee-43e9-8977-950cb0188887-workload-certs\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7\" (UID: \"134eb93c-b6ee-43e9-8977-950cb0188887\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7"
Oct 30 13:03:22.543139 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.543105    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/134eb93c-b6ee-43e9-8977-950cb0188887-workload-socket\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7\" (UID: \"134eb93c-b6ee-43e9-8977-950cb0188887\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7"
Oct 30 13:03:22.543241 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.543146    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/134eb93c-b6ee-43e9-8977-950cb0188887-istio-data\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7\" (UID: \"134eb93c-b6ee-43e9-8977-950cb0188887\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7"
Oct 30 13:03:22.543421 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.543405    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/134eb93c-b6ee-43e9-8977-950cb0188887-istiod-ca-cert\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7\" (UID: \"134eb93c-b6ee-43e9-8977-950cb0188887\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7"
Oct 30 13:03:22.544934 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.544911    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/134eb93c-b6ee-43e9-8977-950cb0188887-istio-envoy\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7\" (UID: \"134eb93c-b6ee-43e9-8977-950cb0188887\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7"
Oct 30 13:03:22.545496 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.545475    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/134eb93c-b6ee-43e9-8977-950cb0188887-istio-podinfo\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7\" (UID: \"134eb93c-b6ee-43e9-8977-950cb0188887\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7"
Oct 30 13:03:22.570209 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.570186    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc"]
Oct 30 13:03:22.570510 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:03:22.570486    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4710cc0d_1402_489e_826b_5c44a8b6f9f0.slice/crio-9796fac3320cfec9612196021d5f43926ffb793eab0ecb1a4fa64deb618882f6 WatchSource:0}: Error finding container 9796fac3320cfec9612196021d5f43926ffb793eab0ecb1a4fa64deb618882f6: Status 404 returned error can't find the container with id 9796fac3320cfec9612196021d5f43926ffb793eab0ecb1a4fa64deb618882f6
Oct 30 13:03:22.573086 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.573043    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:03:22.573178 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.573136    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:03:22.573219 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.573182    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:03:22.574454 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.574435    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txpk9\" (UniqueName: \"kubernetes.io/projected/134eb93c-b6ee-43e9-8977-950cb0188887-kube-api-access-txpk9\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7\" (UID: \"134eb93c-b6ee-43e9-8977-950cb0188887\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7"
Oct 30 13:03:22.574532 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.574468    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/134eb93c-b6ee-43e9-8977-950cb0188887-istio-token\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7\" (UID: \"134eb93c-b6ee-43e9-8977-950cb0188887\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7"
Oct 30 13:03:22.589496 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.589474    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g"]
Oct 30 13:03:22.592813 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.592795    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g"
Oct 30 13:03:22.600270 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.600246    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7"
Oct 30 13:03:22.612036 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.611996    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-e34fd5-openshift-default-dockercfg-mz9s9\""
Oct 30 13:03:22.619472 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.619023    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g"]
Oct 30 13:03:22.745408 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.745319    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/370ee2a9-2643-4e24-81e4-ab56f3caefb3-istiod-ca-cert\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g\" (UID: \"370ee2a9-2643-4e24-81e4-ab56f3caefb3\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g"
Oct 30 13:03:22.745408 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.745370    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/370ee2a9-2643-4e24-81e4-ab56f3caefb3-istio-token\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g\" (UID: \"370ee2a9-2643-4e24-81e4-ab56f3caefb3\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g"
Oct 30 13:03:22.745754 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.745437    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/370ee2a9-2643-4e24-81e4-ab56f3caefb3-workload-socket\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g\" (UID: \"370ee2a9-2643-4e24-81e4-ab56f3caefb3\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g"
Oct 30 13:03:22.745754 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.745477    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/370ee2a9-2643-4e24-81e4-ab56f3caefb3-istio-envoy\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g\" (UID: \"370ee2a9-2643-4e24-81e4-ab56f3caefb3\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g"
Oct 30 13:03:22.745754 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.745548    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/370ee2a9-2643-4e24-81e4-ab56f3caefb3-credential-socket\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g\" (UID: \"370ee2a9-2643-4e24-81e4-ab56f3caefb3\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g"
Oct 30 13:03:22.745754 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.745588    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/370ee2a9-2643-4e24-81e4-ab56f3caefb3-istio-data\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g\" (UID: \"370ee2a9-2643-4e24-81e4-ab56f3caefb3\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g"
Oct 30 13:03:22.745754 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.745639    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/370ee2a9-2643-4e24-81e4-ab56f3caefb3-workload-certs\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g\" (UID: \"370ee2a9-2643-4e24-81e4-ab56f3caefb3\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g"
Oct 30 13:03:22.745754 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.745689    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/370ee2a9-2643-4e24-81e4-ab56f3caefb3-istio-podinfo\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g\" (UID: \"370ee2a9-2643-4e24-81e4-ab56f3caefb3\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g"
Oct 30 13:03:22.745754 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.745740    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26z2l\" (UniqueName: \"kubernetes.io/projected/370ee2a9-2643-4e24-81e4-ab56f3caefb3-kube-api-access-26z2l\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g\" (UID: \"370ee2a9-2643-4e24-81e4-ab56f3caefb3\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g"
Oct 30 13:03:22.846405 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.846333    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/370ee2a9-2643-4e24-81e4-ab56f3caefb3-workload-socket\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g\" (UID: \"370ee2a9-2643-4e24-81e4-ab56f3caefb3\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g"
Oct 30 13:03:22.846405 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.846380    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/370ee2a9-2643-4e24-81e4-ab56f3caefb3-istio-envoy\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g\" (UID: \"370ee2a9-2643-4e24-81e4-ab56f3caefb3\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g"
Oct 30 13:03:22.846726 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.846414    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/370ee2a9-2643-4e24-81e4-ab56f3caefb3-credential-socket\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g\" (UID: \"370ee2a9-2643-4e24-81e4-ab56f3caefb3\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g"
Oct 30 13:03:22.846726 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.846551    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/370ee2a9-2643-4e24-81e4-ab56f3caefb3-istio-data\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g\" (UID: \"370ee2a9-2643-4e24-81e4-ab56f3caefb3\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g"
Oct 30 13:03:22.846726 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.846640    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/370ee2a9-2643-4e24-81e4-ab56f3caefb3-workload-certs\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g\" (UID: \"370ee2a9-2643-4e24-81e4-ab56f3caefb3\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g"
Oct 30 13:03:22.846726 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.846695    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/370ee2a9-2643-4e24-81e4-ab56f3caefb3-istio-podinfo\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g\" (UID: \"370ee2a9-2643-4e24-81e4-ab56f3caefb3\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g"
Oct 30 13:03:22.846938 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.846757    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26z2l\" (UniqueName: \"kubernetes.io/projected/370ee2a9-2643-4e24-81e4-ab56f3caefb3-kube-api-access-26z2l\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g\" (UID: \"370ee2a9-2643-4e24-81e4-ab56f3caefb3\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g"
Oct 30 13:03:22.846938 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.846822    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/370ee2a9-2643-4e24-81e4-ab56f3caefb3-istiod-ca-cert\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g\" (UID: \"370ee2a9-2643-4e24-81e4-ab56f3caefb3\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g"
Oct 30 13:03:22.846938 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.846850    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/370ee2a9-2643-4e24-81e4-ab56f3caefb3-istio-token\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g\" (UID: \"370ee2a9-2643-4e24-81e4-ab56f3caefb3\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g"
Oct 30 13:03:22.849397 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.847743    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/370ee2a9-2643-4e24-81e4-ab56f3caefb3-istio-data\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g\" (UID: \"370ee2a9-2643-4e24-81e4-ab56f3caefb3\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g"
Oct 30 13:03:22.849397 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.847789    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/370ee2a9-2643-4e24-81e4-ab56f3caefb3-credential-socket\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g\" (UID: \"370ee2a9-2643-4e24-81e4-ab56f3caefb3\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g"
Oct 30 13:03:22.849397 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.848095    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/370ee2a9-2643-4e24-81e4-ab56f3caefb3-workload-certs\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g\" (UID: \"370ee2a9-2643-4e24-81e4-ab56f3caefb3\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g"
Oct 30 13:03:22.849397 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.848734    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/370ee2a9-2643-4e24-81e4-ab56f3caefb3-istiod-ca-cert\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g\" (UID: \"370ee2a9-2643-4e24-81e4-ab56f3caefb3\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g"
Oct 30 13:03:22.849686 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.849439    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/370ee2a9-2643-4e24-81e4-ab56f3caefb3-istio-envoy\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g\" (UID: \"370ee2a9-2643-4e24-81e4-ab56f3caefb3\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g"
Oct 30 13:03:22.852718 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.851204    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/370ee2a9-2643-4e24-81e4-ab56f3caefb3-workload-socket\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g\" (UID: \"370ee2a9-2643-4e24-81e4-ab56f3caefb3\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g"
Oct 30 13:03:22.852718 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.851553    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/370ee2a9-2643-4e24-81e4-ab56f3caefb3-istio-podinfo\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g\" (UID: \"370ee2a9-2643-4e24-81e4-ab56f3caefb3\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g"
Oct 30 13:03:22.872939 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.872909    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/370ee2a9-2643-4e24-81e4-ab56f3caefb3-istio-token\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g\" (UID: \"370ee2a9-2643-4e24-81e4-ab56f3caefb3\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g"
Oct 30 13:03:22.872939 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.872931    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26z2l\" (UniqueName: \"kubernetes.io/projected/370ee2a9-2643-4e24-81e4-ab56f3caefb3-kube-api-access-26z2l\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g\" (UID: \"370ee2a9-2643-4e24-81e4-ab56f3caefb3\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g"
Oct 30 13:03:22.943010 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.942977    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g"
Oct 30 13:03:22.944563 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.944531    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7"]
Oct 30 13:03:22.960208 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:03:22.960177    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod134eb93c_b6ee_43e9_8977_950cb0188887.slice/crio-aada12b6f818ef67b6a50fc035fda26b7614a1bd660367754cb1130c292c253f WatchSource:0}: Error finding container aada12b6f818ef67b6a50fc035fda26b7614a1bd660367754cb1130c292c253f: Status 404 returned error can't find the container with id aada12b6f818ef67b6a50fc035fda26b7614a1bd660367754cb1130c292c253f
Oct 30 13:03:22.963086 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.963054    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:03:22.963173 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.963130    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:03:22.963173 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:22.963161    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:03:23.233558 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:23.233526    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7" event={"ID":"134eb93c-b6ee-43e9-8977-950cb0188887","Type":"ContainerStarted","Data":"8109b3a388ae1a87ab9161858a90b4118be61694758679b68e06332870a8a139"}
Oct 30 13:03:23.233731 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:23.233585    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7" event={"ID":"134eb93c-b6ee-43e9-8977-950cb0188887","Type":"ContainerStarted","Data":"aada12b6f818ef67b6a50fc035fda26b7614a1bd660367754cb1130c292c253f"}
Oct 30 13:03:23.235357 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:23.235324    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc" event={"ID":"4710cc0d-1402-489e-826b-5c44a8b6f9f0","Type":"ContainerStarted","Data":"cf0f3f3a05778ebb10b88bdada7234adf4c2c51ebe5e2dce344bdd78a1e7b373"}
Oct 30 13:03:23.235511 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:23.235375    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc" event={"ID":"4710cc0d-1402-489e-826b-5c44a8b6f9f0","Type":"ContainerStarted","Data":"9796fac3320cfec9612196021d5f43926ffb793eab0ecb1a4fa64deb618882f6"}
Oct 30 13:03:23.250062 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:23.250040    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc"
Oct 30 13:03:23.255116 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:23.255096    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc"
Oct 30 13:03:23.273672 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:03:23.273649    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod370ee2a9_2643_4e24_81e4_ab56f3caefb3.slice/crio-df1d24e41308066232260a33b84ffdea7ed5a6a30c9a1ea7dcb4f7f7cba210be WatchSource:0}: Error finding container df1d24e41308066232260a33b84ffdea7ed5a6a30c9a1ea7dcb4f7f7cba210be: Status 404 returned error can't find the container with id df1d24e41308066232260a33b84ffdea7ed5a6a30c9a1ea7dcb4f7f7cba210be
Oct 30 13:03:23.275827 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:23.275801    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:03:23.275913 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:23.275857    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:03:23.275913 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:23.275887    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:03:23.277108 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:23.277086    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g"]
Oct 30 13:03:23.285076 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:23.285035    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7" podStartSLOduration=1.285023239 podStartE2EDuration="1.285023239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:23.282834447 +0000 UTC m=+702.449199096" watchObservedRunningTime="2026-10-30 13:03:23.285023239 +0000 UTC m=+702.451387964"
Oct 30 13:03:23.333797 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:23.333752    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc" podStartSLOduration=2.33373978 podStartE2EDuration="2.33373978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:23.333387523 +0000 UTC m=+702.499752171" watchObservedRunningTime="2026-10-30 13:03:23.33373978 +0000 UTC m=+702.500104424"
Oct 30 13:03:23.601366 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:23.601282    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7"
Oct 30 13:03:24.242104 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:24.242065    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g" event={"ID":"370ee2a9-2643-4e24-81e4-ab56f3caefb3","Type":"ContainerStarted","Data":"294b3d510361d2ce3da926630f9b2f86e2f8539a6c5aac27ae90f6e2b0b5b2bc"}
Oct 30 13:03:24.242567 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:24.242127    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g" event={"ID":"370ee2a9-2643-4e24-81e4-ab56f3caefb3","Type":"ContainerStarted","Data":"df1d24e41308066232260a33b84ffdea7ed5a6a30c9a1ea7dcb4f7f7cba210be"}
Oct 30 13:03:24.242567 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:24.242317    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc"
Oct 30 13:03:24.243138 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:24.243121    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc"
Oct 30 13:03:24.319639 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:24.319579    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g" podStartSLOduration=2.319566564 podStartE2EDuration="2.319566564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:24.317789347 +0000 UTC m=+703.484153995" watchObservedRunningTime="2026-10-30 13:03:24.319566564 +0000 UTC m=+703.485931276"
Oct 30 13:03:24.605411 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:24.605344    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7"
Oct 30 13:03:24.943819 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:24.943791    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g"
Oct 30 13:03:24.948534 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:24.948513    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g"
Oct 30 13:03:25.247416 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:25.247339    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g"
Oct 30 13:03:25.247416 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:25.247387    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7"
Oct 30 13:03:25.248239 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:25.248220    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g"
Oct 30 13:03:25.248338 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:25.248275    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7"
Oct 30 13:03:27.268053 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:27.268018    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-52a649-65c5fbbff9-n6rqr"]
Oct 30 13:03:27.271641 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:27.271618    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-52a649-65c5fbbff9-n6rqr"
Oct 30 13:03:27.291850 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:27.291826    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-52a649\""
Oct 30 13:03:27.306217 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:27.306195    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-52a649-dockercfg-dnbdp\""
Oct 30 13:03:27.308726 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:27.308706    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-52a649-65c5fbbff9-n6rqr"]
Oct 30 13:03:27.399453 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:27.396229    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22d0f282-215c-45d9-9fbb-15190b9ebd10-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-52a649-65c5fbbff9-n6rqr\" (UID: \"22d0f282-215c-45d9-9fbb-15190b9ebd10\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-52a649-65c5fbbff9-n6rqr"
Oct 30 13:03:27.399665 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:27.399525    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmghd\" (UniqueName: \"kubernetes.io/projected/22d0f282-215c-45d9-9fbb-15190b9ebd10-kube-api-access-hmghd\") pod \"payload-pre-processing-e2e-worker-w1-52a649-65c5fbbff9-n6rqr\" (UID: \"22d0f282-215c-45d9-9fbb-15190b9ebd10\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-52a649-65c5fbbff9-n6rqr"
Oct 30 13:03:27.500334 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:27.500303    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22d0f282-215c-45d9-9fbb-15190b9ebd10-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-52a649-65c5fbbff9-n6rqr\" (UID: \"22d0f282-215c-45d9-9fbb-15190b9ebd10\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-52a649-65c5fbbff9-n6rqr"
Oct 30 13:03:27.500502 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:27.500344    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmghd\" (UniqueName: \"kubernetes.io/projected/22d0f282-215c-45d9-9fbb-15190b9ebd10-kube-api-access-hmghd\") pod \"payload-pre-processing-e2e-worker-w1-52a649-65c5fbbff9-n6rqr\" (UID: \"22d0f282-215c-45d9-9fbb-15190b9ebd10\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-52a649-65c5fbbff9-n6rqr"
Oct 30 13:03:27.501084 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:27.501062    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22d0f282-215c-45d9-9fbb-15190b9ebd10-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-52a649-65c5fbbff9-n6rqr\" (UID: \"22d0f282-215c-45d9-9fbb-15190b9ebd10\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-52a649-65c5fbbff9-n6rqr"
Oct 30 13:03:27.539930 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:27.539866    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmghd\" (UniqueName: \"kubernetes.io/projected/22d0f282-215c-45d9-9fbb-15190b9ebd10-kube-api-access-hmghd\") pod \"payload-pre-processing-e2e-worker-w1-52a649-65c5fbbff9-n6rqr\" (UID: \"22d0f282-215c-45d9-9fbb-15190b9ebd10\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-52a649-65c5fbbff9-n6rqr"
Oct 30 13:03:27.585071 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:27.585045    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-52a649-65c5fbbff9-n6rqr"
Oct 30 13:03:27.905201 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:03:27.905171    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22d0f282_215c_45d9_9fbb_15190b9ebd10.slice/crio-f672930499f4e4635e653d77df06bbdae3a07332debde0fae786c72578b4b08e WatchSource:0}: Error finding container f672930499f4e4635e653d77df06bbdae3a07332debde0fae786c72578b4b08e: Status 404 returned error can't find the container with id f672930499f4e4635e653d77df06bbdae3a07332debde0fae786c72578b4b08e
Oct 30 13:03:27.905637 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:27.905583    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-52a649-65c5fbbff9-n6rqr"]
Oct 30 13:03:28.264898 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:28.264824    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-52a649-65c5fbbff9-n6rqr" event={"ID":"22d0f282-215c-45d9-9fbb-15190b9ebd10","Type":"ContainerStarted","Data":"70236c8c1dd87912d3fa93edd91532b2c1d9d7ac2b49a60c725d3ac8c3ba5e54"}
Oct 30 13:03:28.264898 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:28.264875    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-52a649-65c5fbbff9-n6rqr" event={"ID":"22d0f282-215c-45d9-9fbb-15190b9ebd10","Type":"ContainerStarted","Data":"f672930499f4e4635e653d77df06bbdae3a07332debde0fae786c72578b4b08e"}
Oct 30 13:03:28.265076 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:28.264939    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-52a649-65c5fbbff9-n6rqr"
Oct 30 13:03:28.320752 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:28.320704    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-52a649-65c5fbbff9-n6rqr" podStartSLOduration=1.320691859 podStartE2EDuration="1.320691859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:28.318320714 +0000 UTC m=+707.484685361" watchObservedRunningTime="2026-10-30 13:03:28.320691859 +0000 UTC m=+707.487056537"
Oct 30 13:03:29.260898 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:29.260868    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-ee7d65-58cd4c5bbb-6tv8w"]
Oct 30 13:03:29.264507 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:29.264491    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-ee7d65-58cd4c5bbb-6tv8w"
Oct 30 13:03:29.283573 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:29.283551    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-ee7d65-5b75dc6cc5-sv4rn"]
Oct 30 13:03:29.284619 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:29.284588    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-ee7d65-dockercfg-p4sbc\""
Oct 30 13:03:29.286991 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:29.286974    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ee7d65-5b75dc6cc5-sv4rn"
Oct 30 13:03:29.295451 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:29.295428    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-ee7d65\""
Oct 30 13:03:29.301351 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:29.301330    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-ee7d65-58cd4c5bbb-6tv8w"]
Oct 30 13:03:29.325058 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:29.325032    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-ee7d65-5b75dc6cc5-sv4rn"]
Oct 30 13:03:29.419428 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:29.419401    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cbfe471-67e7-43cd-9524-771994bbdb14-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-ee7d65-5b75dc6cc5-sv4rn\" (UID: \"9cbfe471-67e7-43cd-9524-771994bbdb14\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ee7d65-5b75dc6cc5-sv4rn"
Oct 30 13:03:29.419553 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:29.419449    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbqdd\" (UniqueName: \"kubernetes.io/projected/4dc7d59a-ca00-49e0-8f9c-eabe22d71c17-kube-api-access-pbqdd\") pod \"payload-processing-e2e-worker-w6-ee7d65-58cd4c5bbb-6tv8w\" (UID: \"4dc7d59a-ca00-49e0-8f9c-eabe22d71c17\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-ee7d65-58cd4c5bbb-6tv8w"
Oct 30 13:03:29.419553 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:29.419532    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlhrm\" (UniqueName: \"kubernetes.io/projected/9cbfe471-67e7-43cd-9524-771994bbdb14-kube-api-access-jlhrm\") pod \"payload-pre-processing-e2e-worker-w6-ee7d65-5b75dc6cc5-sv4rn\" (UID: \"9cbfe471-67e7-43cd-9524-771994bbdb14\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ee7d65-5b75dc6cc5-sv4rn"
Oct 30 13:03:29.419683 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:29.419574    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dc7d59a-ca00-49e0-8f9c-eabe22d71c17-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-ee7d65-58cd4c5bbb-6tv8w\" (UID: \"4dc7d59a-ca00-49e0-8f9c-eabe22d71c17\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-ee7d65-58cd4c5bbb-6tv8w"
Oct 30 13:03:29.521478 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:29.521380    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cbfe471-67e7-43cd-9524-771994bbdb14-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-ee7d65-5b75dc6cc5-sv4rn\" (UID: \"9cbfe471-67e7-43cd-9524-771994bbdb14\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ee7d65-5b75dc6cc5-sv4rn"
Oct 30 13:03:29.521478 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:29.521473    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbqdd\" (UniqueName: \"kubernetes.io/projected/4dc7d59a-ca00-49e0-8f9c-eabe22d71c17-kube-api-access-pbqdd\") pod \"payload-processing-e2e-worker-w6-ee7d65-58cd4c5bbb-6tv8w\" (UID: \"4dc7d59a-ca00-49e0-8f9c-eabe22d71c17\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-ee7d65-58cd4c5bbb-6tv8w"
Oct 30 13:03:29.521768 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:29.521529    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlhrm\" (UniqueName: \"kubernetes.io/projected/9cbfe471-67e7-43cd-9524-771994bbdb14-kube-api-access-jlhrm\") pod \"payload-pre-processing-e2e-worker-w6-ee7d65-5b75dc6cc5-sv4rn\" (UID: \"9cbfe471-67e7-43cd-9524-771994bbdb14\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ee7d65-5b75dc6cc5-sv4rn"
Oct 30 13:03:29.521768 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:29.521550    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dc7d59a-ca00-49e0-8f9c-eabe22d71c17-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-ee7d65-58cd4c5bbb-6tv8w\" (UID: \"4dc7d59a-ca00-49e0-8f9c-eabe22d71c17\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-ee7d65-58cd4c5bbb-6tv8w"
Oct 30 13:03:29.522698 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:29.522672    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dc7d59a-ca00-49e0-8f9c-eabe22d71c17-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-ee7d65-58cd4c5bbb-6tv8w\" (UID: \"4dc7d59a-ca00-49e0-8f9c-eabe22d71c17\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-ee7d65-58cd4c5bbb-6tv8w"
Oct 30 13:03:29.523572 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:29.523551    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cbfe471-67e7-43cd-9524-771994bbdb14-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-ee7d65-5b75dc6cc5-sv4rn\" (UID: \"9cbfe471-67e7-43cd-9524-771994bbdb14\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ee7d65-5b75dc6cc5-sv4rn"
Oct 30 13:03:29.563479 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:29.563452    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbqdd\" (UniqueName: \"kubernetes.io/projected/4dc7d59a-ca00-49e0-8f9c-eabe22d71c17-kube-api-access-pbqdd\") pod \"payload-processing-e2e-worker-w6-ee7d65-58cd4c5bbb-6tv8w\" (UID: \"4dc7d59a-ca00-49e0-8f9c-eabe22d71c17\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-ee7d65-58cd4c5bbb-6tv8w"
Oct 30 13:03:29.563635 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:29.563583    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlhrm\" (UniqueName: \"kubernetes.io/projected/9cbfe471-67e7-43cd-9524-771994bbdb14-kube-api-access-jlhrm\") pod \"payload-pre-processing-e2e-worker-w6-ee7d65-5b75dc6cc5-sv4rn\" (UID: \"9cbfe471-67e7-43cd-9524-771994bbdb14\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ee7d65-5b75dc6cc5-sv4rn"
Oct 30 13:03:29.578239 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:29.578219    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-ee7d65-58cd4c5bbb-6tv8w"
Oct 30 13:03:29.601220 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:29.601200    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ee7d65-5b75dc6cc5-sv4rn"
Oct 30 13:03:29.917393 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:29.917370    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-ee7d65-58cd4c5bbb-6tv8w"]
Oct 30 13:03:29.919894 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:03:29.919868    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dc7d59a_ca00_49e0_8f9c_eabe22d71c17.slice/crio-4b9a7704fedfa50b5b313eba31c28e36ac26912316b11c2b2db4cc7f811bf705 WatchSource:0}: Error finding container 4b9a7704fedfa50b5b313eba31c28e36ac26912316b11c2b2db4cc7f811bf705: Status 404 returned error can't find the container with id 4b9a7704fedfa50b5b313eba31c28e36ac26912316b11c2b2db4cc7f811bf705
Oct 30 13:03:29.923830 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:29.923805    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-ee7d65-5b75dc6cc5-sv4rn"]
Oct 30 13:03:29.930925 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:03:29.930899    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cbfe471_67e7_43cd_9524_771994bbdb14.slice/crio-23f73d43f3bf8ea81d71e199c8de20153e83b8b4a9835ab0d378479b877073ee WatchSource:0}: Error finding container 23f73d43f3bf8ea81d71e199c8de20153e83b8b4a9835ab0d378479b877073ee: Status 404 returned error can't find the container with id 23f73d43f3bf8ea81d71e199c8de20153e83b8b4a9835ab0d378479b877073ee
Oct 30 13:03:30.276161 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:30.276074    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ee7d65-5b75dc6cc5-sv4rn" event={"ID":"9cbfe471-67e7-43cd-9524-771994bbdb14","Type":"ContainerStarted","Data":"cb5c4578d610d53beff85a1a3201787e5e2ac75254cdc8e63b5b1d469935f87d"}
Oct 30 13:03:30.276161 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:30.276135    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ee7d65-5b75dc6cc5-sv4rn"
Oct 30 13:03:30.276161 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:30.276157    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ee7d65-5b75dc6cc5-sv4rn" event={"ID":"9cbfe471-67e7-43cd-9524-771994bbdb14","Type":"ContainerStarted","Data":"23f73d43f3bf8ea81d71e199c8de20153e83b8b4a9835ab0d378479b877073ee"}
Oct 30 13:03:30.277738 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:30.277713    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-ee7d65-58cd4c5bbb-6tv8w" event={"ID":"4dc7d59a-ca00-49e0-8f9c-eabe22d71c17","Type":"ContainerStarted","Data":"bbb2700a8683299448b69f70ad21143d3d2a0ddec715b7b006760f597b2b48be"}
Oct 30 13:03:30.277844 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:30.277762    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-ee7d65-58cd4c5bbb-6tv8w"
Oct 30 13:03:30.277844 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:30.277783    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-ee7d65-58cd4c5bbb-6tv8w" event={"ID":"4dc7d59a-ca00-49e0-8f9c-eabe22d71c17","Type":"ContainerStarted","Data":"4b9a7704fedfa50b5b313eba31c28e36ac26912316b11c2b2db4cc7f811bf705"}
Oct 30 13:03:30.325305 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:30.325256    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ee7d65-5b75dc6cc5-sv4rn" podStartSLOduration=1.325243372 podStartE2EDuration="1.325243372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:30.324250252 +0000 UTC m=+709.490614921" watchObservedRunningTime="2026-10-30 13:03:30.325243372 +0000 UTC m=+709.491608019"
Oct 30 13:03:30.372702 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:30.372661    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-ee7d65-58cd4c5bbb-6tv8w" podStartSLOduration=1.372649279 podStartE2EDuration="1.372649279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:30.371110643 +0000 UTC m=+709.537475292" watchObservedRunningTime="2026-10-30 13:03:30.372649279 +0000 UTC m=+709.539013926"
Oct 30 13:03:31.744723 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:31.744696    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-cb58b8-b449f8b58-r97fs"]
Oct 30 13:03:31.748434 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:31.748413    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-cb58b8-b449f8b58-r97fs"
Oct 30 13:03:31.752813 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:31.752790    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f42a48a8-5f5e-459c-8276-dd30d02052a7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-cb58b8-b449f8b58-r97fs\" (UID: \"f42a48a8-5f5e-459c-8276-dd30d02052a7\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-cb58b8-b449f8b58-r97fs"
Oct 30 13:03:31.752941 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:31.752845    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7mlx\" (UniqueName: \"kubernetes.io/projected/f42a48a8-5f5e-459c-8276-dd30d02052a7-kube-api-access-n7mlx\") pod \"payload-processing-e2e-worker-w2-cb58b8-b449f8b58-r97fs\" (UID: \"f42a48a8-5f5e-459c-8276-dd30d02052a7\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-cb58b8-b449f8b58-r97fs"
Oct 30 13:03:31.766803 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:31.766782    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-cb58b8-78744489c9-lshhg"]
Oct 30 13:03:31.768477 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:31.768455    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-cb58b8-dockercfg-qc97g\""
Oct 30 13:03:31.770082 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:31.770069    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-cb58b8-78744489c9-lshhg"
Oct 30 13:03:31.778658 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:31.778639    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-cb58b8\""
Oct 30 13:03:31.785036 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:31.785015    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-cb58b8-b449f8b58-r97fs"]
Oct 30 13:03:31.791526 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:31.791500    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-cb58b8-78744489c9-lshhg"]
Oct 30 13:03:31.855510 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:31.855471    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7mlx\" (UniqueName: \"kubernetes.io/projected/f42a48a8-5f5e-459c-8276-dd30d02052a7-kube-api-access-n7mlx\") pod \"payload-processing-e2e-worker-w2-cb58b8-b449f8b58-r97fs\" (UID: \"f42a48a8-5f5e-459c-8276-dd30d02052a7\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-cb58b8-b449f8b58-r97fs"
Oct 30 13:03:31.855728 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:31.855570    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p25fk\" (UniqueName: \"kubernetes.io/projected/0b45c4c8-a116-43ce-9e06-6b4ceca907e2-kube-api-access-p25fk\") pod \"payload-pre-processing-e2e-worker-w2-cb58b8-78744489c9-lshhg\" (UID: \"0b45c4c8-a116-43ce-9e06-6b4ceca907e2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-cb58b8-78744489c9-lshhg"
Oct 30 13:03:31.855728 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:31.855669    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f42a48a8-5f5e-459c-8276-dd30d02052a7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-cb58b8-b449f8b58-r97fs\" (UID: \"f42a48a8-5f5e-459c-8276-dd30d02052a7\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-cb58b8-b449f8b58-r97fs"
Oct 30 13:03:31.855852 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:31.855733    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b45c4c8-a116-43ce-9e06-6b4ceca907e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-cb58b8-78744489c9-lshhg\" (UID: \"0b45c4c8-a116-43ce-9e06-6b4ceca907e2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-cb58b8-78744489c9-lshhg"
Oct 30 13:03:31.856952 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:31.856925    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f42a48a8-5f5e-459c-8276-dd30d02052a7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-cb58b8-b449f8b58-r97fs\" (UID: \"f42a48a8-5f5e-459c-8276-dd30d02052a7\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-cb58b8-b449f8b58-r97fs"
Oct 30 13:03:31.894745 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:31.894719    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7mlx\" (UniqueName: \"kubernetes.io/projected/f42a48a8-5f5e-459c-8276-dd30d02052a7-kube-api-access-n7mlx\") pod \"payload-processing-e2e-worker-w2-cb58b8-b449f8b58-r97fs\" (UID: \"f42a48a8-5f5e-459c-8276-dd30d02052a7\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-cb58b8-b449f8b58-r97fs"
Oct 30 13:03:31.956327 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:31.956302    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p25fk\" (UniqueName: \"kubernetes.io/projected/0b45c4c8-a116-43ce-9e06-6b4ceca907e2-kube-api-access-p25fk\") pod \"payload-pre-processing-e2e-worker-w2-cb58b8-78744489c9-lshhg\" (UID: \"0b45c4c8-a116-43ce-9e06-6b4ceca907e2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-cb58b8-78744489c9-lshhg"
Oct 30 13:03:31.956459 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:31.956370    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b45c4c8-a116-43ce-9e06-6b4ceca907e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-cb58b8-78744489c9-lshhg\" (UID: \"0b45c4c8-a116-43ce-9e06-6b4ceca907e2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-cb58b8-78744489c9-lshhg"
Oct 30 13:03:31.957023 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:31.957004    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b45c4c8-a116-43ce-9e06-6b4ceca907e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-cb58b8-78744489c9-lshhg\" (UID: \"0b45c4c8-a116-43ce-9e06-6b4ceca907e2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-cb58b8-78744489c9-lshhg"
Oct 30 13:03:31.981884 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:31.981863    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p25fk\" (UniqueName: \"kubernetes.io/projected/0b45c4c8-a116-43ce-9e06-6b4ceca907e2-kube-api-access-p25fk\") pod \"payload-pre-processing-e2e-worker-w2-cb58b8-78744489c9-lshhg\" (UID: \"0b45c4c8-a116-43ce-9e06-6b4ceca907e2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-cb58b8-78744489c9-lshhg"
Oct 30 13:03:32.029784 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:32.029677    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:03:32.062196 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:32.062162    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-cb58b8-b449f8b58-r97fs"
Oct 30 13:03:32.084378 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:32.084357    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-cb58b8-78744489c9-lshhg"
Oct 30 13:03:32.375881 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:32.375851    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-cb58b8-b449f8b58-r97fs"]
Oct 30 13:03:32.383099 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:03:32.383067    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf42a48a8_5f5e_459c_8276_dd30d02052a7.slice/crio-42d3168f4971db601c8f2a2ebcf5ac1263a13aff27c7ba46ee30c5893497c35e WatchSource:0}: Error finding container 42d3168f4971db601c8f2a2ebcf5ac1263a13aff27c7ba46ee30c5893497c35e: Status 404 returned error can't find the container with id 42d3168f4971db601c8f2a2ebcf5ac1263a13aff27c7ba46ee30c5893497c35e
Oct 30 13:03:32.398775 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:32.398460    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-cb58b8-78744489c9-lshhg"]
Oct 30 13:03:32.399236 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:03:32.399212    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b45c4c8_a116_43ce_9e06_6b4ceca907e2.slice/crio-0e8dd6761ea66081d27d6f36c998af0a00ee82f5ceb374b12478fb33bb0350e4 WatchSource:0}: Error finding container 0e8dd6761ea66081d27d6f36c998af0a00ee82f5ceb374b12478fb33bb0350e4: Status 404 returned error can't find the container with id 0e8dd6761ea66081d27d6f36c998af0a00ee82f5ceb374b12478fb33bb0350e4
Oct 30 13:03:33.296058 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:33.296012    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-cb58b8-b449f8b58-r97fs" event={"ID":"f42a48a8-5f5e-459c-8276-dd30d02052a7","Type":"ContainerStarted","Data":"4dee5c972eb3862e27b351da3a3746d5616c2fc60f0d461cf7e7cc0dc3ea87d3"}
Oct 30 13:03:33.296547 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:33.296083    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-cb58b8-b449f8b58-r97fs" event={"ID":"f42a48a8-5f5e-459c-8276-dd30d02052a7","Type":"ContainerStarted","Data":"42d3168f4971db601c8f2a2ebcf5ac1263a13aff27c7ba46ee30c5893497c35e"}
Oct 30 13:03:33.296547 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:33.296204    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-cb58b8-b449f8b58-r97fs"
Oct 30 13:03:33.297855 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:33.297831    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-cb58b8-78744489c9-lshhg" event={"ID":"0b45c4c8-a116-43ce-9e06-6b4ceca907e2","Type":"ContainerStarted","Data":"063041b92ffbf4394ddc2c754fcddb54dfb173cc1618a5510d9fb1273f0b2ed3"}
Oct 30 13:03:33.298003 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:33.297865    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-cb58b8-78744489c9-lshhg" event={"ID":"0b45c4c8-a116-43ce-9e06-6b4ceca907e2","Type":"ContainerStarted","Data":"0e8dd6761ea66081d27d6f36c998af0a00ee82f5ceb374b12478fb33bb0350e4"}
Oct 30 13:03:33.298003 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:33.297944    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-cb58b8-78744489c9-lshhg"
Oct 30 13:03:33.357972 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:33.357932    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-cb58b8-b449f8b58-r97fs" podStartSLOduration=2.357921659 podStartE2EDuration="2.357921659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:33.35424803 +0000 UTC m=+712.520612680" watchObservedRunningTime="2026-10-30 13:03:33.357921659 +0000 UTC m=+712.524286307"
Oct 30 13:03:33.417713 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:33.417662    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-cb58b8-78744489c9-lshhg" podStartSLOduration=2.417646799 podStartE2EDuration="2.417646799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:33.415794489 +0000 UTC m=+712.582159148" watchObservedRunningTime="2026-10-30 13:03:33.417646799 +0000 UTC m=+712.584011447"
Oct 30 13:03:35.808673 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:35.808642    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-d7e62b-6869d5f5b7-ztlqq"]
Oct 30 13:03:35.812498 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:35.812473    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d7e62b-6869d5f5b7-ztlqq"
Oct 30 13:03:35.833957 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:35.833931    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-d7e62b-6869d5f5b7-ztlqq"]
Oct 30 13:03:35.844775 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:35.844752    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-d7e62b\""
Oct 30 13:03:35.844881 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:35.844842    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-d7e62b-dockercfg-4ln2g\""
Oct 30 13:03:35.851885 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:35.851863    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-d7e62b-6fb765d78-mqjvs"]
Oct 30 13:03:35.856165 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:35.856148    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-d7e62b-6fb765d78-mqjvs"
Oct 30 13:03:35.878834 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:35.878810    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-d7e62b-6fb765d78-mqjvs"]
Oct 30 13:03:35.911170 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:35.911143    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9195c96-60b4-49ff-a742-34b7ce75c7c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-d7e62b-6869d5f5b7-ztlqq\" (UID: \"c9195c96-60b4-49ff-a742-34b7ce75c7c6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d7e62b-6869d5f5b7-ztlqq"
Oct 30 13:03:35.911287 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:35.911250    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lqrj\" (UniqueName: \"kubernetes.io/projected/c9195c96-60b4-49ff-a742-34b7ce75c7c6-kube-api-access-4lqrj\") pod \"payload-pre-processing-e2e-worker-w4-d7e62b-6869d5f5b7-ztlqq\" (UID: \"c9195c96-60b4-49ff-a742-34b7ce75c7c6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d7e62b-6869d5f5b7-ztlqq"
Oct 30 13:03:35.911388 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:35.911358    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkrxt\" (UniqueName: \"kubernetes.io/projected/48e64d9a-c945-4316-b973-6829e30b051b-kube-api-access-zkrxt\") pod \"payload-processing-e2e-worker-w4-d7e62b-6fb765d78-mqjvs\" (UID: \"48e64d9a-c945-4316-b973-6829e30b051b\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-d7e62b-6fb765d78-mqjvs"
Oct 30 13:03:35.911455 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:35.911440    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48e64d9a-c945-4316-b973-6829e30b051b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-d7e62b-6fb765d78-mqjvs\" (UID: \"48e64d9a-c945-4316-b973-6829e30b051b\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-d7e62b-6fb765d78-mqjvs"
Oct 30 13:03:36.012199 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:36.012168    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48e64d9a-c945-4316-b973-6829e30b051b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-d7e62b-6fb765d78-mqjvs\" (UID: \"48e64d9a-c945-4316-b973-6829e30b051b\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-d7e62b-6fb765d78-mqjvs"
Oct 30 13:03:36.012376 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:36.012216    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9195c96-60b4-49ff-a742-34b7ce75c7c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-d7e62b-6869d5f5b7-ztlqq\" (UID: \"c9195c96-60b4-49ff-a742-34b7ce75c7c6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d7e62b-6869d5f5b7-ztlqq"
Oct 30 13:03:36.012376 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:36.012267    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lqrj\" (UniqueName: \"kubernetes.io/projected/c9195c96-60b4-49ff-a742-34b7ce75c7c6-kube-api-access-4lqrj\") pod \"payload-pre-processing-e2e-worker-w4-d7e62b-6869d5f5b7-ztlqq\" (UID: \"c9195c96-60b4-49ff-a742-34b7ce75c7c6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d7e62b-6869d5f5b7-ztlqq"
Oct 30 13:03:36.012376 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:36.012314    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkrxt\" (UniqueName: \"kubernetes.io/projected/48e64d9a-c945-4316-b973-6829e30b051b-kube-api-access-zkrxt\") pod \"payload-processing-e2e-worker-w4-d7e62b-6fb765d78-mqjvs\" (UID: \"48e64d9a-c945-4316-b973-6829e30b051b\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-d7e62b-6fb765d78-mqjvs"
Oct 30 13:03:36.012955 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:36.012933    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48e64d9a-c945-4316-b973-6829e30b051b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-d7e62b-6fb765d78-mqjvs\" (UID: \"48e64d9a-c945-4316-b973-6829e30b051b\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-d7e62b-6fb765d78-mqjvs"
Oct 30 13:03:36.013049 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:36.013029    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9195c96-60b4-49ff-a742-34b7ce75c7c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-d7e62b-6869d5f5b7-ztlqq\" (UID: \"c9195c96-60b4-49ff-a742-34b7ce75c7c6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d7e62b-6869d5f5b7-ztlqq"
Oct 30 13:03:36.037878 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:36.037850    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkrxt\" (UniqueName: \"kubernetes.io/projected/48e64d9a-c945-4316-b973-6829e30b051b-kube-api-access-zkrxt\") pod \"payload-processing-e2e-worker-w4-d7e62b-6fb765d78-mqjvs\" (UID: \"48e64d9a-c945-4316-b973-6829e30b051b\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-d7e62b-6fb765d78-mqjvs"
Oct 30 13:03:36.038004 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:36.037947    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lqrj\" (UniqueName: \"kubernetes.io/projected/c9195c96-60b4-49ff-a742-34b7ce75c7c6-kube-api-access-4lqrj\") pod \"payload-pre-processing-e2e-worker-w4-d7e62b-6869d5f5b7-ztlqq\" (UID: \"c9195c96-60b4-49ff-a742-34b7ce75c7c6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d7e62b-6869d5f5b7-ztlqq"
Oct 30 13:03:36.125341 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:36.125322    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d7e62b-6869d5f5b7-ztlqq"
Oct 30 13:03:36.170963 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:36.170939    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-d7e62b-6fb765d78-mqjvs"
Oct 30 13:03:36.434933 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:36.434907    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-d7e62b-6869d5f5b7-ztlqq"]
Oct 30 13:03:36.488050 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:36.488023    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-d7e62b-6fb765d78-mqjvs"]
Oct 30 13:03:36.488255 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:03:36.488231    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48e64d9a_c945_4316_b973_6829e30b051b.slice/crio-66c76fccd1c125a7a72cebf8394a6d64841f26e4740d27f7137a0b45dfb7c284 WatchSource:0}: Error finding container 66c76fccd1c125a7a72cebf8394a6d64841f26e4740d27f7137a0b45dfb7c284: Status 404 returned error can't find the container with id 66c76fccd1c125a7a72cebf8394a6d64841f26e4740d27f7137a0b45dfb7c284
Oct 30 13:03:37.031418 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:37.031375    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-d833bd-5cf45d8d99-8jl44"]
Oct 30 13:03:37.045635 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:37.037781    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d833bd-5cf45d8d99-8jl44"
Oct 30 13:03:37.050462 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:37.050433    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-d833bd-5cf45d8d99-8jl44"]
Oct 30 13:03:37.060645 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:37.060577    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:03:37.060645 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:37.060588    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-zw9vx\""
Oct 30 13:03:37.061784 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:37.060963    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 13:03:37.072296 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:37.072276    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:03:37.075056 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:37.075038    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-d833bd-serving-cert\""
Oct 30 13:03:37.129577 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:37.129551    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8188711a-ba90-4fdd-9184-1b604692e9bb-metrics-certs\") pod \"maas-api-e2e-worker-w3-d833bd-5cf45d8d99-8jl44\" (UID: \"8188711a-ba90-4fdd-9184-1b604692e9bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d833bd-5cf45d8d99-8jl44"
Oct 30 13:03:37.129696 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:37.129584    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8188711a-ba90-4fdd-9184-1b604692e9bb-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-d833bd-5cf45d8d99-8jl44\" (UID: \"8188711a-ba90-4fdd-9184-1b604692e9bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d833bd-5cf45d8d99-8jl44"
Oct 30 13:03:37.129736 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:37.129711    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8188711a-ba90-4fdd-9184-1b604692e9bb-maas-api-tls\") pod \"maas-api-e2e-worker-w3-d833bd-5cf45d8d99-8jl44\" (UID: \"8188711a-ba90-4fdd-9184-1b604692e9bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d833bd-5cf45d8d99-8jl44"
Oct 30 13:03:37.129875 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:37.129857    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n88j2\" (UniqueName: \"kubernetes.io/projected/8188711a-ba90-4fdd-9184-1b604692e9bb-kube-api-access-n88j2\") pod \"maas-api-e2e-worker-w3-d833bd-5cf45d8d99-8jl44\" (UID: \"8188711a-ba90-4fdd-9184-1b604692e9bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d833bd-5cf45d8d99-8jl44"
Oct 30 13:03:37.230730 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:37.230668    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8188711a-ba90-4fdd-9184-1b604692e9bb-maas-api-tls\") pod \"maas-api-e2e-worker-w3-d833bd-5cf45d8d99-8jl44\" (UID: \"8188711a-ba90-4fdd-9184-1b604692e9bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d833bd-5cf45d8d99-8jl44"
Oct 30 13:03:37.230892 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:37.230871    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n88j2\" (UniqueName: \"kubernetes.io/projected/8188711a-ba90-4fdd-9184-1b604692e9bb-kube-api-access-n88j2\") pod \"maas-api-e2e-worker-w3-d833bd-5cf45d8d99-8jl44\" (UID: \"8188711a-ba90-4fdd-9184-1b604692e9bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d833bd-5cf45d8d99-8jl44"
Oct 30 13:03:37.230968 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:37.230900    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8188711a-ba90-4fdd-9184-1b604692e9bb-metrics-certs\") pod \"maas-api-e2e-worker-w3-d833bd-5cf45d8d99-8jl44\" (UID: \"8188711a-ba90-4fdd-9184-1b604692e9bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d833bd-5cf45d8d99-8jl44"
Oct 30 13:03:37.230968 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:37.230923    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8188711a-ba90-4fdd-9184-1b604692e9bb-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-d833bd-5cf45d8d99-8jl44\" (UID: \"8188711a-ba90-4fdd-9184-1b604692e9bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d833bd-5cf45d8d99-8jl44"
Oct 30 13:03:37.231738 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:37.231716    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8188711a-ba90-4fdd-9184-1b604692e9bb-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-d833bd-5cf45d8d99-8jl44\" (UID: \"8188711a-ba90-4fdd-9184-1b604692e9bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d833bd-5cf45d8d99-8jl44"
Oct 30 13:03:37.233136 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:37.233115    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8188711a-ba90-4fdd-9184-1b604692e9bb-maas-api-tls\") pod \"maas-api-e2e-worker-w3-d833bd-5cf45d8d99-8jl44\" (UID: \"8188711a-ba90-4fdd-9184-1b604692e9bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d833bd-5cf45d8d99-8jl44"
Oct 30 13:03:37.233868 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:37.233847    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8188711a-ba90-4fdd-9184-1b604692e9bb-metrics-certs\") pod \"maas-api-e2e-worker-w3-d833bd-5cf45d8d99-8jl44\" (UID: \"8188711a-ba90-4fdd-9184-1b604692e9bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d833bd-5cf45d8d99-8jl44"
Oct 30 13:03:37.258106 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:37.258082    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n88j2\" (UniqueName: \"kubernetes.io/projected/8188711a-ba90-4fdd-9184-1b604692e9bb-kube-api-access-n88j2\") pod \"maas-api-e2e-worker-w3-d833bd-5cf45d8d99-8jl44\" (UID: \"8188711a-ba90-4fdd-9184-1b604692e9bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d833bd-5cf45d8d99-8jl44"
Oct 30 13:03:37.327482 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:37.327410    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d7e62b-6869d5f5b7-ztlqq" event={"ID":"c9195c96-60b4-49ff-a742-34b7ce75c7c6","Type":"ContainerStarted","Data":"52e08edc6f1f993e1fdc01e00a46db6a51137044a501ff637f35432ac33c5b1b"}
Oct 30 13:03:37.327575 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:37.327484    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d7e62b-6869d5f5b7-ztlqq"
Oct 30 13:03:37.327575 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:37.327507    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d7e62b-6869d5f5b7-ztlqq" event={"ID":"c9195c96-60b4-49ff-a742-34b7ce75c7c6","Type":"ContainerStarted","Data":"90215cfafa5af1c2f59e20baec7db46fbfce7301eb10e4c5665c18ae2f8c8147"}
Oct 30 13:03:37.329020 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:37.328999    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-d7e62b-6fb765d78-mqjvs" event={"ID":"48e64d9a-c945-4316-b973-6829e30b051b","Type":"ContainerStarted","Data":"6eb00dee5eb32002ae86054ce73c90c8f9396a3696721df345a9bef23945fe8c"}
Oct 30 13:03:37.329114 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:37.329044    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-d7e62b-6fb765d78-mqjvs" event={"ID":"48e64d9a-c945-4316-b973-6829e30b051b","Type":"ContainerStarted","Data":"66c76fccd1c125a7a72cebf8394a6d64841f26e4740d27f7137a0b45dfb7c284"}
Oct 30 13:03:37.329114 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:37.329072    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-d7e62b-6fb765d78-mqjvs"
Oct 30 13:03:37.370359 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:37.370335    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d833bd-5cf45d8d99-8jl44"
Oct 30 13:03:37.418956 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:37.418904    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d7e62b-6869d5f5b7-ztlqq" podStartSLOduration=2.41888961 podStartE2EDuration="2.41888961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:37.412068565 +0000 UTC m=+716.578433212" watchObservedRunningTime="2026-10-30 13:03:37.41888961 +0000 UTC m=+716.585254260"
Oct 30 13:03:37.506192 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:37.505516    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-d7e62b-6fb765d78-mqjvs" podStartSLOduration=2.505497545 podStartE2EDuration="2.505497545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:37.499187019 +0000 UTC m=+716.665551669" watchObservedRunningTime="2026-10-30 13:03:37.505497545 +0000 UTC m=+716.671862192"
Oct 30 13:03:37.689162 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:37.689131    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-d833bd-5cf45d8d99-8jl44"]
Oct 30 13:03:37.690334 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:03:37.690311    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8188711a_ba90_4fdd_9184_1b604692e9bb.slice/crio-a6a90edbd76f034f20d0ede2efa336e3382082f3f2dd96a034e1e5e65b925279 WatchSource:0}: Error finding container a6a90edbd76f034f20d0ede2efa336e3382082f3f2dd96a034e1e5e65b925279: Status 404 returned error can't find the container with id a6a90edbd76f034f20d0ede2efa336e3382082f3f2dd96a034e1e5e65b925279
Oct 30 13:03:38.337569 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:38.337526    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d833bd-5cf45d8d99-8jl44" event={"ID":"8188711a-ba90-4fdd-9184-1b604692e9bb","Type":"ContainerStarted","Data":"a6a90edbd76f034f20d0ede2efa336e3382082f3f2dd96a034e1e5e65b925279"}
Oct 30 13:03:39.273665 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:39.273630    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-52a649-65c5fbbff9-n6rqr"
Oct 30 13:03:41.304663 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:41.304561    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ee7d65-5b75dc6cc5-sv4rn"
Oct 30 13:03:41.305370 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:41.305347    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-ee7d65-58cd4c5bbb-6tv8w"
Oct 30 13:03:41.362083 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:41.362055    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d833bd-5cf45d8d99-8jl44" event={"ID":"8188711a-ba90-4fdd-9184-1b604692e9bb","Type":"ContainerStarted","Data":"b372056a6f75c6ae7ebea664eb874e25338ffd0fbcc9ca07e9362a8d08d549e0"}
Oct 30 13:03:41.362227 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:41.362147    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d833bd-5cf45d8d99-8jl44"
Oct 30 13:03:41.508207 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:41.508099    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:03:41.510052 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:41.509990    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d833bd-5cf45d8d99-8jl44" podStartSLOduration=2.539011376 podStartE2EDuration="5.509972044s" totalImagesPullingTime="2.970960668s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:41.501213263 +0000 UTC m=+720.667577912" watchObservedRunningTime="2026-10-30 13:03:41.509972044 +0000 UTC m=+720.676336693"
Oct 30 13:03:41.529852 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:03:41.529823    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:03:44.308098 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:44.308063    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-cb58b8-78744489c9-lshhg"
Oct 30 13:03:44.308637 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:44.308144    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-cb58b8-b449f8b58-r97fs"
Oct 30 13:03:47.372826 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:47.372801    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d833bd-5cf45d8d99-8jl44"
Oct 30 13:03:47.409958 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:47.409922    4364 generic.go:356] "Generic (PLEG): container finished" podID="4710cc0d-1402-489e-826b-5c44a8b6f9f0" containerID="cf0f3f3a05778ebb10b88bdada7234adf4c2c51ebe5e2dce344bdd78a1e7b373" exitCode=0
Oct 30 13:03:47.410151 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:47.410017    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc" event={"ID":"4710cc0d-1402-489e-826b-5c44a8b6f9f0","Type":"ContainerDied","Data":"cf0f3f3a05778ebb10b88bdada7234adf4c2c51ebe5e2dce344bdd78a1e7b373"}
Oct 30 13:03:47.410526 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:47.410506    4364 scope.go:122] "RemoveContainer" containerID="cf0f3f3a05778ebb10b88bdada7234adf4c2c51ebe5e2dce344bdd78a1e7b373"
Oct 30 13:03:47.411934 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:47.411912    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:03:47.412000 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:47.411978    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:03:47.412038 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:47.412010    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:03:47.911231 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:47.911189    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-e34fd5-758756996-tl6mw"]
Oct 30 13:03:47.913898 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:47.913878    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e34fd5-758756996-tl6mw"
Oct 30 13:03:47.933271 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:47.933240    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-e34fd5-758756996-tl6mw"]
Oct 30 13:03:47.936310 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:47.936291    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-e34fd5-serving-cert\""
Oct 30 13:03:48.076029 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:48.075992    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/638d73f1-8339-4b13-a492-2ab426a0ddd6-maas-api-tls\") pod \"maas-api-e2e-worker-w5-e34fd5-758756996-tl6mw\" (UID: \"638d73f1-8339-4b13-a492-2ab426a0ddd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e34fd5-758756996-tl6mw"
Oct 30 13:03:48.076029 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:48.076033    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/638d73f1-8339-4b13-a492-2ab426a0ddd6-metrics-certs\") pod \"maas-api-e2e-worker-w5-e34fd5-758756996-tl6mw\" (UID: \"638d73f1-8339-4b13-a492-2ab426a0ddd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e34fd5-758756996-tl6mw"
Oct 30 13:03:48.076270 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:48.076064    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-584zn\" (UniqueName: \"kubernetes.io/projected/638d73f1-8339-4b13-a492-2ab426a0ddd6-kube-api-access-584zn\") pod \"maas-api-e2e-worker-w5-e34fd5-758756996-tl6mw\" (UID: \"638d73f1-8339-4b13-a492-2ab426a0ddd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e34fd5-758756996-tl6mw"
Oct 30 13:03:48.076270 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:48.076200    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/638d73f1-8339-4b13-a492-2ab426a0ddd6-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-e34fd5-758756996-tl6mw\" (UID: \"638d73f1-8339-4b13-a492-2ab426a0ddd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e34fd5-758756996-tl6mw"
Oct 30 13:03:48.177097 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:48.177011    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/638d73f1-8339-4b13-a492-2ab426a0ddd6-maas-api-tls\") pod \"maas-api-e2e-worker-w5-e34fd5-758756996-tl6mw\" (UID: \"638d73f1-8339-4b13-a492-2ab426a0ddd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e34fd5-758756996-tl6mw"
Oct 30 13:03:48.177097 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:48.177055    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/638d73f1-8339-4b13-a492-2ab426a0ddd6-metrics-certs\") pod \"maas-api-e2e-worker-w5-e34fd5-758756996-tl6mw\" (UID: \"638d73f1-8339-4b13-a492-2ab426a0ddd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e34fd5-758756996-tl6mw"
Oct 30 13:03:48.177097 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:48.177081    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-584zn\" (UniqueName: \"kubernetes.io/projected/638d73f1-8339-4b13-a492-2ab426a0ddd6-kube-api-access-584zn\") pod \"maas-api-e2e-worker-w5-e34fd5-758756996-tl6mw\" (UID: \"638d73f1-8339-4b13-a492-2ab426a0ddd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e34fd5-758756996-tl6mw"
Oct 30 13:03:48.177378 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:48.177123    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/638d73f1-8339-4b13-a492-2ab426a0ddd6-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-e34fd5-758756996-tl6mw\" (UID: \"638d73f1-8339-4b13-a492-2ab426a0ddd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e34fd5-758756996-tl6mw"
Oct 30 13:03:48.177866 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:48.177846    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/638d73f1-8339-4b13-a492-2ab426a0ddd6-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-e34fd5-758756996-tl6mw\" (UID: \"638d73f1-8339-4b13-a492-2ab426a0ddd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e34fd5-758756996-tl6mw"
Oct 30 13:03:48.201621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:48.198071    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/638d73f1-8339-4b13-a492-2ab426a0ddd6-metrics-certs\") pod \"maas-api-e2e-worker-w5-e34fd5-758756996-tl6mw\" (UID: \"638d73f1-8339-4b13-a492-2ab426a0ddd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e34fd5-758756996-tl6mw"
Oct 30 13:03:48.206501 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:48.206472    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/638d73f1-8339-4b13-a492-2ab426a0ddd6-maas-api-tls\") pod \"maas-api-e2e-worker-w5-e34fd5-758756996-tl6mw\" (UID: \"638d73f1-8339-4b13-a492-2ab426a0ddd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e34fd5-758756996-tl6mw"
Oct 30 13:03:48.225219 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:48.225188    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-584zn\" (UniqueName: \"kubernetes.io/projected/638d73f1-8339-4b13-a492-2ab426a0ddd6-kube-api-access-584zn\") pod \"maas-api-e2e-worker-w5-e34fd5-758756996-tl6mw\" (UID: \"638d73f1-8339-4b13-a492-2ab426a0ddd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e34fd5-758756996-tl6mw"
Oct 30 13:03:48.225655 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:48.225571    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e34fd5-758756996-tl6mw"
Oct 30 13:03:48.345912 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:48.345879    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d7e62b-6869d5f5b7-ztlqq"
Oct 30 13:03:48.347712 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:48.347688    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-d7e62b-6fb765d78-mqjvs"
Oct 30 13:03:48.446675 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:48.446572    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc" event={"ID":"4710cc0d-1402-489e-826b-5c44a8b6f9f0","Type":"ContainerStarted","Data":"c2420b0d52c0c1fce4aae2114400d2c4a23e4850cd720b7e4909ce684e796ac9"}
Oct 30 13:03:48.640723 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:48.640119    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-e34fd5-758756996-tl6mw"]
Oct 30 13:03:49.250296 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:49.250255    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc"
Oct 30 13:03:49.250859 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:49.250823    4364 patch_prober.go:29] interesting pod/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Oct 30 13:03:49.250922 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:49.250896    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc" podUID="4710cc0d-1402-489e-826b-5c44a8b6f9f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Oct 30 13:03:49.453548 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:49.453510    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e34fd5-758756996-tl6mw" event={"ID":"638d73f1-8339-4b13-a492-2ab426a0ddd6","Type":"ContainerStarted","Data":"5cef9e3c86a225340ff0075624244a18a0290ed9a09086736ccfb65cb4ff3850"}
Oct 30 13:03:49.454051 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:49.453562    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e34fd5-758756996-tl6mw" event={"ID":"638d73f1-8339-4b13-a492-2ab426a0ddd6","Type":"ContainerStarted","Data":"abf86361b0102b39852fdaaef2963c717f917be1eae99447cbc8c4eeda976045"}
Oct 30 13:03:49.454051 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:49.453881    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e34fd5-758756996-tl6mw"
Oct 30 13:03:49.520621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:49.514275    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e34fd5-758756996-tl6mw" podStartSLOduration=2.090236717 podStartE2EDuration="2.51425466s" totalImagesPullingTime="424.017943ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:49.512976097 +0000 UTC m=+728.679340773" watchObservedRunningTime="2026-10-30 13:03:49.51425466 +0000 UTC m=+728.680619309"
Oct 30 13:03:50.250507 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:50.250471    4364 patch_prober.go:29] interesting pod/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Oct 30 13:03:50.250715 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:50.250540    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc" podUID="4710cc0d-1402-489e-826b-5c44a8b6f9f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Oct 30 13:03:51.250112 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:51.250085    4364 patch_prober.go:29] interesting pod/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Oct 30 13:03:51.250581 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:51.250553    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc" podUID="4710cc0d-1402-489e-826b-5c44a8b6f9f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Oct 30 13:03:52.250162 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:52.250115    4364 patch_prober.go:29] interesting pod/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Oct 30 13:03:52.250580 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:52.250184    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc" podUID="4710cc0d-1402-489e-826b-5c44a8b6f9f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Oct 30 13:03:53.250286 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:53.250254    4364 patch_prober.go:29] interesting pod/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Oct 30 13:03:53.250749 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:53.250722    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc" podUID="4710cc0d-1402-489e-826b-5c44a8b6f9f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Oct 30 13:03:53.304734 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:53.304699    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:03:54.242868 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:54.242794    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc"
Oct 30 13:03:54.254166 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:54.254139    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc"
Oct 30 13:03:54.482965 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:54.482929    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc"
Oct 30 13:03:55.465021 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:55.464995    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e34fd5-758756996-tl6mw"
Oct 30 13:03:56.295041 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:56.294690    4364 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:03:56.295041 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:56.294759    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7" podUID="134eb93c-b6ee-43e9-8977-950cb0188887" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:58.527759 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:58.527719    4364 generic.go:356] "Generic (PLEG): container finished" podID="134eb93c-b6ee-43e9-8977-950cb0188887" containerID="8109b3a388ae1a87ab9161858a90b4118be61694758679b68e06332870a8a139" exitCode=0
Oct 30 13:03:58.528244 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:58.527792    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7" event={"ID":"134eb93c-b6ee-43e9-8977-950cb0188887","Type":"ContainerDied","Data":"8109b3a388ae1a87ab9161858a90b4118be61694758679b68e06332870a8a139"}
Oct 30 13:03:58.528244 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:58.528223    4364 scope.go:122] "RemoveContainer" containerID="8109b3a388ae1a87ab9161858a90b4118be61694758679b68e06332870a8a139"
Oct 30 13:03:58.529683 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:58.529656    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:03:58.529779 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:58.529723    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:03:58.529779 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:58.529753    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:03:59.535732 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:59.535698    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7" event={"ID":"134eb93c-b6ee-43e9-8977-950cb0188887","Type":"ContainerStarted","Data":"97cbae683a7c99fa36188a38ee1fb80a181bdd1f0bffcfdf8f0a4e5dd8940371"}
Oct 30 13:03:59.602622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:59.601980    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7"
Oct 30 13:03:59.602622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:59.602101    4364 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Oct 30 13:03:59.602622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:03:59.602152    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7" podUID="134eb93c-b6ee-43e9-8977-950cb0188887" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Oct 30 13:04:00.603957 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:04:00.603873    4364 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Oct 30 13:04:00.603957 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:04:00.603920    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7" podUID="134eb93c-b6ee-43e9-8977-950cb0188887" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Oct 30 13:04:01.603919 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:04:01.603884    4364 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Oct 30 13:04:01.604093 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:04:01.603929    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7" podUID="134eb93c-b6ee-43e9-8977-950cb0188887" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Oct 30 13:04:02.604177 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:04:02.604138    4364 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Oct 30 13:04:02.604723 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:04:02.604542    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7" podUID="134eb93c-b6ee-43e9-8977-950cb0188887" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Oct 30 13:04:03.601368 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:04:03.600939    4364 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Oct 30 13:04:03.601368 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:04:03.600998    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7" podUID="134eb93c-b6ee-43e9-8977-950cb0188887" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Oct 30 13:04:04.568997 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:04:04.568964    4364 generic.go:356] "Generic (PLEG): container finished" podID="370ee2a9-2643-4e24-81e4-ab56f3caefb3" containerID="294b3d510361d2ce3da926630f9b2f86e2f8539a6c5aac27ae90f6e2b0b5b2bc" exitCode=0
Oct 30 13:04:04.569432 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:04:04.569036    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g" event={"ID":"370ee2a9-2643-4e24-81e4-ab56f3caefb3","Type":"ContainerDied","Data":"294b3d510361d2ce3da926630f9b2f86e2f8539a6c5aac27ae90f6e2b0b5b2bc"}
Oct 30 13:04:04.569432 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:04:04.569396    4364 scope.go:122] "RemoveContainer" containerID="294b3d510361d2ce3da926630f9b2f86e2f8539a6c5aac27ae90f6e2b0b5b2bc"
Oct 30 13:04:04.570726 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:04:04.570704    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:04:04.570792 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:04:04.570764    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:04:04.570833 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:04:04.570793    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:04:04.601199 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:04:04.601168    4364 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Oct 30 13:04:04.601341 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:04:04.601238    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7" podUID="134eb93c-b6ee-43e9-8977-950cb0188887" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Oct 30 13:04:05.577090 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:04:05.577059    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g" event={"ID":"370ee2a9-2643-4e24-81e4-ab56f3caefb3","Type":"ContainerStarted","Data":"8d6e9592a05c1a4f1a42b58fc2bb8fadb0c66eef23ed01c3bbb6e7b3b0fb2317"}
Oct 30 13:04:05.608156 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:04:05.608126    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7"
Oct 30 13:04:05.608436 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:04:05.608418    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7"
Oct 30 13:04:05.609193 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:04:05.609172    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7"
Oct 30 13:04:05.943666 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:04:05.943638    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g"
Oct 30 13:04:05.944668 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:04:05.944638    4364 patch_prober.go:29] interesting pod/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 30 13:04:05.944853 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:04:05.944687    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g" podUID="370ee2a9-2643-4e24-81e4-ab56f3caefb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 30 13:04:06.314639 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:04:06.314543    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:04:06.944023 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:04:06.943989    4364 patch_prober.go:29] interesting pod/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 30 13:04:06.944552 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:04:06.944054    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g" podUID="370ee2a9-2643-4e24-81e4-ab56f3caefb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 30 13:04:07.944482 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:04:07.944452    4364 patch_prober.go:29] interesting pod/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 30 13:04:07.944938 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:04:07.944677    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g" podUID="370ee2a9-2643-4e24-81e4-ab56f3caefb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 30 13:04:08.944393 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:04:08.944356    4364 patch_prober.go:29] interesting pod/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 30 13:04:08.944584 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:04:08.944417    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g" podUID="370ee2a9-2643-4e24-81e4-ab56f3caefb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 30 13:04:09.943835 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:04:09.943799    4364 patch_prober.go:29] interesting pod/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 30 13:04:09.944004 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:04:09.943851    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g" podUID="370ee2a9-2643-4e24-81e4-ab56f3caefb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 30 13:04:10.247661 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:04:10.247558    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g"
Oct 30 13:04:10.948499 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:04:10.948471    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g"
Oct 30 13:04:10.949430 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:04:10.949409    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g"
Oct 30 13:04:11.507544 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:04:11.507508    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:04:20.411865 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:04:20.411832    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:04:37.227708 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:04:37.227680    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:04:41.503717 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:04:41.503680    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:04:53.568318 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:04:53.568287    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:04:58.560901 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:04:58.560868    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6"]
Oct 30 13:04:58.564395 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:04:58.564377    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6"
Oct 30 13:04:58.584301 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:04:58.584275    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6"]
Oct 30 13:04:58.595792 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:04:58.595762    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-d833bd\"/\"openshift-service-ca.crt\""
Oct 30 13:04:58.595909 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:04:58.595870    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-d833bd\"/\"kube-root-ca.crt\""
Oct 30 13:04:58.595909 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:04:58.595889    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-d833bd\"/\"default-dockercfg-xwlf5\""
Oct 30 13:04:58.597400 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:04:58.597384    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-d833bd\"/\"premium-w3-d833bd-kserve-self-signed-certs\""
Oct 30 13:04:58.618229 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:04:58.618206    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwmvd\" (UniqueName: \"kubernetes.io/projected/b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1-kube-api-access-mwmvd\") pod \"premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6\" (UID: \"b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1\") " pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6"
Oct 30 13:04:58.618328 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:04:58.618274    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1-tls-certs\") pod \"premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6\" (UID: \"b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1\") " pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6"
Oct 30 13:04:58.618377 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:04:58.618354    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1-model-cache\") pod \"premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6\" (UID: \"b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1\") " pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6"
Oct 30 13:04:58.618429 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:04:58.618392    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1-dshm\") pod \"premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6\" (UID: \"b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1\") " pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6"
Oct 30 13:04:58.618481 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:04:58.618428    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1-tmp-dir\") pod \"premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6\" (UID: \"b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1\") " pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6"
Oct 30 13:04:58.618530 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:04:58.618480    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1-home\") pod \"premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6\" (UID: \"b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1\") " pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6"
Oct 30 13:04:58.719065 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:04:58.719034    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1-tls-certs\") pod \"premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6\" (UID: \"b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1\") " pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6"
Oct 30 13:04:58.719227 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:04:58.719075    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1-model-cache\") pod \"premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6\" (UID: \"b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1\") " pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6"
Oct 30 13:04:58.719227 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:04:58.719196    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1-dshm\") pod \"premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6\" (UID: \"b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1\") " pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6"
Oct 30 13:04:58.719350 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:04:58.719247    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1-tmp-dir\") pod \"premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6\" (UID: \"b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1\") " pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6"
Oct 30 13:04:58.719350 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:04:58.719275    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1-home\") pod \"premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6\" (UID: \"b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1\") " pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6"
Oct 30 13:04:58.719456 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:04:58.719373    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwmvd\" (UniqueName: \"kubernetes.io/projected/b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1-kube-api-access-mwmvd\") pod \"premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6\" (UID: \"b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1\") " pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6"
Oct 30 13:04:58.719518 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:04:58.719480    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1-model-cache\") pod \"premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6\" (UID: \"b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1\") " pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6"
Oct 30 13:04:58.719661 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:04:58.719627    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1-tmp-dir\") pod \"premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6\" (UID: \"b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1\") " pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6"
Oct 30 13:04:58.719736 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:04:58.719664    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1-home\") pod \"premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6\" (UID: \"b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1\") " pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6"
Oct 30 13:04:58.721557 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:04:58.721534    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1-dshm\") pod \"premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6\" (UID: \"b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1\") " pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6"
Oct 30 13:04:58.721724 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:04:58.721705    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1-tls-certs\") pod \"premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6\" (UID: \"b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1\") " pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6"
Oct 30 13:04:58.746849 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:04:58.746813    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwmvd\" (UniqueName: \"kubernetes.io/projected/b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1-kube-api-access-mwmvd\") pod \"premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6\" (UID: \"b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1\") " pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6"
Oct 30 13:04:58.875939 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:04:58.875912    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6"
Oct 30 13:04:59.180745 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:04:59.180721    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6"]
Oct 30 13:04:59.182406 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:04:59.182381    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb2ec47d4_90ed_4752_88b9_f5db4d4ee7e1.slice/crio-361b120a705036f644c9a4c0a479b132c2f22482dfb5459f0a55f6a82f668558 WatchSource:0}: Error finding container 361b120a705036f644c9a4c0a479b132c2f22482dfb5459f0a55f6a82f668558: Status 404 returned error can't find the container with id 361b120a705036f644c9a4c0a479b132c2f22482dfb5459f0a55f6a82f668558
Oct 30 13:04:59.957017 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:04:59.956986    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6" event={"ID":"b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1","Type":"ContainerStarted","Data":"b5f58c075b9206df95f27d39002a830a5366277a39068e8bdac039676de7a398"}
Oct 30 13:04:59.957017 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:04:59.957024    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6" event={"ID":"b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1","Type":"ContainerStarted","Data":"361b120a705036f644c9a4c0a479b132c2f22482dfb5459f0a55f6a82f668558"}
Oct 30 13:05:00.004112 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:00.004062    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6" podStartSLOduration=2.004048351 podStartE2EDuration="2.004048351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:04:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:00.003232212 +0000 UTC m=+799.169596861" watchObservedRunningTime="2026-10-30 13:05:00.004048351 +0000 UTC m=+799.170412998"
Oct 30 13:05:00.876488 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:00.876460    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6"
Oct 30 13:05:07.106221 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:07.106190    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26"]
Oct 30 13:05:07.109838 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:07.109818    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26"
Oct 30 13:05:07.146124 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:07.146093    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-52a649-openshift-default-dockercfg-frxr8\""
Oct 30 13:05:07.148734 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:07.148709    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26"]
Oct 30 13:05:07.209707 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:07.209672    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4b120bf-6e2b-46b5-bda8-76f8963697fb-istio-podinfo\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26\" (UID: \"c4b120bf-6e2b-46b5-bda8-76f8963697fb\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26"
Oct 30 13:05:07.209879 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:07.209730    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4b120bf-6e2b-46b5-bda8-76f8963697fb-credential-socket\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26\" (UID: \"c4b120bf-6e2b-46b5-bda8-76f8963697fb\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26"
Oct 30 13:05:07.209879 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:07.209842    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4b120bf-6e2b-46b5-bda8-76f8963697fb-istio-token\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26\" (UID: \"c4b120bf-6e2b-46b5-bda8-76f8963697fb\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26"
Oct 30 13:05:07.209879 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:07.209873    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4b120bf-6e2b-46b5-bda8-76f8963697fb-istiod-ca-cert\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26\" (UID: \"c4b120bf-6e2b-46b5-bda8-76f8963697fb\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26"
Oct 30 13:05:07.209981 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:07.209942    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4b120bf-6e2b-46b5-bda8-76f8963697fb-workload-certs\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26\" (UID: \"c4b120bf-6e2b-46b5-bda8-76f8963697fb\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26"
Oct 30 13:05:07.210015 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:07.209983    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8blw8\" (UniqueName: \"kubernetes.io/projected/c4b120bf-6e2b-46b5-bda8-76f8963697fb-kube-api-access-8blw8\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26\" (UID: \"c4b120bf-6e2b-46b5-bda8-76f8963697fb\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26"
Oct 30 13:05:07.210054 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:07.210020    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4b120bf-6e2b-46b5-bda8-76f8963697fb-workload-socket\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26\" (UID: \"c4b120bf-6e2b-46b5-bda8-76f8963697fb\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26"
Oct 30 13:05:07.210126 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:07.210105    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4b120bf-6e2b-46b5-bda8-76f8963697fb-istio-envoy\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26\" (UID: \"c4b120bf-6e2b-46b5-bda8-76f8963697fb\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26"
Oct 30 13:05:07.210207 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:07.210185    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4b120bf-6e2b-46b5-bda8-76f8963697fb-istio-data\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26\" (UID: \"c4b120bf-6e2b-46b5-bda8-76f8963697fb\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26"
Oct 30 13:05:07.311557 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:07.311525    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8blw8\" (UniqueName: \"kubernetes.io/projected/c4b120bf-6e2b-46b5-bda8-76f8963697fb-kube-api-access-8blw8\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26\" (UID: \"c4b120bf-6e2b-46b5-bda8-76f8963697fb\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26"
Oct 30 13:05:07.311741 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:07.311564    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4b120bf-6e2b-46b5-bda8-76f8963697fb-workload-socket\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26\" (UID: \"c4b120bf-6e2b-46b5-bda8-76f8963697fb\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26"
Oct 30 13:05:07.311741 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:07.311587    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4b120bf-6e2b-46b5-bda8-76f8963697fb-istio-envoy\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26\" (UID: \"c4b120bf-6e2b-46b5-bda8-76f8963697fb\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26"
Oct 30 13:05:07.311741 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:07.311650    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4b120bf-6e2b-46b5-bda8-76f8963697fb-istio-data\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26\" (UID: \"c4b120bf-6e2b-46b5-bda8-76f8963697fb\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26"
Oct 30 13:05:07.311854 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:07.311805    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4b120bf-6e2b-46b5-bda8-76f8963697fb-istio-podinfo\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26\" (UID: \"c4b120bf-6e2b-46b5-bda8-76f8963697fb\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26"
Oct 30 13:05:07.311905 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:07.311891    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4b120bf-6e2b-46b5-bda8-76f8963697fb-credential-socket\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26\" (UID: \"c4b120bf-6e2b-46b5-bda8-76f8963697fb\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26"
Oct 30 13:05:07.311961 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:07.311947    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4b120bf-6e2b-46b5-bda8-76f8963697fb-istio-token\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26\" (UID: \"c4b120bf-6e2b-46b5-bda8-76f8963697fb\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26"
Oct 30 13:05:07.312014 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:07.311977    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4b120bf-6e2b-46b5-bda8-76f8963697fb-istiod-ca-cert\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26\" (UID: \"c4b120bf-6e2b-46b5-bda8-76f8963697fb\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26"
Oct 30 13:05:07.312067 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:07.312027    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4b120bf-6e2b-46b5-bda8-76f8963697fb-workload-certs\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26\" (UID: \"c4b120bf-6e2b-46b5-bda8-76f8963697fb\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26"
Oct 30 13:05:07.312140 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:07.312111    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4b120bf-6e2b-46b5-bda8-76f8963697fb-istio-data\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26\" (UID: \"c4b120bf-6e2b-46b5-bda8-76f8963697fb\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26"
Oct 30 13:05:07.312203 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:07.312122    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4b120bf-6e2b-46b5-bda8-76f8963697fb-workload-socket\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26\" (UID: \"c4b120bf-6e2b-46b5-bda8-76f8963697fb\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26"
Oct 30 13:05:07.312439 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:07.312411    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4b120bf-6e2b-46b5-bda8-76f8963697fb-workload-certs\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26\" (UID: \"c4b120bf-6e2b-46b5-bda8-76f8963697fb\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26"
Oct 30 13:05:07.312564 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:07.312449    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4b120bf-6e2b-46b5-bda8-76f8963697fb-credential-socket\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26\" (UID: \"c4b120bf-6e2b-46b5-bda8-76f8963697fb\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26"
Oct 30 13:05:07.312758 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:07.312735    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4b120bf-6e2b-46b5-bda8-76f8963697fb-istiod-ca-cert\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26\" (UID: \"c4b120bf-6e2b-46b5-bda8-76f8963697fb\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26"
Oct 30 13:05:07.313985 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:07.313962    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4b120bf-6e2b-46b5-bda8-76f8963697fb-istio-envoy\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26\" (UID: \"c4b120bf-6e2b-46b5-bda8-76f8963697fb\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26"
Oct 30 13:05:07.314385 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:07.314363    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4b120bf-6e2b-46b5-bda8-76f8963697fb-istio-podinfo\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26\" (UID: \"c4b120bf-6e2b-46b5-bda8-76f8963697fb\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26"
Oct 30 13:05:07.336517 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:07.336489    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4b120bf-6e2b-46b5-bda8-76f8963697fb-istio-token\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26\" (UID: \"c4b120bf-6e2b-46b5-bda8-76f8963697fb\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26"
Oct 30 13:05:07.336815 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:07.336797    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8blw8\" (UniqueName: \"kubernetes.io/projected/c4b120bf-6e2b-46b5-bda8-76f8963697fb-kube-api-access-8blw8\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26\" (UID: \"c4b120bf-6e2b-46b5-bda8-76f8963697fb\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26"
Oct 30 13:05:07.422510 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:07.422484    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26"
Oct 30 13:05:07.742705 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:07.742680    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26"]
Oct 30 13:05:07.743371 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:05:07.743350    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4b120bf_6e2b_46b5_bda8_76f8963697fb.slice/crio-defb23c122c278136caa21b1f2a2a997c4c4d2bf64e806f6ba65be84d20224ff WatchSource:0}: Error finding container defb23c122c278136caa21b1f2a2a997c4c4d2bf64e806f6ba65be84d20224ff: Status 404 returned error can't find the container with id defb23c122c278136caa21b1f2a2a997c4c4d2bf64e806f6ba65be84d20224ff
Oct 30 13:05:07.745647 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:07.745619    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:05:07.745725 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:07.745680    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:05:07.745725 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:07.745711    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:05:08.007948 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:08.007845    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26" event={"ID":"c4b120bf-6e2b-46b5-bda8-76f8963697fb","Type":"ContainerStarted","Data":"91e066f39e1a5e9bb338794da9c823c80ea37572f64bd602be992496a3b56b29"}
Oct 30 13:05:08.007948 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:08.007916    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26" event={"ID":"c4b120bf-6e2b-46b5-bda8-76f8963697fb","Type":"ContainerStarted","Data":"defb23c122c278136caa21b1f2a2a997c4c4d2bf64e806f6ba65be84d20224ff"}
Oct 30 13:05:08.422854 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:08.422816    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26"
Oct 30 13:05:08.424565 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:08.424541    4364 patch_prober.go:29] interesting pod/e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 13:05:08.424721 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:08.424630    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26" podUID="c4b120bf-6e2b-46b5-bda8-76f8963697fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 13:05:08.876460 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:08.876429    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6"
Oct 30 13:05:08.912654 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:08.912624    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6"
Oct 30 13:05:08.960943 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:08.960893    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26" podStartSLOduration=1.960879663 podStartE2EDuration="1.960879663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:05:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:08.06050908 +0000 UTC m=+807.226873729" watchObservedRunningTime="2026-10-30 13:05:08.960879663 +0000 UTC m=+808.127244339"
Oct 30 13:05:09.026094 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:09.026058    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6"
Oct 30 13:05:09.439622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:09.425813    4364 patch_prober.go:29] interesting pod/e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 13:05:09.439622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:09.425968    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26" podUID="c4b120bf-6e2b-46b5-bda8-76f8963697fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 13:05:10.423080 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:10.423045    4364 patch_prober.go:29] interesting pod/e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 13:05:10.423254 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:10.423101    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26" podUID="c4b120bf-6e2b-46b5-bda8-76f8963697fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 13:05:11.423074 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:11.423040    4364 patch_prober.go:29] interesting pod/e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 13:05:11.423447 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:11.423091    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26" podUID="c4b120bf-6e2b-46b5-bda8-76f8963697fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 13:05:11.527991 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:05:11.527945    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:05:12.378527 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:12.378495    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:05:12.423416 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:12.423382    4364 patch_prober.go:29] interesting pod/e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 13:05:12.423826 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:12.423432    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26" podUID="c4b120bf-6e2b-46b5-bda8-76f8963697fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 13:05:13.423020 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:13.422989    4364 patch_prober.go:29] interesting pod/e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 13:05:13.423198 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:13.423054    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26" podUID="c4b120bf-6e2b-46b5-bda8-76f8963697fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 13:05:15.156123 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:15.156090    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26"
Oct 30 13:05:15.156650 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:15.156461    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26"
Oct 30 13:05:15.157691 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:15.157663    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26"
Oct 30 13:05:23.045618 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:23.045025    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f"]
Oct 30 13:05:23.057140 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:23.055885    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f"
Oct 30 13:05:23.075670 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:23.075577    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f"]
Oct 30 13:05:23.091727 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:23.091703    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-f07dfe-openshift-default-dockercfg-d79w9\""
Oct 30 13:05:23.141273 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:23.141230    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/086afc53-5c53-42d9-bd7b-b803cc41ed6d-istio-token\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f\" (UID: \"086afc53-5c53-42d9-bd7b-b803cc41ed6d\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f"
Oct 30 13:05:23.141659 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:23.141635    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/086afc53-5c53-42d9-bd7b-b803cc41ed6d-istio-data\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f\" (UID: \"086afc53-5c53-42d9-bd7b-b803cc41ed6d\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f"
Oct 30 13:05:23.142789 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:23.142765    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/086afc53-5c53-42d9-bd7b-b803cc41ed6d-istio-envoy\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f\" (UID: \"086afc53-5c53-42d9-bd7b-b803cc41ed6d\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f"
Oct 30 13:05:23.142998 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:23.142978    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/086afc53-5c53-42d9-bd7b-b803cc41ed6d-istio-podinfo\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f\" (UID: \"086afc53-5c53-42d9-bd7b-b803cc41ed6d\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f"
Oct 30 13:05:23.143191 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:23.143171    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/086afc53-5c53-42d9-bd7b-b803cc41ed6d-workload-socket\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f\" (UID: \"086afc53-5c53-42d9-bd7b-b803cc41ed6d\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f"
Oct 30 13:05:23.143398 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:23.143380    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/086afc53-5c53-42d9-bd7b-b803cc41ed6d-credential-socket\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f\" (UID: \"086afc53-5c53-42d9-bd7b-b803cc41ed6d\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f"
Oct 30 13:05:23.143561 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:23.143541    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/086afc53-5c53-42d9-bd7b-b803cc41ed6d-istiod-ca-cert\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f\" (UID: \"086afc53-5c53-42d9-bd7b-b803cc41ed6d\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f"
Oct 30 13:05:23.143760 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:23.143743    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z5wr\" (UniqueName: \"kubernetes.io/projected/086afc53-5c53-42d9-bd7b-b803cc41ed6d-kube-api-access-5z5wr\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f\" (UID: \"086afc53-5c53-42d9-bd7b-b803cc41ed6d\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f"
Oct 30 13:05:23.143975 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:23.143956    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/086afc53-5c53-42d9-bd7b-b803cc41ed6d-workload-certs\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f\" (UID: \"086afc53-5c53-42d9-bd7b-b803cc41ed6d\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f"
Oct 30 13:05:23.245437 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:23.245405    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/086afc53-5c53-42d9-bd7b-b803cc41ed6d-workload-socket\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f\" (UID: \"086afc53-5c53-42d9-bd7b-b803cc41ed6d\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f"
Oct 30 13:05:23.246266 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:23.246236    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/086afc53-5c53-42d9-bd7b-b803cc41ed6d-credential-socket\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f\" (UID: \"086afc53-5c53-42d9-bd7b-b803cc41ed6d\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f"
Oct 30 13:05:23.246565 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:23.246538    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/086afc53-5c53-42d9-bd7b-b803cc41ed6d-istiod-ca-cert\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f\" (UID: \"086afc53-5c53-42d9-bd7b-b803cc41ed6d\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f"
Oct 30 13:05:23.246837 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:23.246806    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z5wr\" (UniqueName: \"kubernetes.io/projected/086afc53-5c53-42d9-bd7b-b803cc41ed6d-kube-api-access-5z5wr\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f\" (UID: \"086afc53-5c53-42d9-bd7b-b803cc41ed6d\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f"
Oct 30 13:05:23.247062 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:23.247042    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/086afc53-5c53-42d9-bd7b-b803cc41ed6d-workload-certs\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f\" (UID: \"086afc53-5c53-42d9-bd7b-b803cc41ed6d\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f"
Oct 30 13:05:23.247209 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:23.247192    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/086afc53-5c53-42d9-bd7b-b803cc41ed6d-istio-token\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f\" (UID: \"086afc53-5c53-42d9-bd7b-b803cc41ed6d\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f"
Oct 30 13:05:23.247358 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:23.247341    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/086afc53-5c53-42d9-bd7b-b803cc41ed6d-istio-data\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f\" (UID: \"086afc53-5c53-42d9-bd7b-b803cc41ed6d\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f"
Oct 30 13:05:23.247490 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:23.247472    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/086afc53-5c53-42d9-bd7b-b803cc41ed6d-istio-envoy\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f\" (UID: \"086afc53-5c53-42d9-bd7b-b803cc41ed6d\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f"
Oct 30 13:05:23.247649 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:23.247630    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/086afc53-5c53-42d9-bd7b-b803cc41ed6d-istio-podinfo\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f\" (UID: \"086afc53-5c53-42d9-bd7b-b803cc41ed6d\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f"
Oct 30 13:05:23.250682 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:23.250008    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/086afc53-5c53-42d9-bd7b-b803cc41ed6d-credential-socket\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f\" (UID: \"086afc53-5c53-42d9-bd7b-b803cc41ed6d\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f"
Oct 30 13:05:23.252474 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:23.251080    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/086afc53-5c53-42d9-bd7b-b803cc41ed6d-istiod-ca-cert\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f\" (UID: \"086afc53-5c53-42d9-bd7b-b803cc41ed6d\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f"
Oct 30 13:05:23.252474 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:23.251678    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/086afc53-5c53-42d9-bd7b-b803cc41ed6d-workload-certs\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f\" (UID: \"086afc53-5c53-42d9-bd7b-b803cc41ed6d\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f"
Oct 30 13:05:23.252474 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:23.252202    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/086afc53-5c53-42d9-bd7b-b803cc41ed6d-istio-data\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f\" (UID: \"086afc53-5c53-42d9-bd7b-b803cc41ed6d\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f"
Oct 30 13:05:23.252760 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:23.252737    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/086afc53-5c53-42d9-bd7b-b803cc41ed6d-workload-socket\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f\" (UID: \"086afc53-5c53-42d9-bd7b-b803cc41ed6d\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f"
Oct 30 13:05:23.253072 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:23.253043    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/086afc53-5c53-42d9-bd7b-b803cc41ed6d-istio-podinfo\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f\" (UID: \"086afc53-5c53-42d9-bd7b-b803cc41ed6d\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f"
Oct 30 13:05:23.253299 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:23.253274    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t"]
Oct 30 13:05:23.255365 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:23.255318    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/086afc53-5c53-42d9-bd7b-b803cc41ed6d-istio-envoy\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f\" (UID: \"086afc53-5c53-42d9-bd7b-b803cc41ed6d\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f"
Oct 30 13:05:23.257342 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:23.257320    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t"
Oct 30 13:05:23.280211 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:23.279649    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-d833bd-openshift-default-dockercfg-dpcql\""
Oct 30 13:05:23.280211 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:23.279989    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/086afc53-5c53-42d9-bd7b-b803cc41ed6d-istio-token\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f\" (UID: \"086afc53-5c53-42d9-bd7b-b803cc41ed6d\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f"
Oct 30 13:05:23.282133 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:23.282110    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z5wr\" (UniqueName: \"kubernetes.io/projected/086afc53-5c53-42d9-bd7b-b803cc41ed6d-kube-api-access-5z5wr\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f\" (UID: \"086afc53-5c53-42d9-bd7b-b803cc41ed6d\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f"
Oct 30 13:05:23.289484 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:23.289460    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t"]
Oct 30 13:05:23.349123 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:23.349041    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58073475-3ca6-4a52-b021-e08e935fad18-istiod-ca-cert\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t\" (UID: \"58073475-3ca6-4a52-b021-e08e935fad18\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t"
Oct 30 13:05:23.349123 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:23.349090    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58073475-3ca6-4a52-b021-e08e935fad18-istio-token\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t\" (UID: \"58073475-3ca6-4a52-b021-e08e935fad18\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t"
Oct 30 13:05:23.349338 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:23.349129    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58073475-3ca6-4a52-b021-e08e935fad18-istio-podinfo\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t\" (UID: \"58073475-3ca6-4a52-b021-e08e935fad18\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t"
Oct 30 13:05:23.349338 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:23.349248    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5vr6\" (UniqueName: \"kubernetes.io/projected/58073475-3ca6-4a52-b021-e08e935fad18-kube-api-access-z5vr6\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t\" (UID: \"58073475-3ca6-4a52-b021-e08e935fad18\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t"
Oct 30 13:05:23.349338 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:23.349320    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58073475-3ca6-4a52-b021-e08e935fad18-workload-socket\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t\" (UID: \"58073475-3ca6-4a52-b021-e08e935fad18\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t"
Oct 30 13:05:23.349470 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:23.349339    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58073475-3ca6-4a52-b021-e08e935fad18-istio-data\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t\" (UID: \"58073475-3ca6-4a52-b021-e08e935fad18\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t"
Oct 30 13:05:23.349470 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:23.349403    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58073475-3ca6-4a52-b021-e08e935fad18-credential-socket\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t\" (UID: \"58073475-3ca6-4a52-b021-e08e935fad18\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t"
Oct 30 13:05:23.349470 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:23.349459    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58073475-3ca6-4a52-b021-e08e935fad18-workload-certs\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t\" (UID: \"58073475-3ca6-4a52-b021-e08e935fad18\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t"
Oct 30 13:05:23.349646 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:23.349537    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58073475-3ca6-4a52-b021-e08e935fad18-istio-envoy\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t\" (UID: \"58073475-3ca6-4a52-b021-e08e935fad18\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t"
Oct 30 13:05:23.383915 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:23.383884    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f"
Oct 30 13:05:23.457628 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:23.453416    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58073475-3ca6-4a52-b021-e08e935fad18-workload-socket\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t\" (UID: \"58073475-3ca6-4a52-b021-e08e935fad18\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t"
Oct 30 13:05:23.457628 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:23.453464    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58073475-3ca6-4a52-b021-e08e935fad18-istio-data\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t\" (UID: \"58073475-3ca6-4a52-b021-e08e935fad18\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t"
Oct 30 13:05:23.457628 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:23.453499    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58073475-3ca6-4a52-b021-e08e935fad18-credential-socket\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t\" (UID: \"58073475-3ca6-4a52-b021-e08e935fad18\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t"
Oct 30 13:05:23.457628 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:23.453561    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58073475-3ca6-4a52-b021-e08e935fad18-workload-certs\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t\" (UID: \"58073475-3ca6-4a52-b021-e08e935fad18\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t"
Oct 30 13:05:23.457628 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:23.453615    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58073475-3ca6-4a52-b021-e08e935fad18-istio-envoy\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t\" (UID: \"58073475-3ca6-4a52-b021-e08e935fad18\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t"
Oct 30 13:05:23.457628 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:23.453726    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58073475-3ca6-4a52-b021-e08e935fad18-istiod-ca-cert\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t\" (UID: \"58073475-3ca6-4a52-b021-e08e935fad18\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t"
Oct 30 13:05:23.457628 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:23.453801    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58073475-3ca6-4a52-b021-e08e935fad18-istio-token\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t\" (UID: \"58073475-3ca6-4a52-b021-e08e935fad18\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t"
Oct 30 13:05:23.457628 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:23.453838    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58073475-3ca6-4a52-b021-e08e935fad18-istio-podinfo\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t\" (UID: \"58073475-3ca6-4a52-b021-e08e935fad18\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t"
Oct 30 13:05:23.457628 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:23.453881    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5vr6\" (UniqueName: \"kubernetes.io/projected/58073475-3ca6-4a52-b021-e08e935fad18-kube-api-access-z5vr6\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t\" (UID: \"58073475-3ca6-4a52-b021-e08e935fad18\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t"
Oct 30 13:05:23.457628 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:23.454691    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58073475-3ca6-4a52-b021-e08e935fad18-workload-socket\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t\" (UID: \"58073475-3ca6-4a52-b021-e08e935fad18\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t"
Oct 30 13:05:23.457628 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:23.454970    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58073475-3ca6-4a52-b021-e08e935fad18-istio-data\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t\" (UID: \"58073475-3ca6-4a52-b021-e08e935fad18\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t"
Oct 30 13:05:23.457628 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:23.455212    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58073475-3ca6-4a52-b021-e08e935fad18-credential-socket\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t\" (UID: \"58073475-3ca6-4a52-b021-e08e935fad18\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t"
Oct 30 13:05:23.457628 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:23.455462    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58073475-3ca6-4a52-b021-e08e935fad18-workload-certs\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t\" (UID: \"58073475-3ca6-4a52-b021-e08e935fad18\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t"
Oct 30 13:05:23.457628 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:23.456750    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58073475-3ca6-4a52-b021-e08e935fad18-istiod-ca-cert\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t\" (UID: \"58073475-3ca6-4a52-b021-e08e935fad18\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t"
Oct 30 13:05:23.460924 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:23.460896    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58073475-3ca6-4a52-b021-e08e935fad18-istio-envoy\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t\" (UID: \"58073475-3ca6-4a52-b021-e08e935fad18\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t"
Oct 30 13:05:23.462393 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:23.462367    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58073475-3ca6-4a52-b021-e08e935fad18-istio-podinfo\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t\" (UID: \"58073475-3ca6-4a52-b021-e08e935fad18\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t"
Oct 30 13:05:23.493823 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:23.493797    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5vr6\" (UniqueName: \"kubernetes.io/projected/58073475-3ca6-4a52-b021-e08e935fad18-kube-api-access-z5vr6\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t\" (UID: \"58073475-3ca6-4a52-b021-e08e935fad18\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t"
Oct 30 13:05:23.494161 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:23.494129    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58073475-3ca6-4a52-b021-e08e935fad18-istio-token\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t\" (UID: \"58073475-3ca6-4a52-b021-e08e935fad18\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t"
Oct 30 13:05:23.616172 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:23.616086    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t"
Oct 30 13:05:23.706375 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:23.705945    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f"]
Oct 30 13:05:23.709097 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:05:23.709072    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod086afc53_5c53_42d9_bd7b_b803cc41ed6d.slice/crio-c2e6b32bf747756b675e0662d9ba84a022c55e0630b2269ead7c3625a37b9a02 WatchSource:0}: Error finding container c2e6b32bf747756b675e0662d9ba84a022c55e0630b2269ead7c3625a37b9a02: Status 404 returned error can't find the container with id c2e6b32bf747756b675e0662d9ba84a022c55e0630b2269ead7c3625a37b9a02
Oct 30 13:05:23.714241 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:23.714186    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:05:23.714355 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:23.714288    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:05:23.714355 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:23.714338    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:05:23.938234 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:23.938200    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t"]
Oct 30 13:05:23.938381 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:05:23.938279    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58073475_3ca6_4a52_b021_e08e935fad18.slice/crio-a3408f93dd68fda42aeb89a242467ccb7a421eb3e33e5b52eaf89f8eff2e2323 WatchSource:0}: Error finding container a3408f93dd68fda42aeb89a242467ccb7a421eb3e33e5b52eaf89f8eff2e2323: Status 404 returned error can't find the container with id a3408f93dd68fda42aeb89a242467ccb7a421eb3e33e5b52eaf89f8eff2e2323
Oct 30 13:05:23.941680 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:23.941644    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:05:23.941772 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:23.941727    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:05:23.941810 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:23.941774    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:05:24.162376 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:24.162332    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t" event={"ID":"58073475-3ca6-4a52-b021-e08e935fad18","Type":"ContainerStarted","Data":"259a6384f1c83dd7ebaa0c73d9c969906b653928ebce6df6b1842fc2ce0338bf"}
Oct 30 13:05:24.162887 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:24.162396    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t" event={"ID":"58073475-3ca6-4a52-b021-e08e935fad18","Type":"ContainerStarted","Data":"a3408f93dd68fda42aeb89a242467ccb7a421eb3e33e5b52eaf89f8eff2e2323"}
Oct 30 13:05:24.164874 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:24.164850    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f" event={"ID":"086afc53-5c53-42d9-bd7b-b803cc41ed6d","Type":"ContainerStarted","Data":"ffdeb9551c8182040b980864acb868a5d663b60720f0ecc8d9a610cecf1b979f"}
Oct 30 13:05:24.165010 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:24.164886    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f" event={"ID":"086afc53-5c53-42d9-bd7b-b803cc41ed6d","Type":"ContainerStarted","Data":"c2e6b32bf747756b675e0662d9ba84a022c55e0630b2269ead7c3625a37b9a02"}
Oct 30 13:05:24.213322 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:24.213233    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t" podStartSLOduration=1.213220094 podStartE2EDuration="1.213220094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:05:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:24.211592824 +0000 UTC m=+823.377957472" watchObservedRunningTime="2026-10-30 13:05:24.213220094 +0000 UTC m=+823.379584742"
Oct 30 13:05:24.263646 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:24.263555    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f" podStartSLOduration=2.2635385 podStartE2EDuration="2.2635385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:05:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:24.260935912 +0000 UTC m=+823.427300561" watchObservedRunningTime="2026-10-30 13:05:24.2635385 +0000 UTC m=+823.429903149"
Oct 30 13:05:24.385044 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:24.385018    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f"
Oct 30 13:05:24.386439 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:24.386416    4364 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 13:05:24.386539 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:24.386458    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f" podUID="086afc53-5c53-42d9-bd7b-b803cc41ed6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 13:05:24.621078 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:24.621048    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t"
Oct 30 13:05:24.630772 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:24.630738    4364 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 13:05:24.631048 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:24.631019    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t" podUID="58073475-3ca6-4a52-b021-e08e935fad18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 13:05:25.388054 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:25.388019    4364 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 13:05:25.388700 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:25.388659    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f" podUID="086afc53-5c53-42d9-bd7b-b803cc41ed6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 13:05:25.625623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:25.624943    4364 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 13:05:25.625623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:25.625020    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t" podUID="58073475-3ca6-4a52-b021-e08e935fad18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 13:05:26.387623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:26.386947    4364 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 13:05:26.387623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:26.387007    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f" podUID="086afc53-5c53-42d9-bd7b-b803cc41ed6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 13:05:26.628625 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:26.626741    4364 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 13:05:26.628625 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:26.626811    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t" podUID="58073475-3ca6-4a52-b021-e08e935fad18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 13:05:27.395938 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:27.395899    4364 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 13:05:27.396200 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:27.396171    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f" podUID="086afc53-5c53-42d9-bd7b-b803cc41ed6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 13:05:27.628287 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:27.628249    4364 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 13:05:27.628740 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:27.628682    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t" podUID="58073475-3ca6-4a52-b021-e08e935fad18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 13:05:28.390624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:28.387901    4364 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 13:05:28.390624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:28.387962    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f" podUID="086afc53-5c53-42d9-bd7b-b803cc41ed6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 13:05:28.620624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:28.618733    4364 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 13:05:28.620624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:28.618797    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t" podUID="58073475-3ca6-4a52-b021-e08e935fad18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 13:05:29.390623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:29.386937    4364 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 13:05:29.390623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:29.386999    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f" podUID="086afc53-5c53-42d9-bd7b-b803cc41ed6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 13:05:29.623622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:29.621916    4364 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 13:05:29.623622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:29.621975    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t" podUID="58073475-3ca6-4a52-b021-e08e935fad18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 13:05:30.045930 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:30.045889    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec81d790-rl9dp"]
Oct 30 13:05:30.050778 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:30.050720    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec81d790-rl9dp"
Oct 30 13:05:30.066522 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:30.066499    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec81d790-rl9dp"]
Oct 30 13:05:30.077557 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:30.077531    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:05:30.077701 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:30.077570    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\""
Oct 30 13:05:30.151941 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:30.151903    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ee15a09-210f-4eea-b962-6af016052d6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec81d790-rl9dp\" (UID: \"5ee15a09-210f-4eea-b962-6af016052d6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec81d790-rl9dp"
Oct 30 13:05:30.152106 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:30.152030    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g9l4\" (UniqueName: \"kubernetes.io/projected/5ee15a09-210f-4eea-b962-6af016052d6c-kube-api-access-8g9l4\") pod \"maas-api-revoke-sub-ec81d790-rl9dp\" (UID: \"5ee15a09-210f-4eea-b962-6af016052d6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec81d790-rl9dp"
Oct 30 13:05:30.152246 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:30.152223    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ee15a09-210f-4eea-b962-6af016052d6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec81d790-rl9dp\" (UID: \"5ee15a09-210f-4eea-b962-6af016052d6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec81d790-rl9dp"
Oct 30 13:05:30.253684 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:30.253650    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ee15a09-210f-4eea-b962-6af016052d6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec81d790-rl9dp\" (UID: \"5ee15a09-210f-4eea-b962-6af016052d6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec81d790-rl9dp"
Oct 30 13:05:30.253850 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:30.253803    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g9l4\" (UniqueName: \"kubernetes.io/projected/5ee15a09-210f-4eea-b962-6af016052d6c-kube-api-access-8g9l4\") pod \"maas-api-revoke-sub-ec81d790-rl9dp\" (UID: \"5ee15a09-210f-4eea-b962-6af016052d6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec81d790-rl9dp"
Oct 30 13:05:30.253955 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:30.253933    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ee15a09-210f-4eea-b962-6af016052d6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec81d790-rl9dp\" (UID: \"5ee15a09-210f-4eea-b962-6af016052d6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec81d790-rl9dp"
Oct 30 13:05:30.254449 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:30.254428    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ee15a09-210f-4eea-b962-6af016052d6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec81d790-rl9dp\" (UID: \"5ee15a09-210f-4eea-b962-6af016052d6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec81d790-rl9dp"
Oct 30 13:05:30.256353 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:30.256327    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ee15a09-210f-4eea-b962-6af016052d6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec81d790-rl9dp\" (UID: \"5ee15a09-210f-4eea-b962-6af016052d6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec81d790-rl9dp"
Oct 30 13:05:30.279884 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:30.279855    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g9l4\" (UniqueName: \"kubernetes.io/projected/5ee15a09-210f-4eea-b962-6af016052d6c-kube-api-access-8g9l4\") pod \"maas-api-revoke-sub-ec81d790-rl9dp\" (UID: \"5ee15a09-210f-4eea-b962-6af016052d6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec81d790-rl9dp"
Oct 30 13:05:30.365450 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:30.365360    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec81d790-rl9dp"
Oct 30 13:05:30.384871 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:30.384841    4364 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 13:05:30.385009 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:30.384902    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f" podUID="086afc53-5c53-42d9-bd7b-b803cc41ed6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 13:05:30.675368 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:30.675292    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec81d790-rl9dp"]
Oct 30 13:05:30.675368 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:05:30.675328    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ee15a09_210f_4eea_b962_6af016052d6c.slice/crio-168ebbe068ee781ac548974fda22026b75d2fced41aad3acd0ae46a09e636e82 WatchSource:0}: Error finding container 168ebbe068ee781ac548974fda22026b75d2fced41aad3acd0ae46a09e636e82: Status 404 returned error can't find the container with id 168ebbe068ee781ac548974fda22026b75d2fced41aad3acd0ae46a09e636e82
Oct 30 13:05:30.885227 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:30.885190    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t"
Oct 30 13:05:30.885827 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:30.885805    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t"
Oct 30 13:05:30.886695 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:30.886674    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t"
Oct 30 13:05:31.268334 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:31.268303    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec81d790-rl9dp" event={"ID":"5ee15a09-210f-4eea-b962-6af016052d6c","Type":"ContainerStarted","Data":"168ebbe068ee781ac548974fda22026b75d2fced41aad3acd0ae46a09e636e82"}
Oct 30 13:05:31.390016 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:31.389990    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f"
Oct 30 13:05:31.390720 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:31.390461    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f"
Oct 30 13:05:31.391308 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:31.391289    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f"
Oct 30 13:05:33.285072 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:33.285012    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec81d790-rl9dp" event={"ID":"5ee15a09-210f-4eea-b962-6af016052d6c","Type":"ContainerStarted","Data":"03208f80ce282ec29e13fb9236786533688507fd6b6da3fd7f82e744f0b6947d"}
Oct 30 13:05:33.335780 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:33.335731    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec81d790-rl9dp" podStartSLOduration=1.368823207 podStartE2EDuration="3.335718505s" totalImagesPullingTime="1.966895298s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:05:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:33.33449429 +0000 UTC m=+832.500858938" watchObservedRunningTime="2026-10-30 13:05:33.335718505 +0000 UTC m=+832.502083152"
Oct 30 13:05:34.292945 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:34.292909    4364 generic.go:356] "Generic (PLEG): container finished" podID="5ee15a09-210f-4eea-b962-6af016052d6c" containerID="03208f80ce282ec29e13fb9236786533688507fd6b6da3fd7f82e744f0b6947d" exitCode=0
Oct 30 13:05:34.293308 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:34.292987    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec81d790-rl9dp" event={"ID":"5ee15a09-210f-4eea-b962-6af016052d6c","Type":"ContainerDied","Data":"03208f80ce282ec29e13fb9236786533688507fd6b6da3fd7f82e744f0b6947d"}
Oct 30 13:05:35.222483 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:35.222451    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:05:35.458195 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:35.458170    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec81d790-rl9dp"
Oct 30 13:05:35.512385 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:35.512317    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ee15a09-210f-4eea-b962-6af016052d6c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ee15a09-210f-4eea-b962-6af016052d6c-maas-api-openshift-ca\") pod \"5ee15a09-210f-4eea-b962-6af016052d6c\" (UID: \"5ee15a09-210f-4eea-b962-6af016052d6c\") "
Oct 30 13:05:35.512539 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:35.512395    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ee15a09-210f-4eea-b962-6af016052d6c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ee15a09-210f-4eea-b962-6af016052d6c-maas-api-xks-ca\") pod \"5ee15a09-210f-4eea-b962-6af016052d6c\" (UID: \"5ee15a09-210f-4eea-b962-6af016052d6c\") "
Oct 30 13:05:35.512627 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:35.512553    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ee15a09-210f-4eea-b962-6af016052d6c-kube-api-access-8g9l4\" (UniqueName: \"kubernetes.io/projected/5ee15a09-210f-4eea-b962-6af016052d6c-kube-api-access-8g9l4\") pod \"5ee15a09-210f-4eea-b962-6af016052d6c\" (UID: \"5ee15a09-210f-4eea-b962-6af016052d6c\") "
Oct 30 13:05:35.512747 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:35.512717    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ee15a09-210f-4eea-b962-6af016052d6c-maas-api-openshift-ca" pod "5ee15a09-210f-4eea-b962-6af016052d6c" (UID: "5ee15a09-210f-4eea-b962-6af016052d6c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:05:35.512990 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:35.512968    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ee15a09-210f-4eea-b962-6af016052d6c-maas-api-openshift-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:05:35.515100 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:35.515069    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ee15a09-210f-4eea-b962-6af016052d6c-kube-api-access-8g9l4" pod "5ee15a09-210f-4eea-b962-6af016052d6c" (UID: "5ee15a09-210f-4eea-b962-6af016052d6c"). InnerVolumeSpecName "kube-api-access-8g9l4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:05:35.515200 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:35.515073    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ee15a09-210f-4eea-b962-6af016052d6c-maas-api-xks-ca" pod "5ee15a09-210f-4eea-b962-6af016052d6c" (UID: "5ee15a09-210f-4eea-b962-6af016052d6c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:05:35.614003 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:35.613981    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8g9l4\" (UniqueName: \"kubernetes.io/projected/5ee15a09-210f-4eea-b962-6af016052d6c-kube-api-access-8g9l4\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:05:35.614003 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:35.614005    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ee15a09-210f-4eea-b962-6af016052d6c-maas-api-xks-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:05:36.316650 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:36.316467    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec81d790-rl9dp" event={"ID":"5ee15a09-210f-4eea-b962-6af016052d6c","Type":"ContainerDied","Data":"168ebbe068ee781ac548974fda22026b75d2fced41aad3acd0ae46a09e636e82"}
Oct 30 13:05:36.316935 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:36.316916    4364 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="168ebbe068ee781ac548974fda22026b75d2fced41aad3acd0ae46a09e636e82"
Oct 30 13:05:36.317080 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:36.317070    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec81d790-rl9dp"
Oct 30 13:05:41.502396 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:05:41.502354    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:05:42.351490 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:42.351453    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6192dbad-6mrgv"]
Oct 30 13:05:42.352157 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:42.352142    4364 cpu_manager.go:414] "containerMap: removing container" podUID="5ee15a09-210f-4eea-b962-6af016052d6c" containerName="revoke-keys"
Oct 30 13:05:42.352224 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:42.352159    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ee15a09-210f-4eea-b962-6af016052d6c" containerName="revoke-keys"
Oct 30 13:05:42.352261 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:42.352238    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ee15a09-210f-4eea-b962-6af016052d6c" containerName="revoke-keys"
Oct 30 13:05:42.356875 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:42.356857    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6192dbad-6mrgv"
Oct 30 13:05:42.376786 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:42.376759    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6192dbad-6mrgv"]
Oct 30 13:05:42.386387 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:42.386359    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:05:42.386541 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:42.386438    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\""
Oct 30 13:05:42.497367 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:42.497332    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcbcc50f-e0b1-4260-9f8e-73822649fd7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6192dbad-6mrgv\" (UID: \"dcbcc50f-e0b1-4260-9f8e-73822649fd7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6192dbad-6mrgv"
Oct 30 13:05:42.497537 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:42.497466    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcbcc50f-e0b1-4260-9f8e-73822649fd7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6192dbad-6mrgv\" (UID: \"dcbcc50f-e0b1-4260-9f8e-73822649fd7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6192dbad-6mrgv"
Oct 30 13:05:42.497537 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:42.497510    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb26t\" (UniqueName: \"kubernetes.io/projected/dcbcc50f-e0b1-4260-9f8e-73822649fd7f-kube-api-access-fb26t\") pod \"maas-api-revoke-sub-6192dbad-6mrgv\" (UID: \"dcbcc50f-e0b1-4260-9f8e-73822649fd7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6192dbad-6mrgv"
Oct 30 13:05:42.599014 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:42.598970    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcbcc50f-e0b1-4260-9f8e-73822649fd7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6192dbad-6mrgv\" (UID: \"dcbcc50f-e0b1-4260-9f8e-73822649fd7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6192dbad-6mrgv"
Oct 30 13:05:42.599505 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:42.599043    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb26t\" (UniqueName: \"kubernetes.io/projected/dcbcc50f-e0b1-4260-9f8e-73822649fd7f-kube-api-access-fb26t\") pod \"maas-api-revoke-sub-6192dbad-6mrgv\" (UID: \"dcbcc50f-e0b1-4260-9f8e-73822649fd7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6192dbad-6mrgv"
Oct 30 13:05:42.599505 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:42.599183    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcbcc50f-e0b1-4260-9f8e-73822649fd7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6192dbad-6mrgv\" (UID: \"dcbcc50f-e0b1-4260-9f8e-73822649fd7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6192dbad-6mrgv"
Oct 30 13:05:42.599844 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:42.599820    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcbcc50f-e0b1-4260-9f8e-73822649fd7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6192dbad-6mrgv\" (UID: \"dcbcc50f-e0b1-4260-9f8e-73822649fd7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6192dbad-6mrgv"
Oct 30 13:05:42.601755 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:42.601695    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcbcc50f-e0b1-4260-9f8e-73822649fd7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6192dbad-6mrgv\" (UID: \"dcbcc50f-e0b1-4260-9f8e-73822649fd7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6192dbad-6mrgv"
Oct 30 13:05:42.624574 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:42.624547    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb26t\" (UniqueName: \"kubernetes.io/projected/dcbcc50f-e0b1-4260-9f8e-73822649fd7f-kube-api-access-fb26t\") pod \"maas-api-revoke-sub-6192dbad-6mrgv\" (UID: \"dcbcc50f-e0b1-4260-9f8e-73822649fd7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6192dbad-6mrgv"
Oct 30 13:05:42.672364 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:42.672326    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6192dbad-6mrgv"
Oct 30 13:05:42.980534 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:42.980504    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6192dbad-6mrgv"]
Oct 30 13:05:42.981544 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:05:42.981515    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcbcc50f_e0b1_4260_9f8e_73822649fd7f.slice/crio-96c33ace03b8bb935206575f640cbb05440e76c2e3f8bd48b3e1a8f69a095a3c WatchSource:0}: Error finding container 96c33ace03b8bb935206575f640cbb05440e76c2e3f8bd48b3e1a8f69a095a3c: Status 404 returned error can't find the container with id 96c33ace03b8bb935206575f640cbb05440e76c2e3f8bd48b3e1a8f69a095a3c
Oct 30 13:05:43.362551 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:43.362519    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6192dbad-6mrgv" event={"ID":"dcbcc50f-e0b1-4260-9f8e-73822649fd7f","Type":"ContainerStarted","Data":"c99d0b1f391a1b8fc8c79265021f3fb414c436636f3efe640f4e59e19acfca30"}
Oct 30 13:05:43.362709 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:43.362575    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6192dbad-6mrgv" event={"ID":"dcbcc50f-e0b1-4260-9f8e-73822649fd7f","Type":"ContainerStarted","Data":"96c33ace03b8bb935206575f640cbb05440e76c2e3f8bd48b3e1a8f69a095a3c"}
Oct 30 13:05:43.410686 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:43.410623    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6192dbad-6mrgv" podStartSLOduration=1.410585827 podStartE2EDuration="1.410585827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:05:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:43.408246504 +0000 UTC m=+842.574611154" watchObservedRunningTime="2026-10-30 13:05:43.410585827 +0000 UTC m=+842.576950474"
Oct 30 13:05:44.369815 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:44.369782    4364 generic.go:356] "Generic (PLEG): container finished" podID="dcbcc50f-e0b1-4260-9f8e-73822649fd7f" containerID="c99d0b1f391a1b8fc8c79265021f3fb414c436636f3efe640f4e59e19acfca30" exitCode=0
Oct 30 13:05:44.370258 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:44.369867    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6192dbad-6mrgv" event={"ID":"dcbcc50f-e0b1-4260-9f8e-73822649fd7f","Type":"ContainerDied","Data":"c99d0b1f391a1b8fc8c79265021f3fb414c436636f3efe640f4e59e19acfca30"}
Oct 30 13:05:45.529971 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:45.529947    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6192dbad-6mrgv"
Oct 30 13:05:45.632642 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:45.632555    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcbcc50f-e0b1-4260-9f8e-73822649fd7f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcbcc50f-e0b1-4260-9f8e-73822649fd7f-maas-api-openshift-ca\") pod \"dcbcc50f-e0b1-4260-9f8e-73822649fd7f\" (UID: \"dcbcc50f-e0b1-4260-9f8e-73822649fd7f\") "
Oct 30 13:05:45.632775 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:45.632684    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcbcc50f-e0b1-4260-9f8e-73822649fd7f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcbcc50f-e0b1-4260-9f8e-73822649fd7f-maas-api-xks-ca\") pod \"dcbcc50f-e0b1-4260-9f8e-73822649fd7f\" (UID: \"dcbcc50f-e0b1-4260-9f8e-73822649fd7f\") "
Oct 30 13:05:45.632775 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:45.632706    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcbcc50f-e0b1-4260-9f8e-73822649fd7f-kube-api-access-fb26t\" (UniqueName: \"kubernetes.io/projected/dcbcc50f-e0b1-4260-9f8e-73822649fd7f-kube-api-access-fb26t\") pod \"dcbcc50f-e0b1-4260-9f8e-73822649fd7f\" (UID: \"dcbcc50f-e0b1-4260-9f8e-73822649fd7f\") "
Oct 30 13:05:45.632904 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:45.632887    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcbcc50f-e0b1-4260-9f8e-73822649fd7f-maas-api-openshift-ca" pod "dcbcc50f-e0b1-4260-9f8e-73822649fd7f" (UID: "dcbcc50f-e0b1-4260-9f8e-73822649fd7f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:05:45.633119 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:45.633099    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcbcc50f-e0b1-4260-9f8e-73822649fd7f-maas-api-openshift-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:05:45.634757 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:45.634730    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcbcc50f-e0b1-4260-9f8e-73822649fd7f-maas-api-xks-ca" pod "dcbcc50f-e0b1-4260-9f8e-73822649fd7f" (UID: "dcbcc50f-e0b1-4260-9f8e-73822649fd7f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:05:45.634847 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:45.634818    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcbcc50f-e0b1-4260-9f8e-73822649fd7f-kube-api-access-fb26t" pod "dcbcc50f-e0b1-4260-9f8e-73822649fd7f" (UID: "dcbcc50f-e0b1-4260-9f8e-73822649fd7f"). InnerVolumeSpecName "kube-api-access-fb26t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:05:45.734019 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:45.733985    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcbcc50f-e0b1-4260-9f8e-73822649fd7f-maas-api-xks-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:05:45.734019 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:45.734014    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fb26t\" (UniqueName: \"kubernetes.io/projected/dcbcc50f-e0b1-4260-9f8e-73822649fd7f-kube-api-access-fb26t\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:05:46.345643 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:46.345608    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbd8b596-j22lj"]
Oct 30 13:05:46.346153 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:46.346134    4364 cpu_manager.go:414] "containerMap: removing container" podUID="dcbcc50f-e0b1-4260-9f8e-73822649fd7f" containerName="revoke-keys"
Oct 30 13:05:46.346204 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:46.346153    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcbcc50f-e0b1-4260-9f8e-73822649fd7f" containerName="revoke-keys"
Oct 30 13:05:46.346251 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:46.346242    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcbcc50f-e0b1-4260-9f8e-73822649fd7f" containerName="revoke-keys"
Oct 30 13:05:46.349779 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:46.349760    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd8b596-j22lj"
Oct 30 13:05:46.369385 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:46.369354    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbd8b596-j22lj"]
Oct 30 13:05:46.383781 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:46.383744    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6192dbad-6mrgv"
Oct 30 13:05:46.383781 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:46.383753    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6192dbad-6mrgv" event={"ID":"dcbcc50f-e0b1-4260-9f8e-73822649fd7f","Type":"ContainerDied","Data":"96c33ace03b8bb935206575f640cbb05440e76c2e3f8bd48b3e1a8f69a095a3c"}
Oct 30 13:05:46.383983 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:46.383792    4364 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96c33ace03b8bb935206575f640cbb05440e76c2e3f8bd48b3e1a8f69a095a3c"
Oct 30 13:05:46.440269 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:46.440232    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c25ebdb3-1e2a-465b-9afe-68a89715f718-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbd8b596-j22lj\" (UID: \"c25ebdb3-1e2a-465b-9afe-68a89715f718\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd8b596-j22lj"
Oct 30 13:05:46.440461 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:46.440417    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qt4v\" (UniqueName: \"kubernetes.io/projected/c25ebdb3-1e2a-465b-9afe-68a89715f718-kube-api-access-9qt4v\") pod \"maas-api-revoke-sub-dbd8b596-j22lj\" (UID: \"c25ebdb3-1e2a-465b-9afe-68a89715f718\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd8b596-j22lj"
Oct 30 13:05:46.440510 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:46.440462    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c25ebdb3-1e2a-465b-9afe-68a89715f718-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbd8b596-j22lj\" (UID: \"c25ebdb3-1e2a-465b-9afe-68a89715f718\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd8b596-j22lj"
Oct 30 13:05:46.541949 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:46.541914    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c25ebdb3-1e2a-465b-9afe-68a89715f718-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbd8b596-j22lj\" (UID: \"c25ebdb3-1e2a-465b-9afe-68a89715f718\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd8b596-j22lj"
Oct 30 13:05:46.542404 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:46.541979    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c25ebdb3-1e2a-465b-9afe-68a89715f718-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbd8b596-j22lj\" (UID: \"c25ebdb3-1e2a-465b-9afe-68a89715f718\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd8b596-j22lj"
Oct 30 13:05:46.542404 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:46.542173    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qt4v\" (UniqueName: \"kubernetes.io/projected/c25ebdb3-1e2a-465b-9afe-68a89715f718-kube-api-access-9qt4v\") pod \"maas-api-revoke-sub-dbd8b596-j22lj\" (UID: \"c25ebdb3-1e2a-465b-9afe-68a89715f718\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd8b596-j22lj"
Oct 30 13:05:46.542891 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:46.542864    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c25ebdb3-1e2a-465b-9afe-68a89715f718-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbd8b596-j22lj\" (UID: \"c25ebdb3-1e2a-465b-9afe-68a89715f718\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd8b596-j22lj"
Oct 30 13:05:46.544513 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:46.544489    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c25ebdb3-1e2a-465b-9afe-68a89715f718-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbd8b596-j22lj\" (UID: \"c25ebdb3-1e2a-465b-9afe-68a89715f718\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd8b596-j22lj"
Oct 30 13:05:46.567865 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:46.567835    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qt4v\" (UniqueName: \"kubernetes.io/projected/c25ebdb3-1e2a-465b-9afe-68a89715f718-kube-api-access-9qt4v\") pod \"maas-api-revoke-sub-dbd8b596-j22lj\" (UID: \"c25ebdb3-1e2a-465b-9afe-68a89715f718\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd8b596-j22lj"
Oct 30 13:05:46.663217 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:46.663181    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd8b596-j22lj"
Oct 30 13:05:46.968180 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:05:46.968152    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc25ebdb3_1e2a_465b_9afe_68a89715f718.slice/crio-87ea25816fd58368d7622a4fa5ca4a2c9aa7e4dcba1e04f09d245a7deaaa81b5 WatchSource:0}: Error finding container 87ea25816fd58368d7622a4fa5ca4a2c9aa7e4dcba1e04f09d245a7deaaa81b5: Status 404 returned error can't find the container with id 87ea25816fd58368d7622a4fa5ca4a2c9aa7e4dcba1e04f09d245a7deaaa81b5
Oct 30 13:05:46.971353 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:46.971330    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbd8b596-j22lj"]
Oct 30 13:05:47.392132 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:47.392090    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd8b596-j22lj" event={"ID":"c25ebdb3-1e2a-465b-9afe-68a89715f718","Type":"ContainerStarted","Data":"f316989fd066dc19051e435fb98341dcb5d916d229b8ceceff0a51935c4943b8"}
Oct 30 13:05:47.392297 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:47.392151    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd8b596-j22lj" event={"ID":"c25ebdb3-1e2a-465b-9afe-68a89715f718","Type":"ContainerStarted","Data":"87ea25816fd58368d7622a4fa5ca4a2c9aa7e4dcba1e04f09d245a7deaaa81b5"}
Oct 30 13:05:47.439529 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:47.439472    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd8b596-j22lj" podStartSLOduration=1.439457115 podStartE2EDuration="1.439457115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:05:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:47.438341612 +0000 UTC m=+846.604706261" watchObservedRunningTime="2026-10-30 13:05:47.439457115 +0000 UTC m=+846.605821762"
Oct 30 13:05:48.399751 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:48.399720    4364 generic.go:356] "Generic (PLEG): container finished" podID="c25ebdb3-1e2a-465b-9afe-68a89715f718" containerID="f316989fd066dc19051e435fb98341dcb5d916d229b8ceceff0a51935c4943b8" exitCode=0
Oct 30 13:05:48.400150 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:48.399798    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd8b596-j22lj" event={"ID":"c25ebdb3-1e2a-465b-9afe-68a89715f718","Type":"ContainerDied","Data":"f316989fd066dc19051e435fb98341dcb5d916d229b8ceceff0a51935c4943b8"}
Oct 30 13:05:49.550832 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:49.550808    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd8b596-j22lj"
Oct 30 13:05:49.679672 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:49.679591    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c25ebdb3-1e2a-465b-9afe-68a89715f718-kube-api-access-9qt4v\" (UniqueName: \"kubernetes.io/projected/c25ebdb3-1e2a-465b-9afe-68a89715f718-kube-api-access-9qt4v\") pod \"c25ebdb3-1e2a-465b-9afe-68a89715f718\" (UID: \"c25ebdb3-1e2a-465b-9afe-68a89715f718\") "
Oct 30 13:05:49.679672 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:49.679657    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c25ebdb3-1e2a-465b-9afe-68a89715f718-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c25ebdb3-1e2a-465b-9afe-68a89715f718-maas-api-openshift-ca\") pod \"c25ebdb3-1e2a-465b-9afe-68a89715f718\" (UID: \"c25ebdb3-1e2a-465b-9afe-68a89715f718\") "
Oct 30 13:05:49.679832 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:49.679700    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c25ebdb3-1e2a-465b-9afe-68a89715f718-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c25ebdb3-1e2a-465b-9afe-68a89715f718-maas-api-xks-ca\") pod \"c25ebdb3-1e2a-465b-9afe-68a89715f718\" (UID: \"c25ebdb3-1e2a-465b-9afe-68a89715f718\") "
Oct 30 13:05:49.680031 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:49.679997    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c25ebdb3-1e2a-465b-9afe-68a89715f718-maas-api-openshift-ca" pod "c25ebdb3-1e2a-465b-9afe-68a89715f718" (UID: "c25ebdb3-1e2a-465b-9afe-68a89715f718"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:05:49.681701 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:49.681679    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c25ebdb3-1e2a-465b-9afe-68a89715f718-maas-api-xks-ca" pod "c25ebdb3-1e2a-465b-9afe-68a89715f718" (UID: "c25ebdb3-1e2a-465b-9afe-68a89715f718"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:05:49.681777 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:49.681746    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c25ebdb3-1e2a-465b-9afe-68a89715f718-kube-api-access-9qt4v" pod "c25ebdb3-1e2a-465b-9afe-68a89715f718" (UID: "c25ebdb3-1e2a-465b-9afe-68a89715f718"). InnerVolumeSpecName "kube-api-access-9qt4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:05:49.781301 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:49.781275    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c25ebdb3-1e2a-465b-9afe-68a89715f718-maas-api-xks-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:05:49.781301 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:49.781299    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qt4v\" (UniqueName: \"kubernetes.io/projected/c25ebdb3-1e2a-465b-9afe-68a89715f718-kube-api-access-9qt4v\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:05:49.781456 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:49.781309    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c25ebdb3-1e2a-465b-9afe-68a89715f718-maas-api-openshift-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:05:50.415059 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:50.415015    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd8b596-j22lj" event={"ID":"c25ebdb3-1e2a-465b-9afe-68a89715f718","Type":"ContainerDied","Data":"87ea25816fd58368d7622a4fa5ca4a2c9aa7e4dcba1e04f09d245a7deaaa81b5"}
Oct 30 13:05:50.415205 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:50.415071    4364 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87ea25816fd58368d7622a4fa5ca4a2c9aa7e4dcba1e04f09d245a7deaaa81b5"
Oct 30 13:05:50.415205 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:50.415034    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbd8b596-j22lj"
Oct 30 13:05:52.557612 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:52.557565    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww"]
Oct 30 13:05:52.558096 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:52.558081    4364 cpu_manager.go:414] "containerMap: removing container" podUID="c25ebdb3-1e2a-465b-9afe-68a89715f718" containerName="revoke-keys"
Oct 30 13:05:52.558139 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:52.558096    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c25ebdb3-1e2a-465b-9afe-68a89715f718" containerName="revoke-keys"
Oct 30 13:05:52.558188 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:52.558180    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="c25ebdb3-1e2a-465b-9afe-68a89715f718" containerName="revoke-keys"
Oct 30 13:05:52.561753 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:52.561738    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww"
Oct 30 13:05:52.565349 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:52.565318    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f1549da-ef30-4001-9eec-35b1040de3b6-credential-socket\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww\" (UID: \"3f1549da-ef30-4001-9eec-35b1040de3b6\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww"
Oct 30 13:05:52.565464 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:52.565363    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5kdk\" (UniqueName: \"kubernetes.io/projected/3f1549da-ef30-4001-9eec-35b1040de3b6-kube-api-access-t5kdk\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww\" (UID: \"3f1549da-ef30-4001-9eec-35b1040de3b6\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww"
Oct 30 13:05:52.565517 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:52.565502    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f1549da-ef30-4001-9eec-35b1040de3b6-istio-token\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww\" (UID: \"3f1549da-ef30-4001-9eec-35b1040de3b6\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww"
Oct 30 13:05:52.565631 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:52.565587    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f1549da-ef30-4001-9eec-35b1040de3b6-istio-envoy\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww\" (UID: \"3f1549da-ef30-4001-9eec-35b1040de3b6\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww"
Oct 30 13:05:52.565760 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:52.565687    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f1549da-ef30-4001-9eec-35b1040de3b6-workload-certs\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww\" (UID: \"3f1549da-ef30-4001-9eec-35b1040de3b6\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww"
Oct 30 13:05:52.565760 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:52.565728    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f1549da-ef30-4001-9eec-35b1040de3b6-istio-podinfo\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww\" (UID: \"3f1549da-ef30-4001-9eec-35b1040de3b6\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww"
Oct 30 13:05:52.565846 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:52.565796    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f1549da-ef30-4001-9eec-35b1040de3b6-istio-data\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww\" (UID: \"3f1549da-ef30-4001-9eec-35b1040de3b6\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww"
Oct 30 13:05:52.565846 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:52.565831    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f1549da-ef30-4001-9eec-35b1040de3b6-workload-socket\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww\" (UID: \"3f1549da-ef30-4001-9eec-35b1040de3b6\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww"
Oct 30 13:05:52.565920 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:52.565849    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f1549da-ef30-4001-9eec-35b1040de3b6-istiod-ca-cert\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww\" (UID: \"3f1549da-ef30-4001-9eec-35b1040de3b6\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww"
Oct 30 13:05:52.585717 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:52.585697    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww"]
Oct 30 13:05:52.595676 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:52.595653    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-d7e62b-openshift-default-dockercfg-29w5v\""
Oct 30 13:05:52.666495 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:52.666462    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f1549da-ef30-4001-9eec-35b1040de3b6-istio-envoy\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww\" (UID: \"3f1549da-ef30-4001-9eec-35b1040de3b6\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww"
Oct 30 13:05:52.666657 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:52.666516    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f1549da-ef30-4001-9eec-35b1040de3b6-workload-certs\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww\" (UID: \"3f1549da-ef30-4001-9eec-35b1040de3b6\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww"
Oct 30 13:05:52.666657 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:52.666540    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f1549da-ef30-4001-9eec-35b1040de3b6-istio-podinfo\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww\" (UID: \"3f1549da-ef30-4001-9eec-35b1040de3b6\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww"
Oct 30 13:05:52.666657 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:52.666590    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f1549da-ef30-4001-9eec-35b1040de3b6-istio-data\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww\" (UID: \"3f1549da-ef30-4001-9eec-35b1040de3b6\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww"
Oct 30 13:05:52.666657 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:52.666646    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f1549da-ef30-4001-9eec-35b1040de3b6-workload-socket\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww\" (UID: \"3f1549da-ef30-4001-9eec-35b1040de3b6\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww"
Oct 30 13:05:52.666883 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:52.666673    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f1549da-ef30-4001-9eec-35b1040de3b6-istiod-ca-cert\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww\" (UID: \"3f1549da-ef30-4001-9eec-35b1040de3b6\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww"
Oct 30 13:05:52.666883 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:52.666710    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f1549da-ef30-4001-9eec-35b1040de3b6-credential-socket\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww\" (UID: \"3f1549da-ef30-4001-9eec-35b1040de3b6\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww"
Oct 30 13:05:52.666883 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:52.666734    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5kdk\" (UniqueName: \"kubernetes.io/projected/3f1549da-ef30-4001-9eec-35b1040de3b6-kube-api-access-t5kdk\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww\" (UID: \"3f1549da-ef30-4001-9eec-35b1040de3b6\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww"
Oct 30 13:05:52.666883 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:52.666820    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f1549da-ef30-4001-9eec-35b1040de3b6-istio-token\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww\" (UID: \"3f1549da-ef30-4001-9eec-35b1040de3b6\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww"
Oct 30 13:05:52.667070 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:52.667009    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f1549da-ef30-4001-9eec-35b1040de3b6-workload-certs\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww\" (UID: \"3f1549da-ef30-4001-9eec-35b1040de3b6\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww"
Oct 30 13:05:52.667318 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:52.667291    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f1549da-ef30-4001-9eec-35b1040de3b6-istiod-ca-cert\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww\" (UID: \"3f1549da-ef30-4001-9eec-35b1040de3b6\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww"
Oct 30 13:05:52.667407 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:52.667388    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f1549da-ef30-4001-9eec-35b1040de3b6-workload-socket\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww\" (UID: \"3f1549da-ef30-4001-9eec-35b1040de3b6\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww"
Oct 30 13:05:52.667494 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:52.667474    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f1549da-ef30-4001-9eec-35b1040de3b6-credential-socket\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww\" (UID: \"3f1549da-ef30-4001-9eec-35b1040de3b6\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww"
Oct 30 13:05:52.667539 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:52.667508    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f1549da-ef30-4001-9eec-35b1040de3b6-istio-data\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww\" (UID: \"3f1549da-ef30-4001-9eec-35b1040de3b6\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww"
Oct 30 13:05:52.668787 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:52.668769    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f1549da-ef30-4001-9eec-35b1040de3b6-istio-envoy\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww\" (UID: \"3f1549da-ef30-4001-9eec-35b1040de3b6\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww"
Oct 30 13:05:52.669042 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:52.669023    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f1549da-ef30-4001-9eec-35b1040de3b6-istio-podinfo\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww\" (UID: \"3f1549da-ef30-4001-9eec-35b1040de3b6\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww"
Oct 30 13:05:52.692057 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:52.692031    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5kdk\" (UniqueName: \"kubernetes.io/projected/3f1549da-ef30-4001-9eec-35b1040de3b6-kube-api-access-t5kdk\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww\" (UID: \"3f1549da-ef30-4001-9eec-35b1040de3b6\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww"
Oct 30 13:05:52.692057 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:52.692044    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f1549da-ef30-4001-9eec-35b1040de3b6-istio-token\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww\" (UID: \"3f1549da-ef30-4001-9eec-35b1040de3b6\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww"
Oct 30 13:05:52.875795 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:52.875761    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww"
Oct 30 13:05:53.217180 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:53.217150    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww"]
Oct 30 13:05:53.218940 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:05:53.218911    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f1549da_ef30_4001_9eec_35b1040de3b6.slice/crio-9479077bbf563bda5f61013c379a0c4f66ca43b1e97822a4afaae8a09135e95f WatchSource:0}: Error finding container 9479077bbf563bda5f61013c379a0c4f66ca43b1e97822a4afaae8a09135e95f: Status 404 returned error can't find the container with id 9479077bbf563bda5f61013c379a0c4f66ca43b1e97822a4afaae8a09135e95f
Oct 30 13:05:53.221559 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:53.221527    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:05:53.221691 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:53.221619    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:05:53.221691 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:53.221665    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:05:53.454890 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:53.454857    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww" event={"ID":"3f1549da-ef30-4001-9eec-35b1040de3b6","Type":"ContainerStarted","Data":"8245749a780d8d79e3a62f3120c8bfb5456abe2999f435a12322852a787eae13"}
Oct 30 13:05:53.455081 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:53.454909    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww" event={"ID":"3f1549da-ef30-4001-9eec-35b1040de3b6","Type":"ContainerStarted","Data":"9479077bbf563bda5f61013c379a0c4f66ca43b1e97822a4afaae8a09135e95f"}
Oct 30 13:05:53.514755 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:53.514634    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww" podStartSLOduration=1.514587252 podStartE2EDuration="1.514587252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:05:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:53.513009614 +0000 UTC m=+852.679374278" watchObservedRunningTime="2026-10-30 13:05:53.514587252 +0000 UTC m=+852.680951900"
Oct 30 13:05:53.876700 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:53.876662    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww"
Oct 30 13:05:54.877170 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:54.877134    4364 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 13:05:54.877646 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:54.877203    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww" podUID="3f1549da-ef30-4001-9eec-35b1040de3b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 13:05:55.876504 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:55.876470    4364 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 13:05:55.876734 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:55.876540    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww" podUID="3f1549da-ef30-4001-9eec-35b1040de3b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 13:05:56.822272 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:56.822235    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz"]
Oct 30 13:05:56.827336 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:56.827310    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz"
Oct 30 13:05:56.832880 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:56.832851    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc5caf69-a649-47e5-8910-9b8f331dfd4c-tmp-dir\") pod \"distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz\" (UID: \"bc5caf69-a649-47e5-8910-9b8f331dfd4c\") " pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz"
Oct 30 13:05:56.832998 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:56.832895    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc5caf69-a649-47e5-8910-9b8f331dfd4c-dshm\") pod \"distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz\" (UID: \"bc5caf69-a649-47e5-8910-9b8f331dfd4c\") " pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz"
Oct 30 13:05:56.832998 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:56.832920    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc5caf69-a649-47e5-8910-9b8f331dfd4c-tls-certs\") pod \"distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz\" (UID: \"bc5caf69-a649-47e5-8910-9b8f331dfd4c\") " pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz"
Oct 30 13:05:56.833195 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:56.833092    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc5caf69-a649-47e5-8910-9b8f331dfd4c-model-cache\") pod \"distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz\" (UID: \"bc5caf69-a649-47e5-8910-9b8f331dfd4c\") " pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz"
Oct 30 13:05:56.833275 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:56.833216    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gprtw\" (UniqueName: \"kubernetes.io/projected/bc5caf69-a649-47e5-8910-9b8f331dfd4c-kube-api-access-gprtw\") pod \"distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz\" (UID: \"bc5caf69-a649-47e5-8910-9b8f331dfd4c\") " pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz"
Oct 30 13:05:56.833275 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:56.833252    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc5caf69-a649-47e5-8910-9b8f331dfd4c-home\") pod \"distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz\" (UID: \"bc5caf69-a649-47e5-8910-9b8f331dfd4c\") " pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz"
Oct 30 13:05:56.847709 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:56.847682    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz"]
Oct 30 13:05:56.857731 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:56.857701    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f07dfe\"/\"kube-root-ca.crt\""
Oct 30 13:05:56.858084 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:56.857867    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f07dfe\"/\"default-dockercfg-zhhrb\""
Oct 30 13:05:56.858084 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:56.857917    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f07dfe\"/\"openshift-service-ca.crt\""
Oct 30 13:05:56.858084 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:56.858008    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f07dfe\"/\"distinct-w0-f07dfe-kserve-self-signed-certs\""
Oct 30 13:05:56.876708 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:56.876683    4364 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 13:05:56.876829 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:56.876740    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww" podUID="3f1549da-ef30-4001-9eec-35b1040de3b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 13:05:56.933872 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:56.933838    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gprtw\" (UniqueName: \"kubernetes.io/projected/bc5caf69-a649-47e5-8910-9b8f331dfd4c-kube-api-access-gprtw\") pod \"distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz\" (UID: \"bc5caf69-a649-47e5-8910-9b8f331dfd4c\") " pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz"
Oct 30 13:05:56.933872 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:56.933875    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc5caf69-a649-47e5-8910-9b8f331dfd4c-home\") pod \"distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz\" (UID: \"bc5caf69-a649-47e5-8910-9b8f331dfd4c\") " pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz"
Oct 30 13:05:56.934087 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:56.933915    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc5caf69-a649-47e5-8910-9b8f331dfd4c-tmp-dir\") pod \"distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz\" (UID: \"bc5caf69-a649-47e5-8910-9b8f331dfd4c\") " pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz"
Oct 30 13:05:56.934087 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:56.933940    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc5caf69-a649-47e5-8910-9b8f331dfd4c-dshm\") pod \"distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz\" (UID: \"bc5caf69-a649-47e5-8910-9b8f331dfd4c\") " pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz"
Oct 30 13:05:56.934087 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:56.933979    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc5caf69-a649-47e5-8910-9b8f331dfd4c-tls-certs\") pod \"distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz\" (UID: \"bc5caf69-a649-47e5-8910-9b8f331dfd4c\") " pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz"
Oct 30 13:05:56.934087 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:56.934081    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc5caf69-a649-47e5-8910-9b8f331dfd4c-model-cache\") pod \"distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz\" (UID: \"bc5caf69-a649-47e5-8910-9b8f331dfd4c\") " pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz"
Oct 30 13:05:56.934361 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:56.934333    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc5caf69-a649-47e5-8910-9b8f331dfd4c-tmp-dir\") pod \"distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz\" (UID: \"bc5caf69-a649-47e5-8910-9b8f331dfd4c\") " pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz"
Oct 30 13:05:56.934498 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:56.934384    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc5caf69-a649-47e5-8910-9b8f331dfd4c-home\") pod \"distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz\" (UID: \"bc5caf69-a649-47e5-8910-9b8f331dfd4c\") " pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz"
Oct 30 13:05:56.934498 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:56.934486    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc5caf69-a649-47e5-8910-9b8f331dfd4c-model-cache\") pod \"distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz\" (UID: \"bc5caf69-a649-47e5-8910-9b8f331dfd4c\") " pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz"
Oct 30 13:05:56.936359 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:56.936336    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc5caf69-a649-47e5-8910-9b8f331dfd4c-dshm\") pod \"distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz\" (UID: \"bc5caf69-a649-47e5-8910-9b8f331dfd4c\") " pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz"
Oct 30 13:05:56.936609 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:56.936577    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc5caf69-a649-47e5-8910-9b8f331dfd4c-tls-certs\") pod \"distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz\" (UID: \"bc5caf69-a649-47e5-8910-9b8f331dfd4c\") " pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz"
Oct 30 13:05:56.958944 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:56.958920    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gprtw\" (UniqueName: \"kubernetes.io/projected/bc5caf69-a649-47e5-8910-9b8f331dfd4c-kube-api-access-gprtw\") pod \"distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz\" (UID: \"bc5caf69-a649-47e5-8910-9b8f331dfd4c\") " pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz"
Oct 30 13:05:57.142068 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:57.142033    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz"
Oct 30 13:05:57.475356 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:57.475325    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz"]
Oct 30 13:05:57.475784 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:05:57.475753    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbc5caf69_a649_47e5_8910_9b8f331dfd4c.slice/crio-cfc488672987dc30f155d337874f1b808af4826566d274b3a4b9ce71f00aff25 WatchSource:0}: Error finding container cfc488672987dc30f155d337874f1b808af4826566d274b3a4b9ce71f00aff25: Status 404 returned error can't find the container with id cfc488672987dc30f155d337874f1b808af4826566d274b3a4b9ce71f00aff25
Oct 30 13:05:57.876526 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:57.876491    4364 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 13:05:57.877009 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:57.876548    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww" podUID="3f1549da-ef30-4001-9eec-35b1040de3b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 13:05:58.514664 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:58.514629    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz" event={"ID":"bc5caf69-a649-47e5-8910-9b8f331dfd4c","Type":"ContainerStarted","Data":"2ec7063c449c85cf544188eafe6b344cb3344f6f2574924f27ce8248d593d747"}
Oct 30 13:05:58.514842 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:58.514674    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz" event={"ID":"bc5caf69-a649-47e5-8910-9b8f331dfd4c","Type":"ContainerStarted","Data":"cfc488672987dc30f155d337874f1b808af4826566d274b3a4b9ce71f00aff25"}
Oct 30 13:05:58.564060 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:58.564013    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz" podStartSLOduration=2.563981789 podStartE2EDuration="2.563981789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:05:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:58.560092593 +0000 UTC m=+857.726457242" watchObservedRunningTime="2026-10-30 13:05:58.563981789 +0000 UTC m=+857.730346434"
Oct 30 13:05:58.876870 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:58.876836    4364 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 13:05:58.877264 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:58.876885    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww" podUID="3f1549da-ef30-4001-9eec-35b1040de3b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 13:05:59.142923 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:59.142845    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz"
Oct 30 13:05:59.877030 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:59.876995    4364 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 13:05:59.877385 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:05:59.877053    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww" podUID="3f1549da-ef30-4001-9eec-35b1040de3b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 13:06:00.710967 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:00.710935    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cf4d32e-vvmq5"]
Oct 30 13:06:00.714910 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:00.714883    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf4d32e-vvmq5"
Oct 30 13:06:00.731720 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:00.731691    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cf4d32e-vvmq5"]
Oct 30 13:06:00.741530 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:00.741505    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:06:00.741756 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:00.741732    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\""
Oct 30 13:06:00.782494 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:00.782468    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43d5c676-c1f0-4dfc-9ef3-d027f6c9fd2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5cf4d32e-vvmq5\" (UID: \"43d5c676-c1f0-4dfc-9ef3-d027f6c9fd2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf4d32e-vvmq5"
Oct 30 13:06:00.782641 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:00.782501    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43d5c676-c1f0-4dfc-9ef3-d027f6c9fd2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5cf4d32e-vvmq5\" (UID: \"43d5c676-c1f0-4dfc-9ef3-d027f6c9fd2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf4d32e-vvmq5"
Oct 30 13:06:00.782641 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:00.782558    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sntcp\" (UniqueName: \"kubernetes.io/projected/43d5c676-c1f0-4dfc-9ef3-d027f6c9fd2c-kube-api-access-sntcp\") pod \"maas-api-revoke-sub-5cf4d32e-vvmq5\" (UID: \"43d5c676-c1f0-4dfc-9ef3-d027f6c9fd2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf4d32e-vvmq5"
Oct 30 13:06:00.881263 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:00.881236    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww"
Oct 30 13:06:00.881807 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:00.881785    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww"
Oct 30 13:06:00.882557 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:00.882538    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww"
Oct 30 13:06:00.884009 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:00.883987    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43d5c676-c1f0-4dfc-9ef3-d027f6c9fd2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5cf4d32e-vvmq5\" (UID: \"43d5c676-c1f0-4dfc-9ef3-d027f6c9fd2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf4d32e-vvmq5"
Oct 30 13:06:00.884187 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:00.884026    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43d5c676-c1f0-4dfc-9ef3-d027f6c9fd2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5cf4d32e-vvmq5\" (UID: \"43d5c676-c1f0-4dfc-9ef3-d027f6c9fd2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf4d32e-vvmq5"
Oct 30 13:06:00.884187 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:00.884146    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sntcp\" (UniqueName: \"kubernetes.io/projected/43d5c676-c1f0-4dfc-9ef3-d027f6c9fd2c-kube-api-access-sntcp\") pod \"maas-api-revoke-sub-5cf4d32e-vvmq5\" (UID: \"43d5c676-c1f0-4dfc-9ef3-d027f6c9fd2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf4d32e-vvmq5"
Oct 30 13:06:00.884700 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:00.884679    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43d5c676-c1f0-4dfc-9ef3-d027f6c9fd2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5cf4d32e-vvmq5\" (UID: \"43d5c676-c1f0-4dfc-9ef3-d027f6c9fd2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf4d32e-vvmq5"
Oct 30 13:06:00.886742 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:00.886719    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43d5c676-c1f0-4dfc-9ef3-d027f6c9fd2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5cf4d32e-vvmq5\" (UID: \"43d5c676-c1f0-4dfc-9ef3-d027f6c9fd2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf4d32e-vvmq5"
Oct 30 13:06:00.910240 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:00.909854    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sntcp\" (UniqueName: \"kubernetes.io/projected/43d5c676-c1f0-4dfc-9ef3-d027f6c9fd2c-kube-api-access-sntcp\") pod \"maas-api-revoke-sub-5cf4d32e-vvmq5\" (UID: \"43d5c676-c1f0-4dfc-9ef3-d027f6c9fd2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf4d32e-vvmq5"
Oct 30 13:06:01.034974 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:01.034903    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf4d32e-vvmq5"
Oct 30 13:06:01.350487 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:06:01.350460    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43d5c676_c1f0_4dfc_9ef3_d027f6c9fd2c.slice/crio-41012ffb824f3e0d25b353103b96f35b7642f60e015b1ccdf3f028fb620caa44 WatchSource:0}: Error finding container 41012ffb824f3e0d25b353103b96f35b7642f60e015b1ccdf3f028fb620caa44: Status 404 returned error can't find the container with id 41012ffb824f3e0d25b353103b96f35b7642f60e015b1ccdf3f028fb620caa44
Oct 30 13:06:01.350487 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:01.350474    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cf4d32e-vvmq5"]
Oct 30 13:06:01.533809 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:01.533777    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf4d32e-vvmq5" event={"ID":"43d5c676-c1f0-4dfc-9ef3-d027f6c9fd2c","Type":"ContainerStarted","Data":"41012ffb824f3e0d25b353103b96f35b7642f60e015b1ccdf3f028fb620caa44"}
Oct 30 13:06:02.541991 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:02.541915    4364 generic.go:356] "Generic (PLEG): container finished" podID="43d5c676-c1f0-4dfc-9ef3-d027f6c9fd2c" containerID="417d6624328d0a9d834ad2fc8e6a21dcee258a9144a334e3ae2afe295c9b7cad" exitCode=0
Oct 30 13:06:02.541991 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:02.541978    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf4d32e-vvmq5" event={"ID":"43d5c676-c1f0-4dfc-9ef3-d027f6c9fd2c","Type":"ContainerDied","Data":"417d6624328d0a9d834ad2fc8e6a21dcee258a9144a334e3ae2afe295c9b7cad"}
Oct 30 13:06:03.719273 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:03.719251    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf4d32e-vvmq5"
Oct 30 13:06:03.823768 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:03.823700    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43d5c676-c1f0-4dfc-9ef3-d027f6c9fd2c-kube-api-access-sntcp\" (UniqueName: \"kubernetes.io/projected/43d5c676-c1f0-4dfc-9ef3-d027f6c9fd2c-kube-api-access-sntcp\") pod \"43d5c676-c1f0-4dfc-9ef3-d027f6c9fd2c\" (UID: \"43d5c676-c1f0-4dfc-9ef3-d027f6c9fd2c\") "
Oct 30 13:06:03.823768 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:03.823734    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43d5c676-c1f0-4dfc-9ef3-d027f6c9fd2c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43d5c676-c1f0-4dfc-9ef3-d027f6c9fd2c-maas-api-xks-ca\") pod \"43d5c676-c1f0-4dfc-9ef3-d027f6c9fd2c\" (UID: \"43d5c676-c1f0-4dfc-9ef3-d027f6c9fd2c\") "
Oct 30 13:06:03.823917 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:03.823778    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43d5c676-c1f0-4dfc-9ef3-d027f6c9fd2c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43d5c676-c1f0-4dfc-9ef3-d027f6c9fd2c-maas-api-openshift-ca\") pod \"43d5c676-c1f0-4dfc-9ef3-d027f6c9fd2c\" (UID: \"43d5c676-c1f0-4dfc-9ef3-d027f6c9fd2c\") "
Oct 30 13:06:03.824207 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:03.824180    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43d5c676-c1f0-4dfc-9ef3-d027f6c9fd2c-maas-api-openshift-ca" pod "43d5c676-c1f0-4dfc-9ef3-d027f6c9fd2c" (UID: "43d5c676-c1f0-4dfc-9ef3-d027f6c9fd2c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:06:03.825720 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:03.825700    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43d5c676-c1f0-4dfc-9ef3-d027f6c9fd2c-maas-api-xks-ca" pod "43d5c676-c1f0-4dfc-9ef3-d027f6c9fd2c" (UID: "43d5c676-c1f0-4dfc-9ef3-d027f6c9fd2c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:06:03.825821 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:03.825796    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43d5c676-c1f0-4dfc-9ef3-d027f6c9fd2c-kube-api-access-sntcp" pod "43d5c676-c1f0-4dfc-9ef3-d027f6c9fd2c" (UID: "43d5c676-c1f0-4dfc-9ef3-d027f6c9fd2c"). InnerVolumeSpecName "kube-api-access-sntcp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:06:03.925725 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:03.925695    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43d5c676-c1f0-4dfc-9ef3-d027f6c9fd2c-maas-api-openshift-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:06:03.925843 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:03.925725    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sntcp\" (UniqueName: \"kubernetes.io/projected/43d5c676-c1f0-4dfc-9ef3-d027f6c9fd2c-kube-api-access-sntcp\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:06:03.925843 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:03.925740    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43d5c676-c1f0-4dfc-9ef3-d027f6c9fd2c-maas-api-xks-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:06:03.928608 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:03.928574    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tbw9c"]
Oct 30 13:06:03.929107 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:03.929094    4364 cpu_manager.go:414] "containerMap: removing container" podUID="43d5c676-c1f0-4dfc-9ef3-d027f6c9fd2c" containerName="revoke-keys"
Oct 30 13:06:03.929143 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:03.929108    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43d5c676-c1f0-4dfc-9ef3-d027f6c9fd2c" containerName="revoke-keys"
Oct 30 13:06:03.929219 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:03.929210    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="43d5c676-c1f0-4dfc-9ef3-d027f6c9fd2c" containerName="revoke-keys"
Oct 30 13:06:03.932485 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:03.932468    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tbw9c"
Oct 30 13:06:03.959161 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:03.959133    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tbw9c"]
Oct 30 13:06:04.026733 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:04.026702    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e40d7a56-5869-4a42-b8a7-33c02604c302-istio-data\") pod \"maas-default-gateway-openshift-default-64d4cc998c-tbw9c\" (UID: \"e40d7a56-5869-4a42-b8a7-33c02604c302\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tbw9c"
Oct 30 13:06:04.026892 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:04.026748    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e40d7a56-5869-4a42-b8a7-33c02604c302-workload-certs\") pod \"maas-default-gateway-openshift-default-64d4cc998c-tbw9c\" (UID: \"e40d7a56-5869-4a42-b8a7-33c02604c302\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tbw9c"
Oct 30 13:06:04.026892 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:04.026868    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txqdn\" (UniqueName: \"kubernetes.io/projected/e40d7a56-5869-4a42-b8a7-33c02604c302-kube-api-access-txqdn\") pod \"maas-default-gateway-openshift-default-64d4cc998c-tbw9c\" (UID: \"e40d7a56-5869-4a42-b8a7-33c02604c302\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tbw9c"
Oct 30 13:06:04.026986 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:04.026912    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e40d7a56-5869-4a42-b8a7-33c02604c302-workload-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-tbw9c\" (UID: \"e40d7a56-5869-4a42-b8a7-33c02604c302\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tbw9c"
Oct 30 13:06:04.027020 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:04.027001    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e40d7a56-5869-4a42-b8a7-33c02604c302-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-64d4cc998c-tbw9c\" (UID: \"e40d7a56-5869-4a42-b8a7-33c02604c302\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tbw9c"
Oct 30 13:06:04.027050 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:04.027033    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e40d7a56-5869-4a42-b8a7-33c02604c302-istio-token\") pod \"maas-default-gateway-openshift-default-64d4cc998c-tbw9c\" (UID: \"e40d7a56-5869-4a42-b8a7-33c02604c302\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tbw9c"
Oct 30 13:06:04.027110 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:04.027095    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e40d7a56-5869-4a42-b8a7-33c02604c302-credential-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-tbw9c\" (UID: \"e40d7a56-5869-4a42-b8a7-33c02604c302\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tbw9c"
Oct 30 13:06:04.027205 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:04.027190    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e40d7a56-5869-4a42-b8a7-33c02604c302-istio-podinfo\") pod \"maas-default-gateway-openshift-default-64d4cc998c-tbw9c\" (UID: \"e40d7a56-5869-4a42-b8a7-33c02604c302\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tbw9c"
Oct 30 13:06:04.027407 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:04.027393    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e40d7a56-5869-4a42-b8a7-33c02604c302-istio-envoy\") pod \"maas-default-gateway-openshift-default-64d4cc998c-tbw9c\" (UID: \"e40d7a56-5869-4a42-b8a7-33c02604c302\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tbw9c"
Oct 30 13:06:04.128414 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:04.128392    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txqdn\" (UniqueName: \"kubernetes.io/projected/e40d7a56-5869-4a42-b8a7-33c02604c302-kube-api-access-txqdn\") pod \"maas-default-gateway-openshift-default-64d4cc998c-tbw9c\" (UID: \"e40d7a56-5869-4a42-b8a7-33c02604c302\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tbw9c"
Oct 30 13:06:04.128541 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:04.128422    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e40d7a56-5869-4a42-b8a7-33c02604c302-workload-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-tbw9c\" (UID: \"e40d7a56-5869-4a42-b8a7-33c02604c302\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tbw9c"
Oct 30 13:06:04.128541 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:04.128450    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e40d7a56-5869-4a42-b8a7-33c02604c302-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-64d4cc998c-tbw9c\" (UID: \"e40d7a56-5869-4a42-b8a7-33c02604c302\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tbw9c"
Oct 30 13:06:04.128541 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:04.128474    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e40d7a56-5869-4a42-b8a7-33c02604c302-istio-token\") pod \"maas-default-gateway-openshift-default-64d4cc998c-tbw9c\" (UID: \"e40d7a56-5869-4a42-b8a7-33c02604c302\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tbw9c"
Oct 30 13:06:04.128541 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:04.128513    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e40d7a56-5869-4a42-b8a7-33c02604c302-credential-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-tbw9c\" (UID: \"e40d7a56-5869-4a42-b8a7-33c02604c302\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tbw9c"
Oct 30 13:06:04.128824 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:04.128563    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e40d7a56-5869-4a42-b8a7-33c02604c302-istio-podinfo\") pod \"maas-default-gateway-openshift-default-64d4cc998c-tbw9c\" (UID: \"e40d7a56-5869-4a42-b8a7-33c02604c302\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tbw9c"
Oct 30 13:06:04.128824 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:04.128628    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e40d7a56-5869-4a42-b8a7-33c02604c302-istio-envoy\") pod \"maas-default-gateway-openshift-default-64d4cc998c-tbw9c\" (UID: \"e40d7a56-5869-4a42-b8a7-33c02604c302\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tbw9c"
Oct 30 13:06:04.128824 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:04.128668    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e40d7a56-5869-4a42-b8a7-33c02604c302-istio-data\") pod \"maas-default-gateway-openshift-default-64d4cc998c-tbw9c\" (UID: \"e40d7a56-5869-4a42-b8a7-33c02604c302\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tbw9c"
Oct 30 13:06:04.128824 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:04.128712    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e40d7a56-5869-4a42-b8a7-33c02604c302-workload-certs\") pod \"maas-default-gateway-openshift-default-64d4cc998c-tbw9c\" (UID: \"e40d7a56-5869-4a42-b8a7-33c02604c302\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tbw9c"
Oct 30 13:06:04.129043 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:04.128877    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e40d7a56-5869-4a42-b8a7-33c02604c302-workload-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-tbw9c\" (UID: \"e40d7a56-5869-4a42-b8a7-33c02604c302\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tbw9c"
Oct 30 13:06:04.129043 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:04.129005    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e40d7a56-5869-4a42-b8a7-33c02604c302-workload-certs\") pod \"maas-default-gateway-openshift-default-64d4cc998c-tbw9c\" (UID: \"e40d7a56-5869-4a42-b8a7-33c02604c302\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tbw9c"
Oct 30 13:06:04.129140 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:04.129048    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e40d7a56-5869-4a42-b8a7-33c02604c302-credential-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-tbw9c\" (UID: \"e40d7a56-5869-4a42-b8a7-33c02604c302\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tbw9c"
Oct 30 13:06:04.129194 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:04.129137    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e40d7a56-5869-4a42-b8a7-33c02604c302-istio-data\") pod \"maas-default-gateway-openshift-default-64d4cc998c-tbw9c\" (UID: \"e40d7a56-5869-4a42-b8a7-33c02604c302\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tbw9c"
Oct 30 13:06:04.129287 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:04.129270    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e40d7a56-5869-4a42-b8a7-33c02604c302-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-64d4cc998c-tbw9c\" (UID: \"e40d7a56-5869-4a42-b8a7-33c02604c302\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tbw9c"
Oct 30 13:06:04.130888 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:04.130867    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e40d7a56-5869-4a42-b8a7-33c02604c302-istio-envoy\") pod \"maas-default-gateway-openshift-default-64d4cc998c-tbw9c\" (UID: \"e40d7a56-5869-4a42-b8a7-33c02604c302\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tbw9c"
Oct 30 13:06:04.131217 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:04.131200    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e40d7a56-5869-4a42-b8a7-33c02604c302-istio-podinfo\") pod \"maas-default-gateway-openshift-default-64d4cc998c-tbw9c\" (UID: \"e40d7a56-5869-4a42-b8a7-33c02604c302\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tbw9c"
Oct 30 13:06:04.153899 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:04.153878    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txqdn\" (UniqueName: \"kubernetes.io/projected/e40d7a56-5869-4a42-b8a7-33c02604c302-kube-api-access-txqdn\") pod \"maas-default-gateway-openshift-default-64d4cc998c-tbw9c\" (UID: \"e40d7a56-5869-4a42-b8a7-33c02604c302\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tbw9c"
Oct 30 13:06:04.154005 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:04.153974    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e40d7a56-5869-4a42-b8a7-33c02604c302-istio-token\") pod \"maas-default-gateway-openshift-default-64d4cc998c-tbw9c\" (UID: \"e40d7a56-5869-4a42-b8a7-33c02604c302\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tbw9c"
Oct 30 13:06:04.248498 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:04.248478    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tbw9c"
Oct 30 13:06:04.556378 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:04.556307    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf4d32e-vvmq5"
Oct 30 13:06:04.556378 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:04.556319    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf4d32e-vvmq5" event={"ID":"43d5c676-c1f0-4dfc-9ef3-d027f6c9fd2c","Type":"ContainerDied","Data":"41012ffb824f3e0d25b353103b96f35b7642f60e015b1ccdf3f028fb620caa44"}
Oct 30 13:06:04.556378 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:04.556362    4364 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41012ffb824f3e0d25b353103b96f35b7642f60e015b1ccdf3f028fb620caa44"
Oct 30 13:06:04.580970 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:06:04.580945    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode40d7a56_5869_4a42_b8a7_33c02604c302.slice/crio-950326c8f13c13a7bebeef5f6af1ceea2838a674527635afecb36e47f637c0da WatchSource:0}: Error finding container 950326c8f13c13a7bebeef5f6af1ceea2838a674527635afecb36e47f637c0da: Status 404 returned error can't find the container with id 950326c8f13c13a7bebeef5f6af1ceea2838a674527635afecb36e47f637c0da
Oct 30 13:06:04.581071 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:04.581018    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tbw9c"]
Oct 30 13:06:04.583308 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:04.583280    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:06:04.583420 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:04.583350    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:06:04.583420 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:04.583394    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:06:04.732714 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:04.732684    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:06:05.595622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:05.588212    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tbw9c" event={"ID":"e40d7a56-5869-4a42-b8a7-33c02604c302","Type":"ContainerStarted","Data":"4c178610f8dc36de5b4ab212babda50a030c859a9dcabee7d77a761fef593b69"}
Oct 30 13:06:05.595622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:05.588317    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tbw9c" event={"ID":"e40d7a56-5869-4a42-b8a7-33c02604c302","Type":"ContainerStarted","Data":"950326c8f13c13a7bebeef5f6af1ceea2838a674527635afecb36e47f637c0da"}
Oct 30 13:06:06.261809 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:06.257666    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tbw9c"
Oct 30 13:06:06.268661 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:06.267708    4364 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-tbw9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 13:06:06.268661 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:06.267774    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tbw9c" podUID="e40d7a56-5869-4a42-b8a7-33c02604c302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 13:06:07.267625 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:07.236226    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz"
Oct 30 13:06:07.313665 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:07.268431    4364 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-tbw9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 13:06:07.313665 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:07.294691    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tbw9c" podUID="e40d7a56-5869-4a42-b8a7-33c02604c302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 13:06:08.166093 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:08.166051    4364 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz" podUID="bc5caf69-a649-47e5-8910-9b8f331dfd4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:06:08.256638 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:08.251939    4364 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-tbw9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 13:06:08.256638 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:08.252008    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tbw9c" podUID="e40d7a56-5869-4a42-b8a7-33c02604c302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 13:06:09.249784 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:09.249743    4364 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-tbw9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 13:06:09.250155 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:09.249790    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tbw9c" podUID="e40d7a56-5869-4a42-b8a7-33c02604c302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 13:06:10.249542 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:10.249508    4364 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-tbw9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 13:06:10.249732 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:10.249574    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tbw9c" podUID="e40d7a56-5869-4a42-b8a7-33c02604c302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 13:06:11.249131 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:11.249099    4364 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-tbw9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 13:06:11.249525 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:11.249144    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tbw9c" podUID="e40d7a56-5869-4a42-b8a7-33c02604c302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 13:06:11.503006 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:06:11.502921    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:06:12.249452 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:12.249420    4364 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-tbw9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 13:06:12.249936 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:12.249477    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tbw9c" podUID="e40d7a56-5869-4a42-b8a7-33c02604c302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 13:06:13.845830 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:13.845793    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tbw9c"
Oct 30 13:06:13.846926 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:13.846897    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tbw9c"
Oct 30 13:06:13.847358 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:13.847328    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tbw9c"
Oct 30 13:06:13.961778 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:13.961734    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tbw9c" podStartSLOduration=10.96172238 podStartE2EDuration="10.96172238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:05.664904775 +0000 UTC m=+864.831269423" watchObservedRunningTime="2026-10-30 13:06:13.96172238 +0000 UTC m=+873.128087027"
Oct 30 13:06:17.153804 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:17.153751    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz"
Oct 30 13:06:17.164957 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:17.164932    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz"
Oct 30 13:06:18.982426 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:18.982394    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-zl7wp"]
Oct 30 13:06:18.987744 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:18.987721    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-zl7wp"
Oct 30 13:06:19.022274 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:19.022249    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-zl7wp"]
Oct 30 13:06:19.146998 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:19.146976    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e3c6b89-88e3-460f-8d61-ca9438b25ddb-credential-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-zl7wp\" (UID: \"3e3c6b89-88e3-460f-8d61-ca9438b25ddb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-zl7wp"
Oct 30 13:06:19.147129 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:19.147011    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e3c6b89-88e3-460f-8d61-ca9438b25ddb-workload-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-zl7wp\" (UID: \"3e3c6b89-88e3-460f-8d61-ca9438b25ddb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-zl7wp"
Oct 30 13:06:19.147129 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:19.147045    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e3c6b89-88e3-460f-8d61-ca9438b25ddb-istio-token\") pod \"maas-default-gateway-openshift-default-64d4cc998c-zl7wp\" (UID: \"3e3c6b89-88e3-460f-8d61-ca9438b25ddb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-zl7wp"
Oct 30 13:06:19.147207 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:19.147182    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e3c6b89-88e3-460f-8d61-ca9438b25ddb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-64d4cc998c-zl7wp\" (UID: \"3e3c6b89-88e3-460f-8d61-ca9438b25ddb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-zl7wp"
Oct 30 13:06:19.147242 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:19.147216    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e3c6b89-88e3-460f-8d61-ca9438b25ddb-istio-data\") pod \"maas-default-gateway-openshift-default-64d4cc998c-zl7wp\" (UID: \"3e3c6b89-88e3-460f-8d61-ca9438b25ddb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-zl7wp"
Oct 30 13:06:19.147299 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:19.147285    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h45dg\" (UniqueName: \"kubernetes.io/projected/3e3c6b89-88e3-460f-8d61-ca9438b25ddb-kube-api-access-h45dg\") pod \"maas-default-gateway-openshift-default-64d4cc998c-zl7wp\" (UID: \"3e3c6b89-88e3-460f-8d61-ca9438b25ddb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-zl7wp"
Oct 30 13:06:19.147335 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:19.147317    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e3c6b89-88e3-460f-8d61-ca9438b25ddb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-64d4cc998c-zl7wp\" (UID: \"3e3c6b89-88e3-460f-8d61-ca9438b25ddb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-zl7wp"
Oct 30 13:06:19.147368 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:19.147335    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e3c6b89-88e3-460f-8d61-ca9438b25ddb-workload-certs\") pod \"maas-default-gateway-openshift-default-64d4cc998c-zl7wp\" (UID: \"3e3c6b89-88e3-460f-8d61-ca9438b25ddb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-zl7wp"
Oct 30 13:06:19.147368 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:19.147353    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e3c6b89-88e3-460f-8d61-ca9438b25ddb-istio-envoy\") pod \"maas-default-gateway-openshift-default-64d4cc998c-zl7wp\" (UID: \"3e3c6b89-88e3-460f-8d61-ca9438b25ddb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-zl7wp"
Oct 30 13:06:19.248238 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:19.248181    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e3c6b89-88e3-460f-8d61-ca9438b25ddb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-64d4cc998c-zl7wp\" (UID: \"3e3c6b89-88e3-460f-8d61-ca9438b25ddb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-zl7wp"
Oct 30 13:06:19.248238 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:19.248211    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e3c6b89-88e3-460f-8d61-ca9438b25ddb-workload-certs\") pod \"maas-default-gateway-openshift-default-64d4cc998c-zl7wp\" (UID: \"3e3c6b89-88e3-460f-8d61-ca9438b25ddb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-zl7wp"
Oct 30 13:06:19.248238 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:19.248234    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e3c6b89-88e3-460f-8d61-ca9438b25ddb-istio-envoy\") pod \"maas-default-gateway-openshift-default-64d4cc998c-zl7wp\" (UID: \"3e3c6b89-88e3-460f-8d61-ca9438b25ddb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-zl7wp"
Oct 30 13:06:19.248552 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:19.248280    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e3c6b89-88e3-460f-8d61-ca9438b25ddb-credential-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-zl7wp\" (UID: \"3e3c6b89-88e3-460f-8d61-ca9438b25ddb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-zl7wp"
Oct 30 13:06:19.248552 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:19.248323    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e3c6b89-88e3-460f-8d61-ca9438b25ddb-workload-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-zl7wp\" (UID: \"3e3c6b89-88e3-460f-8d61-ca9438b25ddb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-zl7wp"
Oct 30 13:06:19.248552 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:19.248366    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e3c6b89-88e3-460f-8d61-ca9438b25ddb-istio-token\") pod \"maas-default-gateway-openshift-default-64d4cc998c-zl7wp\" (UID: \"3e3c6b89-88e3-460f-8d61-ca9438b25ddb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-zl7wp"
Oct 30 13:06:19.248552 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:19.248422    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e3c6b89-88e3-460f-8d61-ca9438b25ddb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-64d4cc998c-zl7wp\" (UID: \"3e3c6b89-88e3-460f-8d61-ca9438b25ddb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-zl7wp"
Oct 30 13:06:19.248812 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:19.248683    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e3c6b89-88e3-460f-8d61-ca9438b25ddb-workload-certs\") pod \"maas-default-gateway-openshift-default-64d4cc998c-zl7wp\" (UID: \"3e3c6b89-88e3-460f-8d61-ca9438b25ddb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-zl7wp"
Oct 30 13:06:19.248812 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:19.248724    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e3c6b89-88e3-460f-8d61-ca9438b25ddb-credential-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-zl7wp\" (UID: \"3e3c6b89-88e3-460f-8d61-ca9438b25ddb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-zl7wp"
Oct 30 13:06:19.248812 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:19.248739    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e3c6b89-88e3-460f-8d61-ca9438b25ddb-istio-data\") pod \"maas-default-gateway-openshift-default-64d4cc998c-zl7wp\" (UID: \"3e3c6b89-88e3-460f-8d61-ca9438b25ddb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-zl7wp"
Oct 30 13:06:19.248812 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:19.248814    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h45dg\" (UniqueName: \"kubernetes.io/projected/3e3c6b89-88e3-460f-8d61-ca9438b25ddb-kube-api-access-h45dg\") pod \"maas-default-gateway-openshift-default-64d4cc998c-zl7wp\" (UID: \"3e3c6b89-88e3-460f-8d61-ca9438b25ddb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-zl7wp"
Oct 30 13:06:19.248812 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:19.248820    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e3c6b89-88e3-460f-8d61-ca9438b25ddb-workload-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-zl7wp\" (UID: \"3e3c6b89-88e3-460f-8d61-ca9438b25ddb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-zl7wp"
Oct 30 13:06:19.248812 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:19.248991    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e3c6b89-88e3-460f-8d61-ca9438b25ddb-istio-data\") pod \"maas-default-gateway-openshift-default-64d4cc998c-zl7wp\" (UID: \"3e3c6b89-88e3-460f-8d61-ca9438b25ddb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-zl7wp"
Oct 30 13:06:19.249474 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:19.249167    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e3c6b89-88e3-460f-8d61-ca9438b25ddb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-64d4cc998c-zl7wp\" (UID: \"3e3c6b89-88e3-460f-8d61-ca9438b25ddb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-zl7wp"
Oct 30 13:06:19.251146 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:19.251124    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e3c6b89-88e3-460f-8d61-ca9438b25ddb-istio-envoy\") pod \"maas-default-gateway-openshift-default-64d4cc998c-zl7wp\" (UID: \"3e3c6b89-88e3-460f-8d61-ca9438b25ddb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-zl7wp"
Oct 30 13:06:19.251249 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:19.251239    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e3c6b89-88e3-460f-8d61-ca9438b25ddb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-64d4cc998c-zl7wp\" (UID: \"3e3c6b89-88e3-460f-8d61-ca9438b25ddb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-zl7wp"
Oct 30 13:06:19.274950 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:19.274925    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e3c6b89-88e3-460f-8d61-ca9438b25ddb-istio-token\") pod \"maas-default-gateway-openshift-default-64d4cc998c-zl7wp\" (UID: \"3e3c6b89-88e3-460f-8d61-ca9438b25ddb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-zl7wp"
Oct 30 13:06:19.275095 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:19.275076    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h45dg\" (UniqueName: \"kubernetes.io/projected/3e3c6b89-88e3-460f-8d61-ca9438b25ddb-kube-api-access-h45dg\") pod \"maas-default-gateway-openshift-default-64d4cc998c-zl7wp\" (UID: \"3e3c6b89-88e3-460f-8d61-ca9438b25ddb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-zl7wp"
Oct 30 13:06:19.306070 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:19.306050    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-zl7wp"
Oct 30 13:06:19.623877 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:06:19.623838    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e3c6b89_88e3_460f_8d61_ca9438b25ddb.slice/crio-075255871c1fcb97cb9a09b0d719245a8baeff0be8c9ebbc8962c01c2fc4ed2d WatchSource:0}: Error finding container 075255871c1fcb97cb9a09b0d719245a8baeff0be8c9ebbc8962c01c2fc4ed2d: Status 404 returned error can't find the container with id 075255871c1fcb97cb9a09b0d719245a8baeff0be8c9ebbc8962c01c2fc4ed2d
Oct 30 13:06:19.624495 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:19.624440    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-zl7wp"]
Oct 30 13:06:19.626237 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:19.626204    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:06:19.626375 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:19.626282    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:06:19.626375 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:19.626332    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:06:19.910218 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:19.910111    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-zl7wp" event={"ID":"3e3c6b89-88e3-460f-8d61-ca9438b25ddb","Type":"ContainerStarted","Data":"4f73548b61d2e5b2df82d766d1e87d97da56a53da7928db1c7c92c14a466f03b"}
Oct 30 13:06:19.910218 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:19.910198    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-zl7wp" event={"ID":"3e3c6b89-88e3-460f-8d61-ca9438b25ddb","Type":"ContainerStarted","Data":"075255871c1fcb97cb9a09b0d719245a8baeff0be8c9ebbc8962c01c2fc4ed2d"}
Oct 30 13:06:19.961788 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:19.961742    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-zl7wp" podStartSLOduration=1.961727829 podStartE2EDuration="1.961727829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:19.958199493 +0000 UTC m=+879.124564140" watchObservedRunningTime="2026-10-30 13:06:19.961727829 +0000 UTC m=+879.128092474"
Oct 30 13:06:20.308749 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:20.308649    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-zl7wp"
Oct 30 13:06:20.309234 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:20.308794    4364 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-zl7wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 13:06:20.309234 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:20.308861    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-zl7wp" podUID="3e3c6b89-88e3-460f-8d61-ca9438b25ddb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 13:06:21.307238 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:21.307203    4364 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-zl7wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 13:06:21.307458 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:21.307263    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-zl7wp" podUID="3e3c6b89-88e3-460f-8d61-ca9438b25ddb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 13:06:22.315634 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:22.315006    4364 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-zl7wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 13:06:22.315634 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:22.315083    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-zl7wp" podUID="3e3c6b89-88e3-460f-8d61-ca9438b25ddb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 13:06:22.886622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:22.882644    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4"]
Oct 30 13:06:23.108040 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:23.108001    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4"
Oct 30 13:06:23.137614 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:23.135643    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-d7e62b\"/\"openshift-service-ca.crt\""
Oct 30 13:06:23.152886 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:23.152849    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-d7e62b\"/\"distinct-w4-d7e62b-kserve-self-signed-certs\""
Oct 30 13:06:23.153149 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:23.153130    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-d7e62b\"/\"kube-root-ca.crt\""
Oct 30 13:06:23.153371 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:23.153354    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-d7e62b\"/\"default-dockercfg-x76nv\""
Oct 30 13:06:23.231082 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:23.231055    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4"]
Oct 30 13:06:23.248139 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:23.246679    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cbe6af7-d884-4613-90b8-192e5ffdb027-model-cache\") pod \"distinct-w4-d7e62b-kserve-577bbb658c-2gch4\" (UID: \"0cbe6af7-d884-4613-90b8-192e5ffdb027\") " pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4"
Oct 30 13:06:23.248139 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:23.246760    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cbe6af7-d884-4613-90b8-192e5ffdb027-tls-certs\") pod \"distinct-w4-d7e62b-kserve-577bbb658c-2gch4\" (UID: \"0cbe6af7-d884-4613-90b8-192e5ffdb027\") " pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4"
Oct 30 13:06:23.248139 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:23.246897    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f4lk\" (UniqueName: \"kubernetes.io/projected/0cbe6af7-d884-4613-90b8-192e5ffdb027-kube-api-access-4f4lk\") pod \"distinct-w4-d7e62b-kserve-577bbb658c-2gch4\" (UID: \"0cbe6af7-d884-4613-90b8-192e5ffdb027\") " pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4"
Oct 30 13:06:23.248139 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:23.246990    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cbe6af7-d884-4613-90b8-192e5ffdb027-dshm\") pod \"distinct-w4-d7e62b-kserve-577bbb658c-2gch4\" (UID: \"0cbe6af7-d884-4613-90b8-192e5ffdb027\") " pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4"
Oct 30 13:06:23.248139 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:23.247057    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cbe6af7-d884-4613-90b8-192e5ffdb027-home\") pod \"distinct-w4-d7e62b-kserve-577bbb658c-2gch4\" (UID: \"0cbe6af7-d884-4613-90b8-192e5ffdb027\") " pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4"
Oct 30 13:06:23.248139 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:23.247098    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cbe6af7-d884-4613-90b8-192e5ffdb027-tmp-dir\") pod \"distinct-w4-d7e62b-kserve-577bbb658c-2gch4\" (UID: \"0cbe6af7-d884-4613-90b8-192e5ffdb027\") " pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4"
Oct 30 13:06:23.306660 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:23.306623    4364 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-zl7wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 13:06:23.306805 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:23.306686    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-zl7wp" podUID="3e3c6b89-88e3-460f-8d61-ca9438b25ddb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 13:06:23.348182 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:23.348152    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cbe6af7-d884-4613-90b8-192e5ffdb027-dshm\") pod \"distinct-w4-d7e62b-kserve-577bbb658c-2gch4\" (UID: \"0cbe6af7-d884-4613-90b8-192e5ffdb027\") " pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4"
Oct 30 13:06:23.348555 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:23.348203    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cbe6af7-d884-4613-90b8-192e5ffdb027-home\") pod \"distinct-w4-d7e62b-kserve-577bbb658c-2gch4\" (UID: \"0cbe6af7-d884-4613-90b8-192e5ffdb027\") " pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4"
Oct 30 13:06:23.348555 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:23.348320    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cbe6af7-d884-4613-90b8-192e5ffdb027-tmp-dir\") pod \"distinct-w4-d7e62b-kserve-577bbb658c-2gch4\" (UID: \"0cbe6af7-d884-4613-90b8-192e5ffdb027\") " pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4"
Oct 30 13:06:23.348555 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:23.348377    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cbe6af7-d884-4613-90b8-192e5ffdb027-model-cache\") pod \"distinct-w4-d7e62b-kserve-577bbb658c-2gch4\" (UID: \"0cbe6af7-d884-4613-90b8-192e5ffdb027\") " pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4"
Oct 30 13:06:23.348555 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:23.348449    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cbe6af7-d884-4613-90b8-192e5ffdb027-tls-certs\") pod \"distinct-w4-d7e62b-kserve-577bbb658c-2gch4\" (UID: \"0cbe6af7-d884-4613-90b8-192e5ffdb027\") " pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4"
Oct 30 13:06:23.348797 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:23.348555    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cbe6af7-d884-4613-90b8-192e5ffdb027-home\") pod \"distinct-w4-d7e62b-kserve-577bbb658c-2gch4\" (UID: \"0cbe6af7-d884-4613-90b8-192e5ffdb027\") " pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4"
Oct 30 13:06:23.348797 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:23.348560    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f4lk\" (UniqueName: \"kubernetes.io/projected/0cbe6af7-d884-4613-90b8-192e5ffdb027-kube-api-access-4f4lk\") pod \"distinct-w4-d7e62b-kserve-577bbb658c-2gch4\" (UID: \"0cbe6af7-d884-4613-90b8-192e5ffdb027\") " pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4"
Oct 30 13:06:23.348797 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:23.348684    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cbe6af7-d884-4613-90b8-192e5ffdb027-tmp-dir\") pod \"distinct-w4-d7e62b-kserve-577bbb658c-2gch4\" (UID: \"0cbe6af7-d884-4613-90b8-192e5ffdb027\") " pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4"
Oct 30 13:06:23.348797 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:23.348735    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cbe6af7-d884-4613-90b8-192e5ffdb027-model-cache\") pod \"distinct-w4-d7e62b-kserve-577bbb658c-2gch4\" (UID: \"0cbe6af7-d884-4613-90b8-192e5ffdb027\") " pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4"
Oct 30 13:06:23.350478 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:23.350455    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cbe6af7-d884-4613-90b8-192e5ffdb027-dshm\") pod \"distinct-w4-d7e62b-kserve-577bbb658c-2gch4\" (UID: \"0cbe6af7-d884-4613-90b8-192e5ffdb027\") " pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4"
Oct 30 13:06:23.350810 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:23.350789    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cbe6af7-d884-4613-90b8-192e5ffdb027-tls-certs\") pod \"distinct-w4-d7e62b-kserve-577bbb658c-2gch4\" (UID: \"0cbe6af7-d884-4613-90b8-192e5ffdb027\") " pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4"
Oct 30 13:06:23.373701 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:23.373674    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f4lk\" (UniqueName: \"kubernetes.io/projected/0cbe6af7-d884-4613-90b8-192e5ffdb027-kube-api-access-4f4lk\") pod \"distinct-w4-d7e62b-kserve-577bbb658c-2gch4\" (UID: \"0cbe6af7-d884-4613-90b8-192e5ffdb027\") " pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4"
Oct 30 13:06:23.527071 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:23.526993    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4"
Oct 30 13:06:23.872427 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:23.872396    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4"]
Oct 30 13:06:23.872824 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:06:23.872798    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0cbe6af7_d884_4613_90b8_192e5ffdb027.slice/crio-afe7bb8a4b232148ec5e82d2f6eee1fc9b894750d5282ea32ee6da7bb1559a9c WatchSource:0}: Error finding container afe7bb8a4b232148ec5e82d2f6eee1fc9b894750d5282ea32ee6da7bb1559a9c: Status 404 returned error can't find the container with id afe7bb8a4b232148ec5e82d2f6eee1fc9b894750d5282ea32ee6da7bb1559a9c
Oct 30 13:06:24.111904 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:24.111868    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4" event={"ID":"0cbe6af7-d884-4613-90b8-192e5ffdb027","Type":"ContainerStarted","Data":"1e9f15efe880081793ca793be333bbf83b44adcd393205596ea5e290842a2fa4"}
Oct 30 13:06:24.111904 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:24.111915    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4" event={"ID":"0cbe6af7-d884-4613-90b8-192e5ffdb027","Type":"ContainerStarted","Data":"afe7bb8a4b232148ec5e82d2f6eee1fc9b894750d5282ea32ee6da7bb1559a9c"}
Oct 30 13:06:24.161375 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:24.161292    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4" podStartSLOduration=2.161278209 podStartE2EDuration="2.161278209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:24.157259302 +0000 UTC m=+883.323623950" watchObservedRunningTime="2026-10-30 13:06:24.161278209 +0000 UTC m=+883.327642856"
Oct 30 13:06:24.311108 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:24.310648    4364 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-zl7wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 13:06:24.311108 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:24.310721    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-zl7wp" podUID="3e3c6b89-88e3-460f-8d61-ca9438b25ddb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 13:06:24.527621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:24.527527    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4"
Oct 30 13:06:25.307131 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:25.307096    4364 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-zl7wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 13:06:25.307308 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:25.307152    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-zl7wp" podUID="3e3c6b89-88e3-460f-8d61-ca9438b25ddb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 13:06:26.306574 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:26.306541    4364 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-zl7wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 13:06:26.306979 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:26.306590    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-zl7wp" podUID="3e3c6b89-88e3-460f-8d61-ca9438b25ddb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 13:06:27.306986 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:27.306948    4364 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-zl7wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 13:06:27.307449 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:27.307003    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-zl7wp" podUID="3e3c6b89-88e3-460f-8d61-ca9438b25ddb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 13:06:28.459091 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:28.459056    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-zl7wp"
Oct 30 13:06:28.459573 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:28.459522    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-zl7wp"
Oct 30 13:06:28.460507 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:28.460482    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-zl7wp"
Oct 30 13:06:33.527608 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:33.527564    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4"
Oct 30 13:06:33.539976 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:33.539952    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4"
Oct 30 13:06:33.850132 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:33.850071    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:06:34.181853 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:34.181817    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4"
Oct 30 13:06:37.382398 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:37.382370    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m"]
Oct 30 13:06:37.386653 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:37.386637    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m"
Oct 30 13:06:37.423746 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:37.423720    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m"]
Oct 30 13:06:37.533107 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:37.533081    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b-istio-envoy\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m\" (UID: \"5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m"
Oct 30 13:06:37.533272 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:37.533118    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b-credential-socket\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m\" (UID: \"5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m"
Oct 30 13:06:37.533272 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:37.533136    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b-workload-certs\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m\" (UID: \"5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m"
Oct 30 13:06:37.533272 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:37.533253    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b-workload-socket\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m\" (UID: \"5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m"
Oct 30 13:06:37.533382 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:37.533301    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b-istio-token\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m\" (UID: \"5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m"
Oct 30 13:06:37.533418 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:37.533406    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b-istio-podinfo\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m\" (UID: \"5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m"
Oct 30 13:06:37.533451 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:37.533427    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5h9s\" (UniqueName: \"kubernetes.io/projected/5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b-kube-api-access-r5h9s\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m\" (UID: \"5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m"
Oct 30 13:06:37.533484 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:37.533471    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b-istiod-ca-cert\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m\" (UID: \"5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m"
Oct 30 13:06:37.533532 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:37.533518    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b-istio-data\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m\" (UID: \"5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m"
Oct 30 13:06:37.633936 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:37.633883    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b-istio-podinfo\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m\" (UID: \"5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m"
Oct 30 13:06:37.633936 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:37.633913    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5h9s\" (UniqueName: \"kubernetes.io/projected/5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b-kube-api-access-r5h9s\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m\" (UID: \"5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m"
Oct 30 13:06:37.634089 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:37.633952    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b-istiod-ca-cert\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m\" (UID: \"5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m"
Oct 30 13:06:37.634089 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:37.633982    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b-istio-data\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m\" (UID: \"5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m"
Oct 30 13:06:37.634089 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:37.634023    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b-istio-envoy\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m\" (UID: \"5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m"
Oct 30 13:06:37.634089 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:37.634056    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b-credential-socket\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m\" (UID: \"5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m"
Oct 30 13:06:37.634278 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:37.634182    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b-workload-certs\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m\" (UID: \"5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m"
Oct 30 13:06:37.634278 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:37.634252    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b-workload-socket\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m\" (UID: \"5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m"
Oct 30 13:06:37.634278 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:37.634273    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b-istio-token\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m\" (UID: \"5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m"
Oct 30 13:06:37.634445 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:37.634425    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b-credential-socket\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m\" (UID: \"5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m"
Oct 30 13:06:37.634592 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:37.634570    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b-istio-data\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m\" (UID: \"5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m"
Oct 30 13:06:37.634726 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:37.634678    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b-workload-socket\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m\" (UID: \"5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m"
Oct 30 13:06:37.634771 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:37.634736    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b-workload-certs\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m\" (UID: \"5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m"
Oct 30 13:06:37.635212 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:37.635190    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b-istiod-ca-cert\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m\" (UID: \"5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m"
Oct 30 13:06:37.636426 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:37.636398    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b-istio-envoy\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m\" (UID: \"5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m"
Oct 30 13:06:37.636659 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:37.636639    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b-istio-podinfo\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m\" (UID: \"5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m"
Oct 30 13:06:37.658792 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:37.658765    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b-istio-token\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m\" (UID: \"5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m"
Oct 30 13:06:37.659867 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:37.659847    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5h9s\" (UniqueName: \"kubernetes.io/projected/5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b-kube-api-access-r5h9s\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m\" (UID: \"5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m"
Oct 30 13:06:37.700386 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:37.700365    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m"
Oct 30 13:06:38.055872 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:38.055837    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m"]
Oct 30 13:06:38.061749 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:38.061702    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:06:38.061851 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:38.061764    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:06:38.061851 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:38.061796    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:06:38.198220 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:38.198189    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m" event={"ID":"5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b","Type":"ContainerStarted","Data":"a1f8f42527e9131515c99a903756f40202e9748f6eb33325be6d202a389d9798"}
Oct 30 13:06:38.198374 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:38.198248    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m" event={"ID":"5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b","Type":"ContainerStarted","Data":"3d65b03acd7ed56ba1441413bf978b8fb2f575ee319f04d6489532d28d1e745e"}
Oct 30 13:06:38.250289 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:38.250236    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m" podStartSLOduration=1.250222164 podStartE2EDuration="1.250222164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:38.247459266 +0000 UTC m=+897.413823914" watchObservedRunningTime="2026-10-30 13:06:38.250222164 +0000 UTC m=+897.416586811"
Oct 30 13:06:38.701449 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:38.701417    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m"
Oct 30 13:06:39.703638 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:39.702547    4364 patch_prober.go:29] interesting pod/e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 13:06:39.703638 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:39.702630    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m" podUID="5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 13:06:40.703996 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:40.703960    4364 patch_prober.go:29] interesting pod/e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 13:06:40.704481 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:40.704317    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m" podUID="5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 13:06:41.527667 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:06:41.527631    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:06:41.703975 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:41.703941    4364 patch_prober.go:29] interesting pod/e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 13:06:41.704139 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:41.704006    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m" podUID="5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 13:06:42.701036 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:42.700999    4364 patch_prober.go:29] interesting pod/e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 13:06:42.701299 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:42.701269    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m" podUID="5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 13:06:43.702012 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:43.701974    4364 patch_prober.go:29] interesting pod/e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 13:06:43.702662 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:43.702629    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m" podUID="5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 13:06:44.706322 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:44.706294    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m"
Oct 30 13:06:44.706846 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:44.706816    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m"
Oct 30 13:06:44.707515 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:44.707496    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m"
Oct 30 13:06:51.749573 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:51.749540    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz"]
Oct 30 13:06:51.754664 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:51.754646    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz"
Oct 30 13:06:51.781978 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:51.781948    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz"]
Oct 30 13:06:51.791919 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:51.791898    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f07dfe\"/\"distinct-2-w0-f07dfe-kserve-self-signed-certs\""
Oct 30 13:06:51.839550 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:51.839528    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a568f1ce-602d-4069-a41c-9fe1a7635c5a-home\") pod \"distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz\" (UID: \"a568f1ce-602d-4069-a41c-9fe1a7635c5a\") " pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz"
Oct 30 13:06:51.839680 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:51.839586    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6jnz\" (UniqueName: \"kubernetes.io/projected/a568f1ce-602d-4069-a41c-9fe1a7635c5a-kube-api-access-n6jnz\") pod \"distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz\" (UID: \"a568f1ce-602d-4069-a41c-9fe1a7635c5a\") " pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz"
Oct 30 13:06:51.839761 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:51.839742    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a568f1ce-602d-4069-a41c-9fe1a7635c5a-tls-certs\") pod \"distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz\" (UID: \"a568f1ce-602d-4069-a41c-9fe1a7635c5a\") " pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz"
Oct 30 13:06:51.839805 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:51.839771    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a568f1ce-602d-4069-a41c-9fe1a7635c5a-tmp-dir\") pod \"distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz\" (UID: \"a568f1ce-602d-4069-a41c-9fe1a7635c5a\") " pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz"
Oct 30 13:06:51.839805 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:51.839799    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a568f1ce-602d-4069-a41c-9fe1a7635c5a-dshm\") pod \"distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz\" (UID: \"a568f1ce-602d-4069-a41c-9fe1a7635c5a\") " pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz"
Oct 30 13:06:51.839887 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:51.839873    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a568f1ce-602d-4069-a41c-9fe1a7635c5a-model-cache\") pod \"distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz\" (UID: \"a568f1ce-602d-4069-a41c-9fe1a7635c5a\") " pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz"
Oct 30 13:06:51.950622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:51.945318    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a568f1ce-602d-4069-a41c-9fe1a7635c5a-home\") pod \"distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz\" (UID: \"a568f1ce-602d-4069-a41c-9fe1a7635c5a\") " pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz"
Oct 30 13:06:51.950622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:51.945400    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6jnz\" (UniqueName: \"kubernetes.io/projected/a568f1ce-602d-4069-a41c-9fe1a7635c5a-kube-api-access-n6jnz\") pod \"distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz\" (UID: \"a568f1ce-602d-4069-a41c-9fe1a7635c5a\") " pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz"
Oct 30 13:06:51.950622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:51.945498    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a568f1ce-602d-4069-a41c-9fe1a7635c5a-tls-certs\") pod \"distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz\" (UID: \"a568f1ce-602d-4069-a41c-9fe1a7635c5a\") " pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz"
Oct 30 13:06:51.950622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:51.945524    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a568f1ce-602d-4069-a41c-9fe1a7635c5a-tmp-dir\") pod \"distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz\" (UID: \"a568f1ce-602d-4069-a41c-9fe1a7635c5a\") " pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz"
Oct 30 13:06:51.950622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:51.945565    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a568f1ce-602d-4069-a41c-9fe1a7635c5a-dshm\") pod \"distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz\" (UID: \"a568f1ce-602d-4069-a41c-9fe1a7635c5a\") " pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz"
Oct 30 13:06:51.950622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:51.945627    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a568f1ce-602d-4069-a41c-9fe1a7635c5a-model-cache\") pod \"distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz\" (UID: \"a568f1ce-602d-4069-a41c-9fe1a7635c5a\") " pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz"
Oct 30 13:06:51.950622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:51.946135    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a568f1ce-602d-4069-a41c-9fe1a7635c5a-model-cache\") pod \"distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz\" (UID: \"a568f1ce-602d-4069-a41c-9fe1a7635c5a\") " pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz"
Oct 30 13:06:51.950622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:51.946391    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a568f1ce-602d-4069-a41c-9fe1a7635c5a-home\") pod \"distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz\" (UID: \"a568f1ce-602d-4069-a41c-9fe1a7635c5a\") " pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz"
Oct 30 13:06:51.967656 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:51.957098    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a568f1ce-602d-4069-a41c-9fe1a7635c5a-tmp-dir\") pod \"distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz\" (UID: \"a568f1ce-602d-4069-a41c-9fe1a7635c5a\") " pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz"
Oct 30 13:06:51.976132 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:51.976091    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a568f1ce-602d-4069-a41c-9fe1a7635c5a-tls-certs\") pod \"distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz\" (UID: \"a568f1ce-602d-4069-a41c-9fe1a7635c5a\") " pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz"
Oct 30 13:06:51.978326 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:51.978272    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6jnz\" (UniqueName: \"kubernetes.io/projected/a568f1ce-602d-4069-a41c-9fe1a7635c5a-kube-api-access-n6jnz\") pod \"distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz\" (UID: \"a568f1ce-602d-4069-a41c-9fe1a7635c5a\") " pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz"
Oct 30 13:06:51.978895 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:51.978872    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a568f1ce-602d-4069-a41c-9fe1a7635c5a-dshm\") pod \"distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz\" (UID: \"a568f1ce-602d-4069-a41c-9fe1a7635c5a\") " pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz"
Oct 30 13:06:52.066653 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:52.066553    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz"
Oct 30 13:06:52.373924 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:06:52.373902    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda568f1ce_602d_4069_a41c_9fe1a7635c5a.slice/crio-738942d921c36bae35b0f7772c729ec21be5dc90a01d975b9d83a4d0c477e406 WatchSource:0}: Error finding container 738942d921c36bae35b0f7772c729ec21be5dc90a01d975b9d83a4d0c477e406: Status 404 returned error can't find the container with id 738942d921c36bae35b0f7772c729ec21be5dc90a01d975b9d83a4d0c477e406
Oct 30 13:06:52.375349 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:52.375328    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz"]
Oct 30 13:06:53.307662 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:53.307627    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz" event={"ID":"a568f1ce-602d-4069-a41c-9fe1a7635c5a","Type":"ContainerStarted","Data":"50aab43258389aa70e33c69e44aad520f63075b8efac299e245b80b9365d852e"}
Oct 30 13:06:53.308044 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:53.307670    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz" event={"ID":"a568f1ce-602d-4069-a41c-9fe1a7635c5a","Type":"ContainerStarted","Data":"738942d921c36bae35b0f7772c729ec21be5dc90a01d975b9d83a4d0c477e406"}
Oct 30 13:06:53.360865 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:53.360457    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz" podStartSLOduration=2.360442534 podStartE2EDuration="2.360442534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:53.35778848 +0000 UTC m=+912.524153130" watchObservedRunningTime="2026-10-30 13:06:53.360442534 +0000 UTC m=+912.526807183"
Oct 30 13:06:53.394883 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:53.394854    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q"]
Oct 30 13:06:53.398852 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:53.398837    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q"
Oct 30 13:06:53.421915 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:53.421888    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q"]
Oct 30 13:06:53.568024 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:53.567943    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ab4d389-3115-40fe-aad2-137fed1ffe54-workload-certs\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q\" (UID: \"9ab4d389-3115-40fe-aad2-137fed1ffe54\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q"
Oct 30 13:06:53.568024 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:53.567993    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ab4d389-3115-40fe-aad2-137fed1ffe54-istio-data\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q\" (UID: \"9ab4d389-3115-40fe-aad2-137fed1ffe54\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q"
Oct 30 13:06:53.568204 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:53.568146    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ab4d389-3115-40fe-aad2-137fed1ffe54-workload-socket\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q\" (UID: \"9ab4d389-3115-40fe-aad2-137fed1ffe54\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q"
Oct 30 13:06:53.568279 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:53.568262    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ab4d389-3115-40fe-aad2-137fed1ffe54-istio-podinfo\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q\" (UID: \"9ab4d389-3115-40fe-aad2-137fed1ffe54\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q"
Oct 30 13:06:53.568326 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:53.568313    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ab4d389-3115-40fe-aad2-137fed1ffe54-istio-envoy\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q\" (UID: \"9ab4d389-3115-40fe-aad2-137fed1ffe54\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q"
Oct 30 13:06:53.568364 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:53.568337    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ab4d389-3115-40fe-aad2-137fed1ffe54-istio-token\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q\" (UID: \"9ab4d389-3115-40fe-aad2-137fed1ffe54\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q"
Oct 30 13:06:53.568397 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:53.568367    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkxpj\" (UniqueName: \"kubernetes.io/projected/9ab4d389-3115-40fe-aad2-137fed1ffe54-kube-api-access-gkxpj\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q\" (UID: \"9ab4d389-3115-40fe-aad2-137fed1ffe54\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q"
Oct 30 13:06:53.568430 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:53.568412    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ab4d389-3115-40fe-aad2-137fed1ffe54-credential-socket\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q\" (UID: \"9ab4d389-3115-40fe-aad2-137fed1ffe54\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q"
Oct 30 13:06:53.568466 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:53.568437    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ab4d389-3115-40fe-aad2-137fed1ffe54-istiod-ca-cert\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q\" (UID: \"9ab4d389-3115-40fe-aad2-137fed1ffe54\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q"
Oct 30 13:06:53.669845 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:53.669810    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ab4d389-3115-40fe-aad2-137fed1ffe54-istio-podinfo\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q\" (UID: \"9ab4d389-3115-40fe-aad2-137fed1ffe54\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q"
Oct 30 13:06:53.670005 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:53.669867    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ab4d389-3115-40fe-aad2-137fed1ffe54-istio-envoy\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q\" (UID: \"9ab4d389-3115-40fe-aad2-137fed1ffe54\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q"
Oct 30 13:06:53.670005 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:53.669895    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ab4d389-3115-40fe-aad2-137fed1ffe54-istio-token\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q\" (UID: \"9ab4d389-3115-40fe-aad2-137fed1ffe54\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q"
Oct 30 13:06:53.670005 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:53.669926    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkxpj\" (UniqueName: \"kubernetes.io/projected/9ab4d389-3115-40fe-aad2-137fed1ffe54-kube-api-access-gkxpj\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q\" (UID: \"9ab4d389-3115-40fe-aad2-137fed1ffe54\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q"
Oct 30 13:06:53.670005 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:53.669966    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ab4d389-3115-40fe-aad2-137fed1ffe54-credential-socket\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q\" (UID: \"9ab4d389-3115-40fe-aad2-137fed1ffe54\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q"
Oct 30 13:06:53.670005 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:53.669996    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ab4d389-3115-40fe-aad2-137fed1ffe54-istiod-ca-cert\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q\" (UID: \"9ab4d389-3115-40fe-aad2-137fed1ffe54\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q"
Oct 30 13:06:53.670257 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:53.670036    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ab4d389-3115-40fe-aad2-137fed1ffe54-workload-certs\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q\" (UID: \"9ab4d389-3115-40fe-aad2-137fed1ffe54\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q"
Oct 30 13:06:53.670257 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:53.670085    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ab4d389-3115-40fe-aad2-137fed1ffe54-istio-data\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q\" (UID: \"9ab4d389-3115-40fe-aad2-137fed1ffe54\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q"
Oct 30 13:06:53.670257 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:53.670184    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ab4d389-3115-40fe-aad2-137fed1ffe54-workload-socket\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q\" (UID: \"9ab4d389-3115-40fe-aad2-137fed1ffe54\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q"
Oct 30 13:06:53.670552 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:53.670481    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ab4d389-3115-40fe-aad2-137fed1ffe54-credential-socket\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q\" (UID: \"9ab4d389-3115-40fe-aad2-137fed1ffe54\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q"
Oct 30 13:06:53.670552 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:53.670534    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ab4d389-3115-40fe-aad2-137fed1ffe54-workload-certs\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q\" (UID: \"9ab4d389-3115-40fe-aad2-137fed1ffe54\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q"
Oct 30 13:06:53.670735 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:53.670628    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ab4d389-3115-40fe-aad2-137fed1ffe54-workload-socket\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q\" (UID: \"9ab4d389-3115-40fe-aad2-137fed1ffe54\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q"
Oct 30 13:06:53.670735 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:53.670669    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ab4d389-3115-40fe-aad2-137fed1ffe54-istio-data\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q\" (UID: \"9ab4d389-3115-40fe-aad2-137fed1ffe54\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q"
Oct 30 13:06:53.670864 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:53.670841    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ab4d389-3115-40fe-aad2-137fed1ffe54-istiod-ca-cert\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q\" (UID: \"9ab4d389-3115-40fe-aad2-137fed1ffe54\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q"
Oct 30 13:06:53.672262 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:53.672243    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ab4d389-3115-40fe-aad2-137fed1ffe54-istio-envoy\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q\" (UID: \"9ab4d389-3115-40fe-aad2-137fed1ffe54\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q"
Oct 30 13:06:53.672330 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:53.672312    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ab4d389-3115-40fe-aad2-137fed1ffe54-istio-podinfo\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q\" (UID: \"9ab4d389-3115-40fe-aad2-137fed1ffe54\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q"
Oct 30 13:06:53.694994 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:53.694971    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ab4d389-3115-40fe-aad2-137fed1ffe54-istio-token\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q\" (UID: \"9ab4d389-3115-40fe-aad2-137fed1ffe54\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q"
Oct 30 13:06:53.695397 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:53.695379    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkxpj\" (UniqueName: \"kubernetes.io/projected/9ab4d389-3115-40fe-aad2-137fed1ffe54-kube-api-access-gkxpj\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q\" (UID: \"9ab4d389-3115-40fe-aad2-137fed1ffe54\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q"
Oct 30 13:06:53.716376 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:53.716354    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q"
Oct 30 13:06:54.067629 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:54.067577    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz"
Oct 30 13:06:54.103495 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:06:54.103462    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ab4d389_3115_40fe_aad2_137fed1ffe54.slice/crio-b63321b011b3e335b150e584f84a14c12d5d8d03bb2a922d9ed742210a8cabef WatchSource:0}: Error finding container b63321b011b3e335b150e584f84a14c12d5d8d03bb2a922d9ed742210a8cabef: Status 404 returned error can't find the container with id b63321b011b3e335b150e584f84a14c12d5d8d03bb2a922d9ed742210a8cabef
Oct 30 13:06:54.104026 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:54.104006    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q"]
Oct 30 13:06:54.105812 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:54.105784    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:06:54.105900 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:54.105861    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:06:54.105900 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:54.105891    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:06:54.317230 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:54.317197    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q" event={"ID":"9ab4d389-3115-40fe-aad2-137fed1ffe54","Type":"ContainerStarted","Data":"ee39c9c380a7d8958155936c2358f5d26cdad6bf1e7a658b3bc56f220deb4e12"}
Oct 30 13:06:54.317750 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:54.317249    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q" event={"ID":"9ab4d389-3115-40fe-aad2-137fed1ffe54","Type":"ContainerStarted","Data":"b63321b011b3e335b150e584f84a14c12d5d8d03bb2a922d9ed742210a8cabef"}
Oct 30 13:06:54.368844 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:54.368803    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q" podStartSLOduration=1.368791485 podStartE2EDuration="1.368791485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:54.365975259 +0000 UTC m=+913.532339907" watchObservedRunningTime="2026-10-30 13:06:54.368791485 +0000 UTC m=+913.535156134"
Oct 30 13:06:54.717134 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:54.717043    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q"
Oct 30 13:06:55.718526 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:55.718480    4364 patch_prober.go:29] interesting pod/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 13:06:55.718962 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:55.718550    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q" podUID="9ab4d389-3115-40fe-aad2-137fed1ffe54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 13:06:56.717537 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:56.717501    4364 patch_prober.go:29] interesting pod/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 13:06:56.717738 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:56.717562    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q" podUID="9ab4d389-3115-40fe-aad2-137fed1ffe54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 13:06:57.717121 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:57.717084    4364 patch_prober.go:29] interesting pod/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 13:06:57.717524 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:57.717139    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q" podUID="9ab4d389-3115-40fe-aad2-137fed1ffe54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 13:06:58.717072 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:58.717036    4364 patch_prober.go:29] interesting pod/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 13:06:58.717248 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:58.717093    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q" podUID="9ab4d389-3115-40fe-aad2-137fed1ffe54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 13:06:59.219163 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:59.219127    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:06:59.717229 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:59.717195    4364 patch_prober.go:29] interesting pod/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 13:06:59.717431 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:06:59.717250    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q" podUID="9ab4d389-3115-40fe-aad2-137fed1ffe54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 13:07:00.716922 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:00.716884    4364 patch_prober.go:29] interesting pod/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 13:07:00.717125 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:00.716963    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q" podUID="9ab4d389-3115-40fe-aad2-137fed1ffe54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 13:07:01.721261 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:01.721234    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q"
Oct 30 13:07:01.721665 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:01.721624    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q"
Oct 30 13:07:01.722393 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:01.722373    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q"
Oct 30 13:07:02.067776 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:02.067706    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz"
Oct 30 13:07:02.079972 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:02.079945    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz"
Oct 30 13:07:02.438897 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:02.438876    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz"
Oct 30 13:07:11.504647 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:07:11.504578    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:07:18.991233 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:18.991196    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc"]
Oct 30 13:07:18.996552 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:18.996530    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc"
Oct 30 13:07:19.011963 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:19.011939    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc"]
Oct 30 13:07:19.022576 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:19.022550    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-e34fd5\"/\"distinct-2-w5-e34fd5-kserve-self-signed-certs\""
Oct 30 13:07:19.022714 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:19.022633    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-e34fd5\"/\"kube-root-ca.crt\""
Oct 30 13:07:19.027408 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:19.027389    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-e34fd5\"/\"default-dockercfg-vgqrm\""
Oct 30 13:07:19.027408 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:19.027393    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-e34fd5\"/\"openshift-service-ca.crt\""
Oct 30 13:07:19.046104 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:19.046080    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8911b790-331d-490c-876b-647e6d2fcd14-model-cache\") pod \"distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc\" (UID: \"8911b790-331d-490c-876b-647e6d2fcd14\") " pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc"
Oct 30 13:07:19.046287 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:19.046263    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8911b790-331d-490c-876b-647e6d2fcd14-tls-certs\") pod \"distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc\" (UID: \"8911b790-331d-490c-876b-647e6d2fcd14\") " pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc"
Oct 30 13:07:19.046438 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:19.046318    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8911b790-331d-490c-876b-647e6d2fcd14-tmp-dir\") pod \"distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc\" (UID: \"8911b790-331d-490c-876b-647e6d2fcd14\") " pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc"
Oct 30 13:07:19.046438 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:19.046360    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjf4m\" (UniqueName: \"kubernetes.io/projected/8911b790-331d-490c-876b-647e6d2fcd14-kube-api-access-jjf4m\") pod \"distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc\" (UID: \"8911b790-331d-490c-876b-647e6d2fcd14\") " pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc"
Oct 30 13:07:19.046683 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:19.046481    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8911b790-331d-490c-876b-647e6d2fcd14-home\") pod \"distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc\" (UID: \"8911b790-331d-490c-876b-647e6d2fcd14\") " pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc"
Oct 30 13:07:19.046683 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:19.046660    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8911b790-331d-490c-876b-647e6d2fcd14-dshm\") pod \"distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc\" (UID: \"8911b790-331d-490c-876b-647e6d2fcd14\") " pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc"
Oct 30 13:07:19.148227 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:19.148201    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8911b790-331d-490c-876b-647e6d2fcd14-tmp-dir\") pod \"distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc\" (UID: \"8911b790-331d-490c-876b-647e6d2fcd14\") " pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc"
Oct 30 13:07:19.148333 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:19.148235    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjf4m\" (UniqueName: \"kubernetes.io/projected/8911b790-331d-490c-876b-647e6d2fcd14-kube-api-access-jjf4m\") pod \"distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc\" (UID: \"8911b790-331d-490c-876b-647e6d2fcd14\") " pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc"
Oct 30 13:07:19.148333 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:19.148272    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8911b790-331d-490c-876b-647e6d2fcd14-home\") pod \"distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc\" (UID: \"8911b790-331d-490c-876b-647e6d2fcd14\") " pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc"
Oct 30 13:07:19.148333 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:19.148314    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8911b790-331d-490c-876b-647e6d2fcd14-dshm\") pod \"distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc\" (UID: \"8911b790-331d-490c-876b-647e6d2fcd14\") " pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc"
Oct 30 13:07:19.148465 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:19.148447    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8911b790-331d-490c-876b-647e6d2fcd14-model-cache\") pod \"distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc\" (UID: \"8911b790-331d-490c-876b-647e6d2fcd14\") " pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc"
Oct 30 13:07:19.148509 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:19.148479    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8911b790-331d-490c-876b-647e6d2fcd14-tls-certs\") pod \"distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc\" (UID: \"8911b790-331d-490c-876b-647e6d2fcd14\") " pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc"
Oct 30 13:07:19.148581 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:19.148558    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8911b790-331d-490c-876b-647e6d2fcd14-tmp-dir\") pod \"distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc\" (UID: \"8911b790-331d-490c-876b-647e6d2fcd14\") " pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc"
Oct 30 13:07:19.148729 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:19.148713    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8911b790-331d-490c-876b-647e6d2fcd14-home\") pod \"distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc\" (UID: \"8911b790-331d-490c-876b-647e6d2fcd14\") " pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc"
Oct 30 13:07:19.148820 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:19.148804    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8911b790-331d-490c-876b-647e6d2fcd14-model-cache\") pod \"distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc\" (UID: \"8911b790-331d-490c-876b-647e6d2fcd14\") " pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc"
Oct 30 13:07:19.150505 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:19.150485    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8911b790-331d-490c-876b-647e6d2fcd14-dshm\") pod \"distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc\" (UID: \"8911b790-331d-490c-876b-647e6d2fcd14\") " pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc"
Oct 30 13:07:19.150917 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:19.150894    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8911b790-331d-490c-876b-647e6d2fcd14-tls-certs\") pod \"distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc\" (UID: \"8911b790-331d-490c-876b-647e6d2fcd14\") " pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc"
Oct 30 13:07:19.197619 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:19.197528    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjf4m\" (UniqueName: \"kubernetes.io/projected/8911b790-331d-490c-876b-647e6d2fcd14-kube-api-access-jjf4m\") pod \"distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc\" (UID: \"8911b790-331d-490c-876b-647e6d2fcd14\") " pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc"
Oct 30 13:07:19.310275 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:19.310199    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc"
Oct 30 13:07:19.614811 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:07:19.614778    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8911b790_331d_490c_876b_647e6d2fcd14.slice/crio-ae22d2f1fa6147c813eb6a60c5a4ad43c4a86dd58d1bf02fbf82aab1e5bafe31 WatchSource:0}: Error finding container ae22d2f1fa6147c813eb6a60c5a4ad43c4a86dd58d1bf02fbf82aab1e5bafe31: Status 404 returned error can't find the container with id ae22d2f1fa6147c813eb6a60c5a4ad43c4a86dd58d1bf02fbf82aab1e5bafe31
Oct 30 13:07:19.615460 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:19.615440    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc"]
Oct 30 13:07:20.547733 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:20.547698    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc" event={"ID":"8911b790-331d-490c-876b-647e6d2fcd14","Type":"ContainerStarted","Data":"0cdd2ab96dce856d79cc9f91304a11358e3d0b285cca47ccf807a03004cf740b"}
Oct 30 13:07:20.548101 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:20.547741    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc" event={"ID":"8911b790-331d-490c-876b-647e6d2fcd14","Type":"ContainerStarted","Data":"ae22d2f1fa6147c813eb6a60c5a4ad43c4a86dd58d1bf02fbf82aab1e5bafe31"}
Oct 30 13:07:20.599179 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:20.599133    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc" podStartSLOduration=2.599119704 podStartE2EDuration="2.599119704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:20.596201463 +0000 UTC m=+939.762566111" watchObservedRunningTime="2026-10-30 13:07:20.599119704 +0000 UTC m=+939.765484351"
Oct 30 13:07:21.310392 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:21.310366    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc"
Oct 30 13:07:28.725649 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:28.725576    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:07:29.311345 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:29.311311    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc"
Oct 30 13:07:29.323687 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:29.323665    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc"
Oct 30 13:07:29.619392 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:29.619349    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc"
Oct 30 13:07:41.503442 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:07:41.503405    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:07:43.302383 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:43.302346    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8234e9d-qxpdh"]
Oct 30 13:07:43.307245 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:43.307217    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8234e9d-qxpdh"
Oct 30 13:07:43.348703 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:43.348669    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\""
Oct 30 13:07:43.350880 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:43.349061    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:07:43.373496 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:43.371084    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8234e9d-qxpdh"]
Oct 30 13:07:43.408375 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:43.408343    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b6051c6-a230-4df5-97d5-77401c83bf95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8234e9d-qxpdh\" (UID: \"2b6051c6-a230-4df5-97d5-77401c83bf95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8234e9d-qxpdh"
Oct 30 13:07:43.408525 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:43.408415    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b6051c6-a230-4df5-97d5-77401c83bf95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8234e9d-qxpdh\" (UID: \"2b6051c6-a230-4df5-97d5-77401c83bf95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8234e9d-qxpdh"
Oct 30 13:07:43.408632 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:43.408585    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv2bf\" (UniqueName: \"kubernetes.io/projected/2b6051c6-a230-4df5-97d5-77401c83bf95-kube-api-access-wv2bf\") pod \"maas-api-revoke-sub-f8234e9d-qxpdh\" (UID: \"2b6051c6-a230-4df5-97d5-77401c83bf95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8234e9d-qxpdh"
Oct 30 13:07:43.509733 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:43.509692    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b6051c6-a230-4df5-97d5-77401c83bf95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8234e9d-qxpdh\" (UID: \"2b6051c6-a230-4df5-97d5-77401c83bf95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8234e9d-qxpdh"
Oct 30 13:07:43.509898 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:43.509746    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b6051c6-a230-4df5-97d5-77401c83bf95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8234e9d-qxpdh\" (UID: \"2b6051c6-a230-4df5-97d5-77401c83bf95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8234e9d-qxpdh"
Oct 30 13:07:43.509898 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:43.509847    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv2bf\" (UniqueName: \"kubernetes.io/projected/2b6051c6-a230-4df5-97d5-77401c83bf95-kube-api-access-wv2bf\") pod \"maas-api-revoke-sub-f8234e9d-qxpdh\" (UID: \"2b6051c6-a230-4df5-97d5-77401c83bf95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8234e9d-qxpdh"
Oct 30 13:07:43.510468 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:43.510444    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b6051c6-a230-4df5-97d5-77401c83bf95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8234e9d-qxpdh\" (UID: \"2b6051c6-a230-4df5-97d5-77401c83bf95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8234e9d-qxpdh"
Oct 30 13:07:43.512284 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:43.512255    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b6051c6-a230-4df5-97d5-77401c83bf95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8234e9d-qxpdh\" (UID: \"2b6051c6-a230-4df5-97d5-77401c83bf95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8234e9d-qxpdh"
Oct 30 13:07:43.543606 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:43.543575    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv2bf\" (UniqueName: \"kubernetes.io/projected/2b6051c6-a230-4df5-97d5-77401c83bf95-kube-api-access-wv2bf\") pod \"maas-api-revoke-sub-f8234e9d-qxpdh\" (UID: \"2b6051c6-a230-4df5-97d5-77401c83bf95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8234e9d-qxpdh"
Oct 30 13:07:43.701867 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:43.701844    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8234e9d-qxpdh"
Oct 30 13:07:44.010700 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:44.010679    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8234e9d-qxpdh"]
Oct 30 13:07:44.011468 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:07:44.011448    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b6051c6_a230_4df5_97d5_77401c83bf95.slice/crio-26cb41808ded97c25d9a9ade6a7250a12d9aebba534ed41805ba791046900375 WatchSource:0}: Error finding container 26cb41808ded97c25d9a9ade6a7250a12d9aebba534ed41805ba791046900375: Status 404 returned error can't find the container with id 26cb41808ded97c25d9a9ade6a7250a12d9aebba534ed41805ba791046900375
Oct 30 13:07:44.730260 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:44.730233    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8234e9d-qxpdh" event={"ID":"2b6051c6-a230-4df5-97d5-77401c83bf95","Type":"ContainerStarted","Data":"3b6043c8778d6b00479bee773a80ba445a4d1f68b4a5460eff7e9b0f03f3d40c"}
Oct 30 13:07:44.730643 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:44.730282    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8234e9d-qxpdh" event={"ID":"2b6051c6-a230-4df5-97d5-77401c83bf95","Type":"ContainerStarted","Data":"26cb41808ded97c25d9a9ade6a7250a12d9aebba534ed41805ba791046900375"}
Oct 30 13:07:44.775140 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:44.775093    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8234e9d-qxpdh" podStartSLOduration=1.775080862 podStartE2EDuration="1.775080862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:44.774392557 +0000 UTC m=+963.940757205" watchObservedRunningTime="2026-10-30 13:07:44.775080862 +0000 UTC m=+963.941445510"
Oct 30 13:07:45.674410 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:45.674377    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh"]
Oct 30 13:07:45.679159 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:45.679138    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh"
Oct 30 13:07:45.696680 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:45.696658    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-52a649\"/\"distinct-w1-52a649-kserve-self-signed-certs\""
Oct 30 13:07:45.696680 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:45.696664    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-52a649\"/\"default-dockercfg-slxg9\""
Oct 30 13:07:45.696853 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:45.696810    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh"]
Oct 30 13:07:45.707296 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:45.707271    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-52a649\"/\"openshift-service-ca.crt\""
Oct 30 13:07:45.707430 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:45.707337    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-52a649\"/\"kube-root-ca.crt\""
Oct 30 13:07:45.738337 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:45.738316    4364 generic.go:356] "Generic (PLEG): container finished" podID="2b6051c6-a230-4df5-97d5-77401c83bf95" containerID="3b6043c8778d6b00479bee773a80ba445a4d1f68b4a5460eff7e9b0f03f3d40c" exitCode=0
Oct 30 13:07:45.738683 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:45.738396    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8234e9d-qxpdh" event={"ID":"2b6051c6-a230-4df5-97d5-77401c83bf95","Type":"ContainerDied","Data":"3b6043c8778d6b00479bee773a80ba445a4d1f68b4a5460eff7e9b0f03f3d40c"}
Oct 30 13:07:45.841841 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:45.841815    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d41ac9e5-eeed-4e07-a7fa-05d3260630b6-home\") pod \"distinct-w1-52a649-kserve-f5d4c7b95-ldndh\" (UID: \"d41ac9e5-eeed-4e07-a7fa-05d3260630b6\") " pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh"
Oct 30 13:07:45.841973 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:45.841849    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d41ac9e5-eeed-4e07-a7fa-05d3260630b6-tmp-dir\") pod \"distinct-w1-52a649-kserve-f5d4c7b95-ldndh\" (UID: \"d41ac9e5-eeed-4e07-a7fa-05d3260630b6\") " pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh"
Oct 30 13:07:45.841973 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:45.841921    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d41ac9e5-eeed-4e07-a7fa-05d3260630b6-dshm\") pod \"distinct-w1-52a649-kserve-f5d4c7b95-ldndh\" (UID: \"d41ac9e5-eeed-4e07-a7fa-05d3260630b6\") " pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh"
Oct 30 13:07:45.842044 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:45.842032    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d41ac9e5-eeed-4e07-a7fa-05d3260630b6-model-cache\") pod \"distinct-w1-52a649-kserve-f5d4c7b95-ldndh\" (UID: \"d41ac9e5-eeed-4e07-a7fa-05d3260630b6\") " pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh"
Oct 30 13:07:45.842118 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:45.842104    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d41ac9e5-eeed-4e07-a7fa-05d3260630b6-tls-certs\") pod \"distinct-w1-52a649-kserve-f5d4c7b95-ldndh\" (UID: \"d41ac9e5-eeed-4e07-a7fa-05d3260630b6\") " pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh"
Oct 30 13:07:45.842163 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:45.842128    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sds5p\" (UniqueName: \"kubernetes.io/projected/d41ac9e5-eeed-4e07-a7fa-05d3260630b6-kube-api-access-sds5p\") pod \"distinct-w1-52a649-kserve-f5d4c7b95-ldndh\" (UID: \"d41ac9e5-eeed-4e07-a7fa-05d3260630b6\") " pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh"
Oct 30 13:07:45.956737 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:45.945381    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d41ac9e5-eeed-4e07-a7fa-05d3260630b6-home\") pod \"distinct-w1-52a649-kserve-f5d4c7b95-ldndh\" (UID: \"d41ac9e5-eeed-4e07-a7fa-05d3260630b6\") " pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh"
Oct 30 13:07:45.956737 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:45.951726    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d41ac9e5-eeed-4e07-a7fa-05d3260630b6-tmp-dir\") pod \"distinct-w1-52a649-kserve-f5d4c7b95-ldndh\" (UID: \"d41ac9e5-eeed-4e07-a7fa-05d3260630b6\") " pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh"
Oct 30 13:07:45.956737 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:45.951802    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d41ac9e5-eeed-4e07-a7fa-05d3260630b6-dshm\") pod \"distinct-w1-52a649-kserve-f5d4c7b95-ldndh\" (UID: \"d41ac9e5-eeed-4e07-a7fa-05d3260630b6\") " pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh"
Oct 30 13:07:45.956737 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:45.951945    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d41ac9e5-eeed-4e07-a7fa-05d3260630b6-model-cache\") pod \"distinct-w1-52a649-kserve-f5d4c7b95-ldndh\" (UID: \"d41ac9e5-eeed-4e07-a7fa-05d3260630b6\") " pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh"
Oct 30 13:07:45.956737 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:45.952064    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d41ac9e5-eeed-4e07-a7fa-05d3260630b6-tls-certs\") pod \"distinct-w1-52a649-kserve-f5d4c7b95-ldndh\" (UID: \"d41ac9e5-eeed-4e07-a7fa-05d3260630b6\") " pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh"
Oct 30 13:07:45.956737 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:45.952097    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sds5p\" (UniqueName: \"kubernetes.io/projected/d41ac9e5-eeed-4e07-a7fa-05d3260630b6-kube-api-access-sds5p\") pod \"distinct-w1-52a649-kserve-f5d4c7b95-ldndh\" (UID: \"d41ac9e5-eeed-4e07-a7fa-05d3260630b6\") " pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh"
Oct 30 13:07:45.956737 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:45.952880    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d41ac9e5-eeed-4e07-a7fa-05d3260630b6-home\") pod \"distinct-w1-52a649-kserve-f5d4c7b95-ldndh\" (UID: \"d41ac9e5-eeed-4e07-a7fa-05d3260630b6\") " pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh"
Oct 30 13:07:45.956737 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:45.953117    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d41ac9e5-eeed-4e07-a7fa-05d3260630b6-tmp-dir\") pod \"distinct-w1-52a649-kserve-f5d4c7b95-ldndh\" (UID: \"d41ac9e5-eeed-4e07-a7fa-05d3260630b6\") " pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh"
Oct 30 13:07:45.956737 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:45.955678    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d41ac9e5-eeed-4e07-a7fa-05d3260630b6-model-cache\") pod \"distinct-w1-52a649-kserve-f5d4c7b95-ldndh\" (UID: \"d41ac9e5-eeed-4e07-a7fa-05d3260630b6\") " pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh"
Oct 30 13:07:45.993019 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:45.992981    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d41ac9e5-eeed-4e07-a7fa-05d3260630b6-tls-certs\") pod \"distinct-w1-52a649-kserve-f5d4c7b95-ldndh\" (UID: \"d41ac9e5-eeed-4e07-a7fa-05d3260630b6\") " pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh"
Oct 30 13:07:45.993769 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:45.993739    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d41ac9e5-eeed-4e07-a7fa-05d3260630b6-dshm\") pod \"distinct-w1-52a649-kserve-f5d4c7b95-ldndh\" (UID: \"d41ac9e5-eeed-4e07-a7fa-05d3260630b6\") " pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh"
Oct 30 13:07:45.994910 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:45.994888    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sds5p\" (UniqueName: \"kubernetes.io/projected/d41ac9e5-eeed-4e07-a7fa-05d3260630b6-kube-api-access-sds5p\") pod \"distinct-w1-52a649-kserve-f5d4c7b95-ldndh\" (UID: \"d41ac9e5-eeed-4e07-a7fa-05d3260630b6\") " pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh"
Oct 30 13:07:46.291591 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:46.291521    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh"
Oct 30 13:07:46.603120 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:07:46.603092    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd41ac9e5_eeed_4e07_a7fa_05d3260630b6.slice/crio-98bcbb0b4ad0d03832c6ffc8a0d77cbf757173f1256260e236e3ecccf2f94bb3 WatchSource:0}: Error finding container 98bcbb0b4ad0d03832c6ffc8a0d77cbf757173f1256260e236e3ecccf2f94bb3: Status 404 returned error can't find the container with id 98bcbb0b4ad0d03832c6ffc8a0d77cbf757173f1256260e236e3ecccf2f94bb3
Oct 30 13:07:46.603244 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:46.603133    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh"]
Oct 30 13:07:46.752814 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:46.752783    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh" event={"ID":"d41ac9e5-eeed-4e07-a7fa-05d3260630b6","Type":"ContainerStarted","Data":"e8043a1abb02b2769b630de3c42da99798366c063cff496dbe5d1c742ef16fe8"}
Oct 30 13:07:46.753182 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:46.752825    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh" event={"ID":"d41ac9e5-eeed-4e07-a7fa-05d3260630b6","Type":"ContainerStarted","Data":"98bcbb0b4ad0d03832c6ffc8a0d77cbf757173f1256260e236e3ecccf2f94bb3"}
Oct 30 13:07:46.799992 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:46.799932    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh" podStartSLOduration=1.799917803 podStartE2EDuration="1.799917803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:46.7973648 +0000 UTC m=+965.963729448" watchObservedRunningTime="2026-10-30 13:07:46.799917803 +0000 UTC m=+965.966282449"
Oct 30 13:07:46.889452 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:46.889430    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8234e9d-qxpdh"
Oct 30 13:07:47.069523 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:47.069496    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b6051c6-a230-4df5-97d5-77401c83bf95-kube-api-access-wv2bf\" (UniqueName: \"kubernetes.io/projected/2b6051c6-a230-4df5-97d5-77401c83bf95-kube-api-access-wv2bf\") pod \"2b6051c6-a230-4df5-97d5-77401c83bf95\" (UID: \"2b6051c6-a230-4df5-97d5-77401c83bf95\") "
Oct 30 13:07:47.069695 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:47.069558    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b6051c6-a230-4df5-97d5-77401c83bf95-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b6051c6-a230-4df5-97d5-77401c83bf95-maas-api-xks-ca\") pod \"2b6051c6-a230-4df5-97d5-77401c83bf95\" (UID: \"2b6051c6-a230-4df5-97d5-77401c83bf95\") "
Oct 30 13:07:47.069751 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:47.069716    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b6051c6-a230-4df5-97d5-77401c83bf95-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b6051c6-a230-4df5-97d5-77401c83bf95-maas-api-openshift-ca\") pod \"2b6051c6-a230-4df5-97d5-77401c83bf95\" (UID: \"2b6051c6-a230-4df5-97d5-77401c83bf95\") "
Oct 30 13:07:47.070109 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:47.070077    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b6051c6-a230-4df5-97d5-77401c83bf95-maas-api-openshift-ca" pod "2b6051c6-a230-4df5-97d5-77401c83bf95" (UID: "2b6051c6-a230-4df5-97d5-77401c83bf95"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:07:47.070653 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:47.070339    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b6051c6-a230-4df5-97d5-77401c83bf95-maas-api-openshift-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:07:47.072176 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:47.072148    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b6051c6-a230-4df5-97d5-77401c83bf95-kube-api-access-wv2bf" pod "2b6051c6-a230-4df5-97d5-77401c83bf95" (UID: "2b6051c6-a230-4df5-97d5-77401c83bf95"). InnerVolumeSpecName "kube-api-access-wv2bf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:07:47.072748 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:47.072668    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b6051c6-a230-4df5-97d5-77401c83bf95-maas-api-xks-ca" pod "2b6051c6-a230-4df5-97d5-77401c83bf95" (UID: "2b6051c6-a230-4df5-97d5-77401c83bf95"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:07:47.200682 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:47.196841    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wv2bf\" (UniqueName: \"kubernetes.io/projected/2b6051c6-a230-4df5-97d5-77401c83bf95-kube-api-access-wv2bf\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:07:47.200682 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:47.196883    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b6051c6-a230-4df5-97d5-77401c83bf95-maas-api-xks-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:07:47.292292 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:47.292264    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh"
Oct 30 13:07:47.760686 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:47.760656    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8234e9d-qxpdh"
Oct 30 13:07:47.760686 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:47.760666    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8234e9d-qxpdh" event={"ID":"2b6051c6-a230-4df5-97d5-77401c83bf95","Type":"ContainerDied","Data":"26cb41808ded97c25d9a9ade6a7250a12d9aebba534ed41805ba791046900375"}
Oct 30 13:07:47.761124 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:47.760717    4364 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26cb41808ded97c25d9a9ade6a7250a12d9aebba534ed41805ba791046900375"
Oct 30 13:07:49.641438 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:49.641408    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:07:53.585696 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:53.585666    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg"]
Oct 30 13:07:53.586222 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:53.586208    4364 cpu_manager.go:414] "containerMap: removing container" podUID="2b6051c6-a230-4df5-97d5-77401c83bf95" containerName="revoke-keys"
Oct 30 13:07:53.586270 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:53.586223    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b6051c6-a230-4df5-97d5-77401c83bf95" containerName="revoke-keys"
Oct 30 13:07:53.586315 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:53.586306    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b6051c6-a230-4df5-97d5-77401c83bf95" containerName="revoke-keys"
Oct 30 13:07:53.590983 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:53.590954    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg"
Oct 30 13:07:53.616883 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:53.616859    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg"]
Oct 30 13:07:53.628378 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:53.628349    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dd8443b-7879-47a7-9916-85844e072002-istio-podinfo\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg\" (UID: \"0dd8443b-7879-47a7-9916-85844e072002\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg"
Oct 30 13:07:53.628520 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:53.628421    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6h5r\" (UniqueName: \"kubernetes.io/projected/0dd8443b-7879-47a7-9916-85844e072002-kube-api-access-g6h5r\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg\" (UID: \"0dd8443b-7879-47a7-9916-85844e072002\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg"
Oct 30 13:07:53.628610 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:53.628531    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dd8443b-7879-47a7-9916-85844e072002-credential-socket\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg\" (UID: \"0dd8443b-7879-47a7-9916-85844e072002\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg"
Oct 30 13:07:53.628610 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:53.628570    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dd8443b-7879-47a7-9916-85844e072002-istio-token\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg\" (UID: \"0dd8443b-7879-47a7-9916-85844e072002\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg"
Oct 30 13:07:53.628728 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:53.628673    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dd8443b-7879-47a7-9916-85844e072002-istio-envoy\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg\" (UID: \"0dd8443b-7879-47a7-9916-85844e072002\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg"
Oct 30 13:07:53.628728 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:53.628703    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dd8443b-7879-47a7-9916-85844e072002-workload-socket\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg\" (UID: \"0dd8443b-7879-47a7-9916-85844e072002\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg"
Oct 30 13:07:53.628831 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:53.628735    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dd8443b-7879-47a7-9916-85844e072002-workload-certs\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg\" (UID: \"0dd8443b-7879-47a7-9916-85844e072002\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg"
Oct 30 13:07:53.628888 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:53.628879    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dd8443b-7879-47a7-9916-85844e072002-istio-data\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg\" (UID: \"0dd8443b-7879-47a7-9916-85844e072002\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg"
Oct 30 13:07:53.628955 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:53.628937    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dd8443b-7879-47a7-9916-85844e072002-istiod-ca-cert\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg\" (UID: \"0dd8443b-7879-47a7-9916-85844e072002\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg"
Oct 30 13:07:53.729924 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:53.729896    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dd8443b-7879-47a7-9916-85844e072002-credential-socket\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg\" (UID: \"0dd8443b-7879-47a7-9916-85844e072002\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg"
Oct 30 13:07:53.730080 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:53.729930    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dd8443b-7879-47a7-9916-85844e072002-istio-token\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg\" (UID: \"0dd8443b-7879-47a7-9916-85844e072002\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg"
Oct 30 13:07:53.730080 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:53.730064    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dd8443b-7879-47a7-9916-85844e072002-istio-envoy\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg\" (UID: \"0dd8443b-7879-47a7-9916-85844e072002\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg"
Oct 30 13:07:53.730183 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:53.730102    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dd8443b-7879-47a7-9916-85844e072002-workload-socket\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg\" (UID: \"0dd8443b-7879-47a7-9916-85844e072002\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg"
Oct 30 13:07:53.730183 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:53.730123    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dd8443b-7879-47a7-9916-85844e072002-workload-certs\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg\" (UID: \"0dd8443b-7879-47a7-9916-85844e072002\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg"
Oct 30 13:07:53.730272 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:53.730227    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dd8443b-7879-47a7-9916-85844e072002-istio-data\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg\" (UID: \"0dd8443b-7879-47a7-9916-85844e072002\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg"
Oct 30 13:07:53.730331 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:53.730271    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dd8443b-7879-47a7-9916-85844e072002-istiod-ca-cert\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg\" (UID: \"0dd8443b-7879-47a7-9916-85844e072002\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg"
Oct 30 13:07:53.730331 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:53.730306    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dd8443b-7879-47a7-9916-85844e072002-istio-podinfo\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg\" (UID: \"0dd8443b-7879-47a7-9916-85844e072002\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg"
Oct 30 13:07:53.730331 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:53.730307    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dd8443b-7879-47a7-9916-85844e072002-credential-socket\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg\" (UID: \"0dd8443b-7879-47a7-9916-85844e072002\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg"
Oct 30 13:07:53.730490 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:53.730378    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6h5r\" (UniqueName: \"kubernetes.io/projected/0dd8443b-7879-47a7-9916-85844e072002-kube-api-access-g6h5r\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg\" (UID: \"0dd8443b-7879-47a7-9916-85844e072002\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg"
Oct 30 13:07:53.730490 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:53.730479    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dd8443b-7879-47a7-9916-85844e072002-workload-socket\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg\" (UID: \"0dd8443b-7879-47a7-9916-85844e072002\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg"
Oct 30 13:07:53.730620 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:53.730580    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dd8443b-7879-47a7-9916-85844e072002-istio-data\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg\" (UID: \"0dd8443b-7879-47a7-9916-85844e072002\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg"
Oct 30 13:07:53.730669 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:53.730578    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dd8443b-7879-47a7-9916-85844e072002-workload-certs\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg\" (UID: \"0dd8443b-7879-47a7-9916-85844e072002\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg"
Oct 30 13:07:53.731020 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:53.730997    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dd8443b-7879-47a7-9916-85844e072002-istiod-ca-cert\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg\" (UID: \"0dd8443b-7879-47a7-9916-85844e072002\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg"
Oct 30 13:07:53.732225 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:53.732207    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dd8443b-7879-47a7-9916-85844e072002-istio-envoy\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg\" (UID: \"0dd8443b-7879-47a7-9916-85844e072002\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg"
Oct 30 13:07:53.732590 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:53.732576    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dd8443b-7879-47a7-9916-85844e072002-istio-podinfo\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg\" (UID: \"0dd8443b-7879-47a7-9916-85844e072002\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg"
Oct 30 13:07:53.754959 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:53.754933    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dd8443b-7879-47a7-9916-85844e072002-istio-token\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg\" (UID: \"0dd8443b-7879-47a7-9916-85844e072002\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg"
Oct 30 13:07:53.755557 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:53.755535    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6h5r\" (UniqueName: \"kubernetes.io/projected/0dd8443b-7879-47a7-9916-85844e072002-kube-api-access-g6h5r\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg\" (UID: \"0dd8443b-7879-47a7-9916-85844e072002\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg"
Oct 30 13:07:53.908785 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:53.908757    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg"
Oct 30 13:07:54.230441 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:54.230415    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg"]
Oct 30 13:07:54.230587 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:07:54.230534    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dd8443b_7879_47a7_9916_85844e072002.slice/crio-ac24cd773670edfb8408ba7d0a5470bef6a340262a1489810a3c70518d546998 WatchSource:0}: Error finding container ac24cd773670edfb8408ba7d0a5470bef6a340262a1489810a3c70518d546998: Status 404 returned error can't find the container with id ac24cd773670edfb8408ba7d0a5470bef6a340262a1489810a3c70518d546998
Oct 30 13:07:54.232879 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:54.232850    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:07:54.232956 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:54.232910    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:07:54.232956 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:54.232942    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:07:54.833373 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:54.833330    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg" event={"ID":"0dd8443b-7879-47a7-9916-85844e072002","Type":"ContainerStarted","Data":"72dea0433e055fac820df54527178d9fab4580f403e4a6dfc155961f4c57421d"}
Oct 30 13:07:54.833918 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:54.833395    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg" event={"ID":"0dd8443b-7879-47a7-9916-85844e072002","Type":"ContainerStarted","Data":"ac24cd773670edfb8408ba7d0a5470bef6a340262a1489810a3c70518d546998"}
Oct 30 13:07:54.920621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:54.910693    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg"
Oct 30 13:07:55.910210 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:55.910175    4364 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 13:07:55.910635 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:55.910227    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg" podUID="0dd8443b-7879-47a7-9916-85844e072002" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 13:07:56.298675 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:56.293844    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh"
Oct 30 13:07:56.315380 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:56.315350    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh"
Oct 30 13:07:56.364509 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:56.364465    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg" podStartSLOduration=3.364453473 podStartE2EDuration="3.364453473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:54.893949767 +0000 UTC m=+974.060314426" watchObservedRunningTime="2026-10-30 13:07:56.364453473 +0000 UTC m=+975.530818121"
Oct 30 13:07:56.858872 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:56.858823    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh"
Oct 30 13:07:56.916621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:56.910974    4364 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 13:07:56.916621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:56.911037    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg" podUID="0dd8443b-7879-47a7-9916-85844e072002" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 13:07:57.909187 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:57.909154    4364 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 13:07:57.909375 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:57.909208    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg" podUID="0dd8443b-7879-47a7-9916-85844e072002" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 13:07:58.246223 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:58.246151    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21460530-c5v8z"]
Oct 30 13:07:58.250360 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:58.250339    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21460530-c5v8z"
Oct 30 13:07:58.269333 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:58.269313    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21460530-c5v8z"]
Oct 30 13:07:58.303622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:58.290829    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:07:58.303622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:58.290955    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\""
Oct 30 13:07:58.318686 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:58.318660    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb2r9\" (UniqueName: \"kubernetes.io/projected/b8bfb829-6b1b-4788-ba44-2c2b92a73f66-kube-api-access-hb2r9\") pod \"maas-api-revoke-sub-21460530-c5v8z\" (UID: \"b8bfb829-6b1b-4788-ba44-2c2b92a73f66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21460530-c5v8z"
Oct 30 13:07:58.318822 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:58.318703    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8bfb829-6b1b-4788-ba44-2c2b92a73f66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21460530-c5v8z\" (UID: \"b8bfb829-6b1b-4788-ba44-2c2b92a73f66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21460530-c5v8z"
Oct 30 13:07:58.318822 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:58.318743    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8bfb829-6b1b-4788-ba44-2c2b92a73f66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21460530-c5v8z\" (UID: \"b8bfb829-6b1b-4788-ba44-2c2b92a73f66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21460530-c5v8z"
Oct 30 13:07:58.420522 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:58.420488    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8bfb829-6b1b-4788-ba44-2c2b92a73f66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21460530-c5v8z\" (UID: \"b8bfb829-6b1b-4788-ba44-2c2b92a73f66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21460530-c5v8z"
Oct 30 13:07:58.420701 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:58.420546    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8bfb829-6b1b-4788-ba44-2c2b92a73f66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21460530-c5v8z\" (UID: \"b8bfb829-6b1b-4788-ba44-2c2b92a73f66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21460530-c5v8z"
Oct 30 13:07:58.420758 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:58.420737    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb2r9\" (UniqueName: \"kubernetes.io/projected/b8bfb829-6b1b-4788-ba44-2c2b92a73f66-kube-api-access-hb2r9\") pod \"maas-api-revoke-sub-21460530-c5v8z\" (UID: \"b8bfb829-6b1b-4788-ba44-2c2b92a73f66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21460530-c5v8z"
Oct 30 13:07:58.421256 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:58.421232    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8bfb829-6b1b-4788-ba44-2c2b92a73f66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21460530-c5v8z\" (UID: \"b8bfb829-6b1b-4788-ba44-2c2b92a73f66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21460530-c5v8z"
Oct 30 13:07:58.422869 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:58.422847    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8bfb829-6b1b-4788-ba44-2c2b92a73f66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21460530-c5v8z\" (UID: \"b8bfb829-6b1b-4788-ba44-2c2b92a73f66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21460530-c5v8z"
Oct 30 13:07:58.446313 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:58.446292    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb2r9\" (UniqueName: \"kubernetes.io/projected/b8bfb829-6b1b-4788-ba44-2c2b92a73f66-kube-api-access-hb2r9\") pod \"maas-api-revoke-sub-21460530-c5v8z\" (UID: \"b8bfb829-6b1b-4788-ba44-2c2b92a73f66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21460530-c5v8z"
Oct 30 13:07:58.564044 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:58.563973    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21460530-c5v8z"
Oct 30 13:07:58.909925 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:58.909893    4364 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 13:07:58.910086 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:58.909947    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg" podUID="0dd8443b-7879-47a7-9916-85844e072002" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 13:07:58.927338 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:07:58.927313    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8bfb829_6b1b_4788_ba44_2c2b92a73f66.slice/crio-f2e939f77082b692f3c3657e802740ebc4cb47ed6f6b2d9fbab502acaf8b41d6 WatchSource:0}: Error finding container f2e939f77082b692f3c3657e802740ebc4cb47ed6f6b2d9fbab502acaf8b41d6: Status 404 returned error can't find the container with id f2e939f77082b692f3c3657e802740ebc4cb47ed6f6b2d9fbab502acaf8b41d6
Oct 30 13:07:58.928156 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:58.928131    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21460530-c5v8z"]
Oct 30 13:07:59.868911 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:59.868877    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21460530-c5v8z" event={"ID":"b8bfb829-6b1b-4788-ba44-2c2b92a73f66","Type":"ContainerStarted","Data":"6a242d72328973a402309222e06bf84a310279229fc7810d7090acb47545fafb"}
Oct 30 13:07:59.869296 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:59.868951    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21460530-c5v8z" event={"ID":"b8bfb829-6b1b-4788-ba44-2c2b92a73f66","Type":"ContainerStarted","Data":"f2e939f77082b692f3c3657e802740ebc4cb47ed6f6b2d9fbab502acaf8b41d6"}
Oct 30 13:07:59.909353 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:59.909319    4364 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 13:07:59.909495 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:59.909378    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg" podUID="0dd8443b-7879-47a7-9916-85844e072002" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 13:07:59.934189 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:07:59.934137    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21460530-c5v8z" podStartSLOduration=1.934124775 podStartE2EDuration="1.934124775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:59.930532578 +0000 UTC m=+979.096897227" watchObservedRunningTime="2026-10-30 13:07:59.934124775 +0000 UTC m=+979.100489423"
Oct 30 13:08:00.876356 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:00.876322    4364 generic.go:356] "Generic (PLEG): container finished" podID="b8bfb829-6b1b-4788-ba44-2c2b92a73f66" containerID="6a242d72328973a402309222e06bf84a310279229fc7810d7090acb47545fafb" exitCode=0
Oct 30 13:08:00.876913 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:00.876448    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21460530-c5v8z" event={"ID":"b8bfb829-6b1b-4788-ba44-2c2b92a73f66","Type":"ContainerDied","Data":"6a242d72328973a402309222e06bf84a310279229fc7810d7090acb47545fafb"}
Oct 30 13:08:00.909656 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:00.909624    4364 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 13:08:00.909802 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:00.909667    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg" podUID="0dd8443b-7879-47a7-9916-85844e072002" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 13:08:01.961226 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:01.961195    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg"
Oct 30 13:08:01.962948 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:01.961989    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg"
Oct 30 13:08:01.983736 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:01.982783    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg"
Oct 30 13:08:02.065361 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:02.065329    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21460530-c5v8z"
Oct 30 13:08:02.189255 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:02.189146    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8bfb829-6b1b-4788-ba44-2c2b92a73f66-kube-api-access-hb2r9\" (UniqueName: \"kubernetes.io/projected/b8bfb829-6b1b-4788-ba44-2c2b92a73f66-kube-api-access-hb2r9\") pod \"b8bfb829-6b1b-4788-ba44-2c2b92a73f66\" (UID: \"b8bfb829-6b1b-4788-ba44-2c2b92a73f66\") "
Oct 30 13:08:02.189426 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:02.189262    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8bfb829-6b1b-4788-ba44-2c2b92a73f66-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8bfb829-6b1b-4788-ba44-2c2b92a73f66-maas-api-xks-ca\") pod \"b8bfb829-6b1b-4788-ba44-2c2b92a73f66\" (UID: \"b8bfb829-6b1b-4788-ba44-2c2b92a73f66\") "
Oct 30 13:08:02.189426 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:02.189328    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8bfb829-6b1b-4788-ba44-2c2b92a73f66-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8bfb829-6b1b-4788-ba44-2c2b92a73f66-maas-api-openshift-ca\") pod \"b8bfb829-6b1b-4788-ba44-2c2b92a73f66\" (UID: \"b8bfb829-6b1b-4788-ba44-2c2b92a73f66\") "
Oct 30 13:08:02.190482 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:02.190436    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8bfb829-6b1b-4788-ba44-2c2b92a73f66-maas-api-openshift-ca" pod "b8bfb829-6b1b-4788-ba44-2c2b92a73f66" (UID: "b8bfb829-6b1b-4788-ba44-2c2b92a73f66"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:08:02.193161 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:02.193123    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8bfb829-6b1b-4788-ba44-2c2b92a73f66-kube-api-access-hb2r9" pod "b8bfb829-6b1b-4788-ba44-2c2b92a73f66" (UID: "b8bfb829-6b1b-4788-ba44-2c2b92a73f66"). InnerVolumeSpecName "kube-api-access-hb2r9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:08:02.193522 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:02.193487    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8bfb829-6b1b-4788-ba44-2c2b92a73f66-maas-api-xks-ca" pod "b8bfb829-6b1b-4788-ba44-2c2b92a73f66" (UID: "b8bfb829-6b1b-4788-ba44-2c2b92a73f66"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:08:02.291388 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:02.291349    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hb2r9\" (UniqueName: \"kubernetes.io/projected/b8bfb829-6b1b-4788-ba44-2c2b92a73f66-kube-api-access-hb2r9\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:08:02.291388 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:02.291389    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8bfb829-6b1b-4788-ba44-2c2b92a73f66-maas-api-xks-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:08:02.291573 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:02.291403    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8bfb829-6b1b-4788-ba44-2c2b92a73f66-maas-api-openshift-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:08:02.893908 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:02.893871    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21460530-c5v8z" event={"ID":"b8bfb829-6b1b-4788-ba44-2c2b92a73f66","Type":"ContainerDied","Data":"f2e939f77082b692f3c3657e802740ebc4cb47ed6f6b2d9fbab502acaf8b41d6"}
Oct 30 13:08:02.893908 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:02.893902    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21460530-c5v8z"
Oct 30 13:08:02.894120 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:02.893956    4364 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2e939f77082b692f3c3657e802740ebc4cb47ed6f6b2d9fbab502acaf8b41d6"
Oct 30 13:08:08.729055 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:08.729022    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp"]
Oct 30 13:08:08.729627 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:08.729592    4364 cpu_manager.go:414] "containerMap: removing container" podUID="b8bfb829-6b1b-4788-ba44-2c2b92a73f66" containerName="revoke-keys"
Oct 30 13:08:08.729684 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:08.729626    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8bfb829-6b1b-4788-ba44-2c2b92a73f66" containerName="revoke-keys"
Oct 30 13:08:08.729758 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:08.729748    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8bfb829-6b1b-4788-ba44-2c2b92a73f66" containerName="revoke-keys"
Oct 30 13:08:08.733430 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:08.733403    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp"
Oct 30 13:08:08.767318 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:08.767293    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp"]
Oct 30 13:08:08.769822 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:08.769796    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faba8da0-1b85-49fc-aa90-f860879f40a5-workload-certs\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp\" (UID: \"faba8da0-1b85-49fc-aa90-f860879f40a5\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp"
Oct 30 13:08:08.769941 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:08.769854    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faba8da0-1b85-49fc-aa90-f860879f40a5-credential-socket\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp\" (UID: \"faba8da0-1b85-49fc-aa90-f860879f40a5\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp"
Oct 30 13:08:08.769990 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:08.769964    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faba8da0-1b85-49fc-aa90-f860879f40a5-istio-token\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp\" (UID: \"faba8da0-1b85-49fc-aa90-f860879f40a5\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp"
Oct 30 13:08:08.770086 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:08.770073    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faba8da0-1b85-49fc-aa90-f860879f40a5-workload-socket\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp\" (UID: \"faba8da0-1b85-49fc-aa90-f860879f40a5\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp"
Oct 30 13:08:08.770129 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:08.770122    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faba8da0-1b85-49fc-aa90-f860879f40a5-istio-podinfo\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp\" (UID: \"faba8da0-1b85-49fc-aa90-f860879f40a5\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp"
Oct 30 13:08:08.770273 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:08.770253    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faba8da0-1b85-49fc-aa90-f860879f40a5-istio-envoy\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp\" (UID: \"faba8da0-1b85-49fc-aa90-f860879f40a5\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp"
Oct 30 13:08:08.770352 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:08.770299    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faba8da0-1b85-49fc-aa90-f860879f40a5-istio-data\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp\" (UID: \"faba8da0-1b85-49fc-aa90-f860879f40a5\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp"
Oct 30 13:08:08.770352 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:08.770319    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faba8da0-1b85-49fc-aa90-f860879f40a5-istiod-ca-cert\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp\" (UID: \"faba8da0-1b85-49fc-aa90-f860879f40a5\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp"
Oct 30 13:08:08.770352 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:08.770334    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldhhb\" (UniqueName: \"kubernetes.io/projected/faba8da0-1b85-49fc-aa90-f860879f40a5-kube-api-access-ldhhb\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp\" (UID: \"faba8da0-1b85-49fc-aa90-f860879f40a5\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp"
Oct 30 13:08:08.874654 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:08.874354    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faba8da0-1b85-49fc-aa90-f860879f40a5-istio-podinfo\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp\" (UID: \"faba8da0-1b85-49fc-aa90-f860879f40a5\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp"
Oct 30 13:08:08.874654 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:08.874480    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faba8da0-1b85-49fc-aa90-f860879f40a5-istio-envoy\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp\" (UID: \"faba8da0-1b85-49fc-aa90-f860879f40a5\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp"
Oct 30 13:08:08.874654 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:08.874527    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faba8da0-1b85-49fc-aa90-f860879f40a5-istio-data\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp\" (UID: \"faba8da0-1b85-49fc-aa90-f860879f40a5\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp"
Oct 30 13:08:08.874654 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:08.874557    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faba8da0-1b85-49fc-aa90-f860879f40a5-istiod-ca-cert\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp\" (UID: \"faba8da0-1b85-49fc-aa90-f860879f40a5\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp"
Oct 30 13:08:08.874654 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:08.874584    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldhhb\" (UniqueName: \"kubernetes.io/projected/faba8da0-1b85-49fc-aa90-f860879f40a5-kube-api-access-ldhhb\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp\" (UID: \"faba8da0-1b85-49fc-aa90-f860879f40a5\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp"
Oct 30 13:08:08.875063 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:08.874701    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faba8da0-1b85-49fc-aa90-f860879f40a5-workload-certs\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp\" (UID: \"faba8da0-1b85-49fc-aa90-f860879f40a5\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp"
Oct 30 13:08:08.875063 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:08.874761    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faba8da0-1b85-49fc-aa90-f860879f40a5-credential-socket\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp\" (UID: \"faba8da0-1b85-49fc-aa90-f860879f40a5\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp"
Oct 30 13:08:08.875063 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:08.874807    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faba8da0-1b85-49fc-aa90-f860879f40a5-istio-token\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp\" (UID: \"faba8da0-1b85-49fc-aa90-f860879f40a5\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp"
Oct 30 13:08:08.875063 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:08.874896    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faba8da0-1b85-49fc-aa90-f860879f40a5-workload-socket\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp\" (UID: \"faba8da0-1b85-49fc-aa90-f860879f40a5\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp"
Oct 30 13:08:08.889638 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:08.875460    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faba8da0-1b85-49fc-aa90-f860879f40a5-workload-socket\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp\" (UID: \"faba8da0-1b85-49fc-aa90-f860879f40a5\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp"
Oct 30 13:08:08.889638 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:08.877493    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faba8da0-1b85-49fc-aa90-f860879f40a5-istiod-ca-cert\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp\" (UID: \"faba8da0-1b85-49fc-aa90-f860879f40a5\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp"
Oct 30 13:08:08.889638 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:08.878046    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faba8da0-1b85-49fc-aa90-f860879f40a5-credential-socket\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp\" (UID: \"faba8da0-1b85-49fc-aa90-f860879f40a5\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp"
Oct 30 13:08:08.889638 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:08.879355    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faba8da0-1b85-49fc-aa90-f860879f40a5-istio-data\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp\" (UID: \"faba8da0-1b85-49fc-aa90-f860879f40a5\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp"
Oct 30 13:08:08.894773 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:08.890304    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faba8da0-1b85-49fc-aa90-f860879f40a5-workload-certs\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp\" (UID: \"faba8da0-1b85-49fc-aa90-f860879f40a5\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp"
Oct 30 13:08:08.895772 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:08.895743    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faba8da0-1b85-49fc-aa90-f860879f40a5-istio-envoy\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp\" (UID: \"faba8da0-1b85-49fc-aa90-f860879f40a5\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp"
Oct 30 13:08:08.904543 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:08.904512    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faba8da0-1b85-49fc-aa90-f860879f40a5-istio-token\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp\" (UID: \"faba8da0-1b85-49fc-aa90-f860879f40a5\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp"
Oct 30 13:08:08.905366 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:08.905334    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faba8da0-1b85-49fc-aa90-f860879f40a5-istio-podinfo\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp\" (UID: \"faba8da0-1b85-49fc-aa90-f860879f40a5\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp"
Oct 30 13:08:08.905486 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:08.905366    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldhhb\" (UniqueName: \"kubernetes.io/projected/faba8da0-1b85-49fc-aa90-f860879f40a5-kube-api-access-ldhhb\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp\" (UID: \"faba8da0-1b85-49fc-aa90-f860879f40a5\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp"
Oct 30 13:08:09.050004 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:09.049920    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp"
Oct 30 13:08:09.372801 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:08:09.372772    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaba8da0_1b85_49fc_aa90_f860879f40a5.slice/crio-33e1a2fa2a816b6e9c14ec476748d2f9cce0ae464412c71f4a39922c0bc3851a WatchSource:0}: Error finding container 33e1a2fa2a816b6e9c14ec476748d2f9cce0ae464412c71f4a39922c0bc3851a: Status 404 returned error can't find the container with id 33e1a2fa2a816b6e9c14ec476748d2f9cce0ae464412c71f4a39922c0bc3851a
Oct 30 13:08:09.373311 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:09.373285    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp"]
Oct 30 13:08:09.375076 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:09.375043    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:08:09.375174 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:09.375107    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:08:09.375174 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:09.375141    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:08:10.167617 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:10.152138    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp" event={"ID":"faba8da0-1b85-49fc-aa90-f860879f40a5","Type":"ContainerStarted","Data":"07f9855fd3f5b075c58f6437479ef56e3ab9421cc19c3ecbfac80e78279ea709"}
Oct 30 13:08:10.167617 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:10.152241    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp" event={"ID":"faba8da0-1b85-49fc-aa90-f860879f40a5","Type":"ContainerStarted","Data":"33e1a2fa2a816b6e9c14ec476748d2f9cce0ae464412c71f4a39922c0bc3851a"}
Oct 30 13:08:10.240862 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:10.239656    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp" podStartSLOduration=2.239637352 podStartE2EDuration="2.239637352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:10.224158143 +0000 UTC m=+989.390522806" watchObservedRunningTime="2026-10-30 13:08:10.239637352 +0000 UTC m=+989.406002000"
Oct 30 13:08:11.275110 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:11.274743    4364 patch_prober.go:29] interesting pod/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 13:08:11.275110 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:11.274809    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp" podUID="faba8da0-1b85-49fc-aa90-f860879f40a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 13:08:11.490620 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:11.487235    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp"
Oct 30 13:08:11.870628 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:11.866980    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh" podUID="d41ac9e5-eeed-4e07-a7fa-05d3260630b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:11.978625 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:08:11.975861    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:08:12.056622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:12.055963    4364 patch_prober.go:29] interesting pod/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 13:08:12.056622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:12.056021    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp" podUID="faba8da0-1b85-49fc-aa90-f860879f40a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 13:08:12.056622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:12.056463    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6" podUID="b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 13:08:12.255681 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:12.255019    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz" podUID="bc5caf69-a649-47e5-8910-9b8f331dfd4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 13:08:12.255681 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:12.255460    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4" podUID="0cbe6af7-d884-4613-90b8-192e5ffdb027" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 13:08:12.444625 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:12.443698    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz" podUID="a568f1ce-602d-4069-a41c-9fe1a7635c5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:12.642624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:12.638273    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc" podUID="8911b790-331d-490c-876b-647e6d2fcd14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:12.902199 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:12.867675    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh" podUID="d41ac9e5-eeed-4e07-a7fa-05d3260630b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 30 13:08:12.902199 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:12.868251    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh"
Oct 30 13:08:13.062653 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:13.056753    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6" podUID="b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:13.077632 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:13.067698    4364 patch_prober.go:29] interesting pod/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 13:08:13.077632 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:13.067752    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp" podUID="faba8da0-1b85-49fc-aa90-f860879f40a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 13:08:13.077986 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:13.077955    4364 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz" podUID="a568f1ce-602d-4069-a41c-9fe1a7635c5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:13.252008 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:13.251919    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6"
Oct 30 13:08:13.255339 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:13.255308    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz" podUID="bc5caf69-a649-47e5-8910-9b8f331dfd4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:13.274726 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:13.274691    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz"
Oct 30 13:08:13.348623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:13.276577    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4" podUID="0cbe6af7-d884-4613-90b8-192e5ffdb027" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 13:08:13.349172 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:13.349048    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4"
Oct 30 13:08:13.475034 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:13.456091    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz" podUID="a568f1ce-602d-4069-a41c-9fe1a7635c5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:13.476530 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:13.475920    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz"
Oct 30 13:08:13.667065 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:13.666558    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc" podUID="8911b790-331d-490c-876b-647e6d2fcd14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded"
Oct 30 13:08:13.669831 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:13.669810    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc"
Oct 30 13:08:13.671526 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:13.671509    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz"
Oct 30 13:08:13.869793 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:13.869713    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh" podUID="d41ac9e5-eeed-4e07-a7fa-05d3260630b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:13.893275 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:13.893246    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz"
Oct 30 13:08:13.935769 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:13.935683    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc"
Oct 30 13:08:14.099817 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:14.099780    4364 patch_prober.go:29] interesting pod/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 13:08:14.100128 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:14.100084    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp" podUID="faba8da0-1b85-49fc-aa90-f860879f40a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 13:08:14.194000 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:14.193900    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6" podUID="b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:14.350300 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:14.350245    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4" podUID="0cbe6af7-d884-4613-90b8-192e5ffdb027" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:14.355073 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:14.355048    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6"
Oct 30 13:08:14.420377 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:14.420346    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh"
Oct 30 13:08:14.426083 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:14.426058    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4"
Oct 30 13:08:15.050756 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:15.050723    4364 patch_prober.go:29] interesting pod/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 13:08:15.051161 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:15.050772    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp" podUID="faba8da0-1b85-49fc-aa90-f860879f40a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 13:08:16.050731 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:16.050698    4364 patch_prober.go:29] interesting pod/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 13:08:16.050917 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:16.050758    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp" podUID="faba8da0-1b85-49fc-aa90-f860879f40a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 13:08:17.050881 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:17.050846    4364 patch_prober.go:29] interesting pod/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 13:08:17.051061 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:17.050904    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp" podUID="faba8da0-1b85-49fc-aa90-f860879f40a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 13:08:18.501531 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:18.501493    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp"
Oct 30 13:08:18.502151 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:18.502119    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp"
Oct 30 13:08:18.502989 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:18.502964    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp"
Oct 30 13:08:23.085565 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:23.085526    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q"]
Oct 30 13:08:23.092235 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:23.092183    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q"
Oct 30 13:08:23.103323 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:23.103298    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425"]
Oct 30 13:08:23.107870 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:23.107825    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425"
Oct 30 13:08:23.118135 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:23.118108    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q"]
Oct 30 13:08:23.137718 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:23.137692    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425"]
Oct 30 13:08:23.218723 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:23.218692    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db45cea6-0b96-46b2-94ff-48caa8747410-workload-socket\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q\" (UID: \"db45cea6-0b96-46b2-94ff-48caa8747410\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q"
Oct 30 13:08:23.218892 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:23.218732    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db45cea6-0b96-46b2-94ff-48caa8747410-istiod-ca-cert\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q\" (UID: \"db45cea6-0b96-46b2-94ff-48caa8747410\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q"
Oct 30 13:08:23.218892 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:23.218813    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db45cea6-0b96-46b2-94ff-48caa8747410-credential-socket\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q\" (UID: \"db45cea6-0b96-46b2-94ff-48caa8747410\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q"
Oct 30 13:08:23.218973 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:23.218885    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db45cea6-0b96-46b2-94ff-48caa8747410-istio-token\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q\" (UID: \"db45cea6-0b96-46b2-94ff-48caa8747410\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q"
Oct 30 13:08:23.218973 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:23.218937    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db45cea6-0b96-46b2-94ff-48caa8747410-istio-podinfo\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q\" (UID: \"db45cea6-0b96-46b2-94ff-48caa8747410\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q"
Oct 30 13:08:23.218973 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:23.218959    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9059a7c7-81e0-4f77-b2dd-2e44f5c10d06-credential-socket\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425\" (UID: \"9059a7c7-81e0-4f77-b2dd-2e44f5c10d06\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425"
Oct 30 13:08:23.219093 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:23.218977    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjj9z\" (UniqueName: \"kubernetes.io/projected/9059a7c7-81e0-4f77-b2dd-2e44f5c10d06-kube-api-access-gjj9z\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425\" (UID: \"9059a7c7-81e0-4f77-b2dd-2e44f5c10d06\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425"
Oct 30 13:08:23.219093 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:23.219012    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9059a7c7-81e0-4f77-b2dd-2e44f5c10d06-istio-envoy\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425\" (UID: \"9059a7c7-81e0-4f77-b2dd-2e44f5c10d06\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425"
Oct 30 13:08:23.219160 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:23.219106    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db45cea6-0b96-46b2-94ff-48caa8747410-workload-certs\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q\" (UID: \"db45cea6-0b96-46b2-94ff-48caa8747410\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q"
Oct 30 13:08:23.219160 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:23.219133    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db45cea6-0b96-46b2-94ff-48caa8747410-istio-data\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q\" (UID: \"db45cea6-0b96-46b2-94ff-48caa8747410\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q"
Oct 30 13:08:23.219224 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:23.219199    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9059a7c7-81e0-4f77-b2dd-2e44f5c10d06-workload-socket\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425\" (UID: \"9059a7c7-81e0-4f77-b2dd-2e44f5c10d06\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425"
Oct 30 13:08:23.219290 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:23.219278    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db45cea6-0b96-46b2-94ff-48caa8747410-istio-envoy\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q\" (UID: \"db45cea6-0b96-46b2-94ff-48caa8747410\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q"
Oct 30 13:08:23.219326 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:23.219300    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfshc\" (UniqueName: \"kubernetes.io/projected/db45cea6-0b96-46b2-94ff-48caa8747410-kube-api-access-lfshc\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q\" (UID: \"db45cea6-0b96-46b2-94ff-48caa8747410\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q"
Oct 30 13:08:23.219326 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:23.219314    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9059a7c7-81e0-4f77-b2dd-2e44f5c10d06-istio-data\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425\" (UID: \"9059a7c7-81e0-4f77-b2dd-2e44f5c10d06\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425"
Oct 30 13:08:23.219401 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:23.219337    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9059a7c7-81e0-4f77-b2dd-2e44f5c10d06-workload-certs\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425\" (UID: \"9059a7c7-81e0-4f77-b2dd-2e44f5c10d06\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425"
Oct 30 13:08:23.219401 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:23.219359    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9059a7c7-81e0-4f77-b2dd-2e44f5c10d06-istiod-ca-cert\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425\" (UID: \"9059a7c7-81e0-4f77-b2dd-2e44f5c10d06\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425"
Oct 30 13:08:23.219488 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:23.219448    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9059a7c7-81e0-4f77-b2dd-2e44f5c10d06-istio-podinfo\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425\" (UID: \"9059a7c7-81e0-4f77-b2dd-2e44f5c10d06\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425"
Oct 30 13:08:23.219488 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:23.219466    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9059a7c7-81e0-4f77-b2dd-2e44f5c10d06-istio-token\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425\" (UID: \"9059a7c7-81e0-4f77-b2dd-2e44f5c10d06\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425"
Oct 30 13:08:23.320716 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:23.320685    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db45cea6-0b96-46b2-94ff-48caa8747410-workload-certs\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q\" (UID: \"db45cea6-0b96-46b2-94ff-48caa8747410\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q"
Oct 30 13:08:23.320716 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:23.320720    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db45cea6-0b96-46b2-94ff-48caa8747410-istio-data\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q\" (UID: \"db45cea6-0b96-46b2-94ff-48caa8747410\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q"
Oct 30 13:08:23.320906 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:23.320751    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9059a7c7-81e0-4f77-b2dd-2e44f5c10d06-workload-socket\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425\" (UID: \"9059a7c7-81e0-4f77-b2dd-2e44f5c10d06\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425"
Oct 30 13:08:23.320906 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:23.320827    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db45cea6-0b96-46b2-94ff-48caa8747410-istio-envoy\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q\" (UID: \"db45cea6-0b96-46b2-94ff-48caa8747410\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q"
Oct 30 13:08:23.320906 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:23.320852    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfshc\" (UniqueName: \"kubernetes.io/projected/db45cea6-0b96-46b2-94ff-48caa8747410-kube-api-access-lfshc\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q\" (UID: \"db45cea6-0b96-46b2-94ff-48caa8747410\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q"
Oct 30 13:08:23.320906 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:23.320871    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9059a7c7-81e0-4f77-b2dd-2e44f5c10d06-istio-data\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425\" (UID: \"9059a7c7-81e0-4f77-b2dd-2e44f5c10d06\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425"
Oct 30 13:08:23.320906 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:23.320898    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9059a7c7-81e0-4f77-b2dd-2e44f5c10d06-workload-certs\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425\" (UID: \"9059a7c7-81e0-4f77-b2dd-2e44f5c10d06\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425"
Oct 30 13:08:23.321175 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:23.320923    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9059a7c7-81e0-4f77-b2dd-2e44f5c10d06-istiod-ca-cert\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425\" (UID: \"9059a7c7-81e0-4f77-b2dd-2e44f5c10d06\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425"
Oct 30 13:08:23.321175 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:23.320995    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9059a7c7-81e0-4f77-b2dd-2e44f5c10d06-istio-podinfo\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425\" (UID: \"9059a7c7-81e0-4f77-b2dd-2e44f5c10d06\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425"
Oct 30 13:08:23.321175 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:23.321021    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9059a7c7-81e0-4f77-b2dd-2e44f5c10d06-istio-token\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425\" (UID: \"9059a7c7-81e0-4f77-b2dd-2e44f5c10d06\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425"
Oct 30 13:08:23.321175 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:23.321072    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db45cea6-0b96-46b2-94ff-48caa8747410-workload-socket\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q\" (UID: \"db45cea6-0b96-46b2-94ff-48caa8747410\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q"
Oct 30 13:08:23.321175 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:23.321083    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db45cea6-0b96-46b2-94ff-48caa8747410-istio-data\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q\" (UID: \"db45cea6-0b96-46b2-94ff-48caa8747410\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q"
Oct 30 13:08:23.321175 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:23.321101    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db45cea6-0b96-46b2-94ff-48caa8747410-istiod-ca-cert\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q\" (UID: \"db45cea6-0b96-46b2-94ff-48caa8747410\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q"
Oct 30 13:08:23.321175 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:23.321144    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db45cea6-0b96-46b2-94ff-48caa8747410-credential-socket\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q\" (UID: \"db45cea6-0b96-46b2-94ff-48caa8747410\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q"
Oct 30 13:08:23.321532 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:23.321175    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9059a7c7-81e0-4f77-b2dd-2e44f5c10d06-workload-socket\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425\" (UID: \"9059a7c7-81e0-4f77-b2dd-2e44f5c10d06\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425"
Oct 30 13:08:23.321532 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:23.321184    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db45cea6-0b96-46b2-94ff-48caa8747410-istio-token\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q\" (UID: \"db45cea6-0b96-46b2-94ff-48caa8747410\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q"
Oct 30 13:08:23.321532 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:23.321220    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9059a7c7-81e0-4f77-b2dd-2e44f5c10d06-workload-certs\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425\" (UID: \"9059a7c7-81e0-4f77-b2dd-2e44f5c10d06\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425"
Oct 30 13:08:23.321532 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:23.321290    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db45cea6-0b96-46b2-94ff-48caa8747410-istio-podinfo\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q\" (UID: \"db45cea6-0b96-46b2-94ff-48caa8747410\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q"
Oct 30 13:08:23.321532 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:23.321325    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9059a7c7-81e0-4f77-b2dd-2e44f5c10d06-credential-socket\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425\" (UID: \"9059a7c7-81e0-4f77-b2dd-2e44f5c10d06\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425"
Oct 30 13:08:23.321532 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:23.321338    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db45cea6-0b96-46b2-94ff-48caa8747410-workload-certs\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q\" (UID: \"db45cea6-0b96-46b2-94ff-48caa8747410\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q"
Oct 30 13:08:23.321532 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:23.321370    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjj9z\" (UniqueName: \"kubernetes.io/projected/9059a7c7-81e0-4f77-b2dd-2e44f5c10d06-kube-api-access-gjj9z\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425\" (UID: \"9059a7c7-81e0-4f77-b2dd-2e44f5c10d06\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425"
Oct 30 13:08:23.321532 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:23.321400    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9059a7c7-81e0-4f77-b2dd-2e44f5c10d06-istio-envoy\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425\" (UID: \"9059a7c7-81e0-4f77-b2dd-2e44f5c10d06\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425"
Oct 30 13:08:23.321532 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:23.321520    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db45cea6-0b96-46b2-94ff-48caa8747410-workload-socket\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q\" (UID: \"db45cea6-0b96-46b2-94ff-48caa8747410\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q"
Oct 30 13:08:23.322036 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:23.321731    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db45cea6-0b96-46b2-94ff-48caa8747410-credential-socket\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q\" (UID: \"db45cea6-0b96-46b2-94ff-48caa8747410\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q"
Oct 30 13:08:23.322036 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:23.321832    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9059a7c7-81e0-4f77-b2dd-2e44f5c10d06-istiod-ca-cert\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425\" (UID: \"9059a7c7-81e0-4f77-b2dd-2e44f5c10d06\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425"
Oct 30 13:08:23.322036 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:23.321964    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db45cea6-0b96-46b2-94ff-48caa8747410-istiod-ca-cert\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q\" (UID: \"db45cea6-0b96-46b2-94ff-48caa8747410\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q"
Oct 30 13:08:23.322210 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:23.322035    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9059a7c7-81e0-4f77-b2dd-2e44f5c10d06-istio-data\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425\" (UID: \"9059a7c7-81e0-4f77-b2dd-2e44f5c10d06\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425"
Oct 30 13:08:23.322314 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:23.322292    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9059a7c7-81e0-4f77-b2dd-2e44f5c10d06-credential-socket\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425\" (UID: \"9059a7c7-81e0-4f77-b2dd-2e44f5c10d06\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425"
Oct 30 13:08:23.323403 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:23.323378    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db45cea6-0b96-46b2-94ff-48caa8747410-istio-envoy\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q\" (UID: \"db45cea6-0b96-46b2-94ff-48caa8747410\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q"
Oct 30 13:08:23.324013 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:23.323994    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9059a7c7-81e0-4f77-b2dd-2e44f5c10d06-istio-podinfo\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425\" (UID: \"9059a7c7-81e0-4f77-b2dd-2e44f5c10d06\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425"
Oct 30 13:08:23.324097 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:23.323997    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db45cea6-0b96-46b2-94ff-48caa8747410-istio-podinfo\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q\" (UID: \"db45cea6-0b96-46b2-94ff-48caa8747410\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q"
Oct 30 13:08:23.324156 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:23.324144    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9059a7c7-81e0-4f77-b2dd-2e44f5c10d06-istio-envoy\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425\" (UID: \"9059a7c7-81e0-4f77-b2dd-2e44f5c10d06\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425"
Oct 30 13:08:23.361588 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:23.361522    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9059a7c7-81e0-4f77-b2dd-2e44f5c10d06-istio-token\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425\" (UID: \"9059a7c7-81e0-4f77-b2dd-2e44f5c10d06\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425"
Oct 30 13:08:23.361965 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:23.361940    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjj9z\" (UniqueName: \"kubernetes.io/projected/9059a7c7-81e0-4f77-b2dd-2e44f5c10d06-kube-api-access-gjj9z\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425\" (UID: \"9059a7c7-81e0-4f77-b2dd-2e44f5c10d06\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425"
Oct 30 13:08:23.362590 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:23.362561    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfshc\" (UniqueName: \"kubernetes.io/projected/db45cea6-0b96-46b2-94ff-48caa8747410-kube-api-access-lfshc\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q\" (UID: \"db45cea6-0b96-46b2-94ff-48caa8747410\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q"
Oct 30 13:08:23.363211 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:23.363187    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db45cea6-0b96-46b2-94ff-48caa8747410-istio-token\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q\" (UID: \"db45cea6-0b96-46b2-94ff-48caa8747410\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q"
Oct 30 13:08:23.407926 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:23.407893    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q"
Oct 30 13:08:23.425017 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:23.424987    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425"
Oct 30 13:08:23.762590 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:23.762564    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425"]
Oct 30 13:08:23.763201 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:08:23.763175    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9059a7c7_81e0_4f77_b2dd_2e44f5c10d06.slice/crio-1878efcbb7be7a5b1617a6575b516dcda284c86e351c736035d0c1d0d3e8f534 WatchSource:0}: Error finding container 1878efcbb7be7a5b1617a6575b516dcda284c86e351c736035d0c1d0d3e8f534: Status 404 returned error can't find the container with id 1878efcbb7be7a5b1617a6575b516dcda284c86e351c736035d0c1d0d3e8f534
Oct 30 13:08:23.765878 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:23.765846    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:08:23.765962 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:23.765924    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:08:23.765962 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:23.765953    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:08:23.773810 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:23.773789    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q"]
Oct 30 13:08:23.774875 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:08:23.774853    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb45cea6_0b96_46b2_94ff_48caa8747410.slice/crio-1ea2b0dc533b14111a786acda78bbc0d21565a66f0ffcd671d259a0870ed72ec WatchSource:0}: Error finding container 1ea2b0dc533b14111a786acda78bbc0d21565a66f0ffcd671d259a0870ed72ec: Status 404 returned error can't find the container with id 1ea2b0dc533b14111a786acda78bbc0d21565a66f0ffcd671d259a0870ed72ec
Oct 30 13:08:23.777114 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:23.777086    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:08:23.777208 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:23.777156    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:08:23.777208 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:23.777200    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:08:24.770066 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:24.770028    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425" event={"ID":"9059a7c7-81e0-4f77-b2dd-2e44f5c10d06","Type":"ContainerStarted","Data":"aa8476f1323579494f81445c97c4f7b9bf1bb9c2704b99d7d691701a3ced19db"}
Oct 30 13:08:24.770495 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:24.770093    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425" event={"ID":"9059a7c7-81e0-4f77-b2dd-2e44f5c10d06","Type":"ContainerStarted","Data":"1878efcbb7be7a5b1617a6575b516dcda284c86e351c736035d0c1d0d3e8f534"}
Oct 30 13:08:24.771543 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:24.771516    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q" event={"ID":"db45cea6-0b96-46b2-94ff-48caa8747410","Type":"ContainerStarted","Data":"82d874cf78bd4de431a430ad9cc436dde8678c678fabea99abe352c627a0e7b2"}
Oct 30 13:08:24.771698 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:24.771564    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q" event={"ID":"db45cea6-0b96-46b2-94ff-48caa8747410","Type":"ContainerStarted","Data":"1ea2b0dc533b14111a786acda78bbc0d21565a66f0ffcd671d259a0870ed72ec"}
Oct 30 13:08:24.859528 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:24.859473    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425" podStartSLOduration=1.859458933 podStartE2EDuration="1.859458933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:24.842758524 +0000 UTC m=+1004.009123172" watchObservedRunningTime="2026-10-30 13:08:24.859458933 +0000 UTC m=+1004.025823580"
Oct 30 13:08:24.922622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:24.919847    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q" podStartSLOduration=1.9198236 podStartE2EDuration="1.9198236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:24.893857727 +0000 UTC m=+1004.060222401" watchObservedRunningTime="2026-10-30 13:08:24.9198236 +0000 UTC m=+1004.086188249"
Oct 30 13:08:25.417064 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:25.417027    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q"
Oct 30 13:08:25.433286 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:25.428951    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425"
Oct 30 13:08:25.433286 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:25.429750    4364 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 13:08:25.433286 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:25.429849    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q" podUID="db45cea6-0b96-46b2-94ff-48caa8747410" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 13:08:25.434123 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:25.434003    4364 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 13:08:25.434123 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:25.434055    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425" podUID="9059a7c7-81e0-4f77-b2dd-2e44f5c10d06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 13:08:25.651780 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:25.651743    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:08:26.438062 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:26.436360    4364 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 13:08:26.438062 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:26.436422    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425" podUID="9059a7c7-81e0-4f77-b2dd-2e44f5c10d06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 13:08:26.438062 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:26.437126    4364 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 13:08:26.438062 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:26.437163    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q" podUID="db45cea6-0b96-46b2-94ff-48caa8747410" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 13:08:27.410312 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:27.409884    4364 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 13:08:27.410312 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:27.409939    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q" podUID="db45cea6-0b96-46b2-94ff-48caa8747410" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 13:08:27.427230 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:27.427192    4364 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 13:08:27.427511 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:27.427483    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425" podUID="9059a7c7-81e0-4f77-b2dd-2e44f5c10d06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 13:08:27.931669 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:27.922356    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz" podUID="a568f1ce-602d-4069-a41c-9fe1a7635c5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:28.248680 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:28.247715    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6" podUID="b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:28.366623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:28.361776    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4" podUID="0cbe6af7-d884-4613-90b8-192e5ffdb027" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:28.410838 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:28.410800    4364 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 13:08:28.411088 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:28.411057    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q" podUID="db45cea6-0b96-46b2-94ff-48caa8747410" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 13:08:28.421734 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:28.421689    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh" podUID="d41ac9e5-eeed-4e07-a7fa-05d3260630b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:28.428999 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:28.428967    4364 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 13:08:28.429239 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:28.429210    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425" podUID="9059a7c7-81e0-4f77-b2dd-2e44f5c10d06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 13:08:28.937726 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:28.937674    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc" podUID="8911b790-331d-490c-876b-647e6d2fcd14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:28.942624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:28.939023    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz" podUID="a568f1ce-602d-4069-a41c-9fe1a7635c5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:28.947818 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:28.947790    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz"
Oct 30 13:08:29.253966 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:29.253848    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6" podUID="b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:29.255034 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:29.255008    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6"
Oct 30 13:08:29.370992 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:29.370934    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4" podUID="0cbe6af7-d884-4613-90b8-192e5ffdb027" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 13:08:29.386626 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:29.375897    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4"
Oct 30 13:08:29.521898 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:29.521796    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh" podUID="d41ac9e5-eeed-4e07-a7fa-05d3260630b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 30 13:08:29.527731 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:29.527698    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh"
Oct 30 13:08:29.565372 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:29.565330    4364 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 13:08:29.565642 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:29.565591    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425" podUID="9059a7c7-81e0-4f77-b2dd-2e44f5c10d06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 13:08:29.565888 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:29.565872    4364 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 13:08:29.566031 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:29.566012    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q" podUID="db45cea6-0b96-46b2-94ff-48caa8747410" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 13:08:29.697793 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:29.697742    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz" podUID="bc5caf69-a649-47e5-8910-9b8f331dfd4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 13:08:29.911430 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:29.911389    4364 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6" podUID="b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": context deadline exceeded"
Oct 30 13:08:29.947097 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:29.947046    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc" podUID="8911b790-331d-490c-876b-647e6d2fcd14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:29.947576 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:29.947414    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc"
Oct 30 13:08:29.948703 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:29.948330    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz" podUID="a568f1ce-602d-4069-a41c-9fe1a7635c5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 30 13:08:30.275015 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:30.274905    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6" podUID="b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:30.348753 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:30.348703    4364 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc" podUID="8911b790-331d-490c-876b-647e6d2fcd14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:30.381855 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:30.381813    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4" podUID="0cbe6af7-d884-4613-90b8-192e5ffdb027" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 13:08:30.410821 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:30.410787    4364 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 13:08:30.412060 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:30.412024    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q" podUID="db45cea6-0b96-46b2-94ff-48caa8747410" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 13:08:30.430030 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:30.429987    4364 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 13:08:30.430340 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:30.430308    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425" podUID="9059a7c7-81e0-4f77-b2dd-2e44f5c10d06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 13:08:30.575842 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:30.575736    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh" podUID="d41ac9e5-eeed-4e07-a7fa-05d3260630b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:30.768831 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:30.768786    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz" podUID="bc5caf69-a649-47e5-8910-9b8f331dfd4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 13:08:30.769941 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:30.769915    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz"
Oct 30 13:08:30.902622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:30.891261    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz"
Oct 30 13:08:30.926498 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:30.926193    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4"
Oct 30 13:08:30.955374 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:30.954515    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc"
Oct 30 13:08:30.955374 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:30.954644    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz" podUID="a568f1ce-602d-4069-a41c-9fe1a7635c5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:31.066883 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:31.066853    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz"
Oct 30 13:08:31.067068 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:31.066906    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6"
Oct 30 13:08:31.067068 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:31.066936    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh"
Oct 30 13:08:31.444509 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:31.444473    4364 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 13:08:31.444818 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:31.444788    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q" podUID="db45cea6-0b96-46b2-94ff-48caa8747410" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 13:08:31.445033 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:31.445006    4364 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 13:08:31.446838 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:31.446688    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425" podUID="9059a7c7-81e0-4f77-b2dd-2e44f5c10d06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 13:08:32.408374 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:32.408337    4364 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 13:08:32.408901 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:32.408391    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q" podUID="db45cea6-0b96-46b2-94ff-48caa8747410" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 13:08:32.425521 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:32.425488    4364 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 13:08:32.425704 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:32.425536    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425" podUID="9059a7c7-81e0-4f77-b2dd-2e44f5c10d06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 13:08:33.481193 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:33.481158    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q"
Oct 30 13:08:33.483042 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:33.483016    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q"
Oct 30 13:08:33.484757 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:33.484737    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q"
Oct 30 13:08:33.558034 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:33.558000    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425"
Oct 30 13:08:33.559027 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:33.558994    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425"
Oct 30 13:08:33.559988 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:33.559967    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425"
Oct 30 13:08:35.432034 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:35.432001    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29ceb91f-lw2hx"]
Oct 30 13:08:35.435838 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:35.435818    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ceb91f-lw2hx"
Oct 30 13:08:35.487254 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:35.487230    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:08:35.487369 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:35.487291    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\""
Oct 30 13:08:35.535981 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:35.535941    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29ceb91f-lw2hx"]
Oct 30 13:08:35.560741 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:35.560714    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a0e173e-b428-45f5-b91a-095475c67cae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29ceb91f-lw2hx\" (UID: \"1a0e173e-b428-45f5-b91a-095475c67cae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ceb91f-lw2hx"
Oct 30 13:08:35.560910 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:35.560747    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nllb\" (UniqueName: \"kubernetes.io/projected/1a0e173e-b428-45f5-b91a-095475c67cae-kube-api-access-2nllb\") pod \"maas-api-revoke-sub-29ceb91f-lw2hx\" (UID: \"1a0e173e-b428-45f5-b91a-095475c67cae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ceb91f-lw2hx"
Oct 30 13:08:35.560910 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:35.560772    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a0e173e-b428-45f5-b91a-095475c67cae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29ceb91f-lw2hx\" (UID: \"1a0e173e-b428-45f5-b91a-095475c67cae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ceb91f-lw2hx"
Oct 30 13:08:35.661903 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:35.661872    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a0e173e-b428-45f5-b91a-095475c67cae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29ceb91f-lw2hx\" (UID: \"1a0e173e-b428-45f5-b91a-095475c67cae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ceb91f-lw2hx"
Oct 30 13:08:35.662086 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:35.661910    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nllb\" (UniqueName: \"kubernetes.io/projected/1a0e173e-b428-45f5-b91a-095475c67cae-kube-api-access-2nllb\") pod \"maas-api-revoke-sub-29ceb91f-lw2hx\" (UID: \"1a0e173e-b428-45f5-b91a-095475c67cae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ceb91f-lw2hx"
Oct 30 13:08:35.662086 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:35.661942    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a0e173e-b428-45f5-b91a-095475c67cae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29ceb91f-lw2hx\" (UID: \"1a0e173e-b428-45f5-b91a-095475c67cae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ceb91f-lw2hx"
Oct 30 13:08:35.662856 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:35.662826    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a0e173e-b428-45f5-b91a-095475c67cae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29ceb91f-lw2hx\" (UID: \"1a0e173e-b428-45f5-b91a-095475c67cae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ceb91f-lw2hx"
Oct 30 13:08:35.664353 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:35.664327    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a0e173e-b428-45f5-b91a-095475c67cae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29ceb91f-lw2hx\" (UID: \"1a0e173e-b428-45f5-b91a-095475c67cae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ceb91f-lw2hx"
Oct 30 13:08:35.693997 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:35.693937    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nllb\" (UniqueName: \"kubernetes.io/projected/1a0e173e-b428-45f5-b91a-095475c67cae-kube-api-access-2nllb\") pod \"maas-api-revoke-sub-29ceb91f-lw2hx\" (UID: \"1a0e173e-b428-45f5-b91a-095475c67cae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ceb91f-lw2hx"
Oct 30 13:08:35.750086 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:35.750055    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ceb91f-lw2hx"
Oct 30 13:08:36.053857 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:08:36.053828    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a0e173e_b428_45f5_b91a_095475c67cae.slice/crio-ec3d4f79ce4fd132e14807c7b8e4335c62db26db0c21329401e16cef5c9597ec WatchSource:0}: Error finding container ec3d4f79ce4fd132e14807c7b8e4335c62db26db0c21329401e16cef5c9597ec: Status 404 returned error can't find the container with id ec3d4f79ce4fd132e14807c7b8e4335c62db26db0c21329401e16cef5c9597ec
Oct 30 13:08:36.054999 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:36.054976    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29ceb91f-lw2hx"]
Oct 30 13:08:36.202418 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:36.202375    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ceb91f-lw2hx" event={"ID":"1a0e173e-b428-45f5-b91a-095475c67cae","Type":"ContainerStarted","Data":"ec3d4f79ce4fd132e14807c7b8e4335c62db26db0c21329401e16cef5c9597ec"}
Oct 30 13:08:37.238646 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:37.238522    4364 generic.go:356] "Generic (PLEG): container finished" podID="1a0e173e-b428-45f5-b91a-095475c67cae" containerID="f635944dcfd581313882ad0be4f66259c0ebfed92bbf7bd52e44161d469f6803" exitCode=0
Oct 30 13:08:37.239062 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:37.238851    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ceb91f-lw2hx" event={"ID":"1a0e173e-b428-45f5-b91a-095475c67cae","Type":"ContainerDied","Data":"f635944dcfd581313882ad0be4f66259c0ebfed92bbf7bd52e44161d469f6803"}
Oct 30 13:08:37.962628 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:37.962584    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb"]
Oct 30 13:08:37.966723 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:37.966702    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb"
Oct 30 13:08:37.994663 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:37.994638    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb"]
Oct 30 13:08:38.096529 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.096499    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3-istiod-ca-cert\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb\" (UID: \"c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb"
Oct 30 13:08:38.096796 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.096765    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3-istio-token\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb\" (UID: \"c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb"
Oct 30 13:08:38.096860 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.096815    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3-workload-certs\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb\" (UID: \"c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb"
Oct 30 13:08:38.096910 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.096895    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3-workload-socket\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb\" (UID: \"c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb"
Oct 30 13:08:38.097026 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.096999    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnfqg\" (UniqueName: \"kubernetes.io/projected/c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3-kube-api-access-bnfqg\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb\" (UID: \"c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb"
Oct 30 13:08:38.097153 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.097135    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3-istio-envoy\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb\" (UID: \"c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb"
Oct 30 13:08:38.097217 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.097166    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3-istio-podinfo\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb\" (UID: \"c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb"
Oct 30 13:08:38.097323 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.097308    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3-istio-data\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb\" (UID: \"c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb"
Oct 30 13:08:38.097385 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.097335    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3-credential-socket\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb\" (UID: \"c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb"
Oct 30 13:08:38.165143 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.165122    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l"]
Oct 30 13:08:38.169640 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.169615    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l"
Oct 30 13:08:38.199453 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.199422    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3-istio-data\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb\" (UID: \"c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb"
Oct 30 13:08:38.200538 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.199474    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3-credential-socket\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb\" (UID: \"c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb"
Oct 30 13:08:38.200538 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.199649    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3-istiod-ca-cert\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb\" (UID: \"c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb"
Oct 30 13:08:38.200538 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.199772    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3-istio-token\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb\" (UID: \"c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb"
Oct 30 13:08:38.200538 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.199809    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3-workload-certs\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb\" (UID: \"c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb"
Oct 30 13:08:38.200538 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.199845    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3-workload-socket\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb\" (UID: \"c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb"
Oct 30 13:08:38.200538 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.199908    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnfqg\" (UniqueName: \"kubernetes.io/projected/c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3-kube-api-access-bnfqg\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb\" (UID: \"c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb"
Oct 30 13:08:38.200538 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.199945    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3-istio-envoy\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb\" (UID: \"c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb"
Oct 30 13:08:38.200538 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.199985    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3-istio-podinfo\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb\" (UID: \"c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb"
Oct 30 13:08:38.201154 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.201125    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3-workload-socket\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb\" (UID: \"c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb"
Oct 30 13:08:38.201378 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.201349    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3-istio-data\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb\" (UID: \"c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb"
Oct 30 13:08:38.201878 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.201851    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3-workload-certs\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb\" (UID: \"c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb"
Oct 30 13:08:38.202168 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.202133    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3-istiod-ca-cert\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb\" (UID: \"c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb"
Oct 30 13:08:38.202410 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.202378    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3-credential-socket\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb\" (UID: \"c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb"
Oct 30 13:08:38.203645 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.203621    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3-istio-podinfo\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb\" (UID: \"c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb"
Oct 30 13:08:38.206773 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.206746    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l"]
Oct 30 13:08:38.209964 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.209714    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3-istio-envoy\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb\" (UID: \"c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb"
Oct 30 13:08:38.229342 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.229282    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnfqg\" (UniqueName: \"kubernetes.io/projected/c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3-kube-api-access-bnfqg\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb\" (UID: \"c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb"
Oct 30 13:08:38.229647 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.229621    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3-istio-token\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb\" (UID: \"c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb"
Oct 30 13:08:38.282321 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.282294    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb"
Oct 30 13:08:38.301011 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.300977    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6ba17d2-cc66-444e-9976-36496fe9db21-istio-podinfo\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l\" (UID: \"c6ba17d2-cc66-444e-9976-36496fe9db21\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l"
Oct 30 13:08:38.301108 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.301076    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6ba17d2-cc66-444e-9976-36496fe9db21-workload-certs\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l\" (UID: \"c6ba17d2-cc66-444e-9976-36496fe9db21\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l"
Oct 30 13:08:38.301166 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.301116    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gl252\" (UniqueName: \"kubernetes.io/projected/c6ba17d2-cc66-444e-9976-36496fe9db21-kube-api-access-gl252\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l\" (UID: \"c6ba17d2-cc66-444e-9976-36496fe9db21\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l"
Oct 30 13:08:38.301166 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.301147    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6ba17d2-cc66-444e-9976-36496fe9db21-istiod-ca-cert\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l\" (UID: \"c6ba17d2-cc66-444e-9976-36496fe9db21\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l"
Oct 30 13:08:38.301279 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.301198    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6ba17d2-cc66-444e-9976-36496fe9db21-istio-token\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l\" (UID: \"c6ba17d2-cc66-444e-9976-36496fe9db21\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l"
Oct 30 13:08:38.301279 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.301238    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6ba17d2-cc66-444e-9976-36496fe9db21-workload-socket\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l\" (UID: \"c6ba17d2-cc66-444e-9976-36496fe9db21\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l"
Oct 30 13:08:38.301375 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.301290    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6ba17d2-cc66-444e-9976-36496fe9db21-istio-data\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l\" (UID: \"c6ba17d2-cc66-444e-9976-36496fe9db21\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l"
Oct 30 13:08:38.301429 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.301378    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6ba17d2-cc66-444e-9976-36496fe9db21-istio-envoy\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l\" (UID: \"c6ba17d2-cc66-444e-9976-36496fe9db21\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l"
Oct 30 13:08:38.301429 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.301416    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6ba17d2-cc66-444e-9976-36496fe9db21-credential-socket\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l\" (UID: \"c6ba17d2-cc66-444e-9976-36496fe9db21\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l"
Oct 30 13:08:38.400516 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.400489    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ceb91f-lw2hx"
Oct 30 13:08:38.402829 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.402803    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6ba17d2-cc66-444e-9976-36496fe9db21-istio-token\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l\" (UID: \"c6ba17d2-cc66-444e-9976-36496fe9db21\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l"
Oct 30 13:08:38.402944 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.402861    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6ba17d2-cc66-444e-9976-36496fe9db21-workload-socket\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l\" (UID: \"c6ba17d2-cc66-444e-9976-36496fe9db21\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l"
Oct 30 13:08:38.402944 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.402919    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6ba17d2-cc66-444e-9976-36496fe9db21-istio-data\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l\" (UID: \"c6ba17d2-cc66-444e-9976-36496fe9db21\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l"
Oct 30 13:08:38.403059 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.403013    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6ba17d2-cc66-444e-9976-36496fe9db21-istio-envoy\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l\" (UID: \"c6ba17d2-cc66-444e-9976-36496fe9db21\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l"
Oct 30 13:08:38.403059 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.403052    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6ba17d2-cc66-444e-9976-36496fe9db21-credential-socket\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l\" (UID: \"c6ba17d2-cc66-444e-9976-36496fe9db21\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l"
Oct 30 13:08:38.403190 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.403180    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6ba17d2-cc66-444e-9976-36496fe9db21-istio-podinfo\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l\" (UID: \"c6ba17d2-cc66-444e-9976-36496fe9db21\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l"
Oct 30 13:08:38.404075 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.403270    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6ba17d2-cc66-444e-9976-36496fe9db21-workload-certs\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l\" (UID: \"c6ba17d2-cc66-444e-9976-36496fe9db21\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l"
Oct 30 13:08:38.404075 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.403302    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6ba17d2-cc66-444e-9976-36496fe9db21-workload-socket\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l\" (UID: \"c6ba17d2-cc66-444e-9976-36496fe9db21\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l"
Oct 30 13:08:38.404075 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.403320    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl252\" (UniqueName: \"kubernetes.io/projected/c6ba17d2-cc66-444e-9976-36496fe9db21-kube-api-access-gl252\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l\" (UID: \"c6ba17d2-cc66-444e-9976-36496fe9db21\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l"
Oct 30 13:08:38.404075 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.403371    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6ba17d2-cc66-444e-9976-36496fe9db21-istiod-ca-cert\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l\" (UID: \"c6ba17d2-cc66-444e-9976-36496fe9db21\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l"
Oct 30 13:08:38.404075 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.403409    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6ba17d2-cc66-444e-9976-36496fe9db21-istio-data\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l\" (UID: \"c6ba17d2-cc66-444e-9976-36496fe9db21\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l"
Oct 30 13:08:38.404075 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.403845    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6ba17d2-cc66-444e-9976-36496fe9db21-workload-certs\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l\" (UID: \"c6ba17d2-cc66-444e-9976-36496fe9db21\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l"
Oct 30 13:08:38.404075 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.403987    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6ba17d2-cc66-444e-9976-36496fe9db21-credential-socket\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l\" (UID: \"c6ba17d2-cc66-444e-9976-36496fe9db21\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l"
Oct 30 13:08:38.404360 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.404136    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6ba17d2-cc66-444e-9976-36496fe9db21-istiod-ca-cert\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l\" (UID: \"c6ba17d2-cc66-444e-9976-36496fe9db21\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l"
Oct 30 13:08:38.406101 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.406074    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6ba17d2-cc66-444e-9976-36496fe9db21-istio-envoy\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l\" (UID: \"c6ba17d2-cc66-444e-9976-36496fe9db21\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l"
Oct 30 13:08:38.406215 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.406194    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6ba17d2-cc66-444e-9976-36496fe9db21-istio-podinfo\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l\" (UID: \"c6ba17d2-cc66-444e-9976-36496fe9db21\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l"
Oct 30 13:08:38.427876 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.427847    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl252\" (UniqueName: \"kubernetes.io/projected/c6ba17d2-cc66-444e-9976-36496fe9db21-kube-api-access-gl252\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l\" (UID: \"c6ba17d2-cc66-444e-9976-36496fe9db21\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l"
Oct 30 13:08:38.427876 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.427862    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6ba17d2-cc66-444e-9976-36496fe9db21-istio-token\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l\" (UID: \"c6ba17d2-cc66-444e-9976-36496fe9db21\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l"
Oct 30 13:08:38.486572 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.486506    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l"
Oct 30 13:08:38.504246 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.504218    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a0e173e-b428-45f5-b91a-095475c67cae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a0e173e-b428-45f5-b91a-095475c67cae-maas-api-openshift-ca\") pod \"1a0e173e-b428-45f5-b91a-095475c67cae\" (UID: \"1a0e173e-b428-45f5-b91a-095475c67cae\") "
Oct 30 13:08:38.504447 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.504429    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a0e173e-b428-45f5-b91a-095475c67cae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a0e173e-b428-45f5-b91a-095475c67cae-maas-api-xks-ca\") pod \"1a0e173e-b428-45f5-b91a-095475c67cae\" (UID: \"1a0e173e-b428-45f5-b91a-095475c67cae\") "
Oct 30 13:08:38.504490 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.504467    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a0e173e-b428-45f5-b91a-095475c67cae-kube-api-access-2nllb\" (UniqueName: \"kubernetes.io/projected/1a0e173e-b428-45f5-b91a-095475c67cae-kube-api-access-2nllb\") pod \"1a0e173e-b428-45f5-b91a-095475c67cae\" (UID: \"1a0e173e-b428-45f5-b91a-095475c67cae\") "
Oct 30 13:08:38.504544 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.504521    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a0e173e-b428-45f5-b91a-095475c67cae-maas-api-openshift-ca" pod "1a0e173e-b428-45f5-b91a-095475c67cae" (UID: "1a0e173e-b428-45f5-b91a-095475c67cae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:08:38.505076 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.505053    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a0e173e-b428-45f5-b91a-095475c67cae-maas-api-openshift-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:08:38.506423 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.506402    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a0e173e-b428-45f5-b91a-095475c67cae-maas-api-xks-ca" pod "1a0e173e-b428-45f5-b91a-095475c67cae" (UID: "1a0e173e-b428-45f5-b91a-095475c67cae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:08:38.506500 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.506482    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a0e173e-b428-45f5-b91a-095475c67cae-kube-api-access-2nllb" pod "1a0e173e-b428-45f5-b91a-095475c67cae" (UID: "1a0e173e-b428-45f5-b91a-095475c67cae"). InnerVolumeSpecName "kube-api-access-2nllb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:08:38.607196 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.607160    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a0e173e-b428-45f5-b91a-095475c67cae-maas-api-xks-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:08:38.607333 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.607203    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nllb\" (UniqueName: \"kubernetes.io/projected/1a0e173e-b428-45f5-b91a-095475c67cae-kube-api-access-2nllb\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:08:38.626293 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:08:38.626267    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0a2a8a5_ed1c_4b5e_829b_0e36073ca5d3.slice/crio-f780ebddbba70b732c603522b531da0e0a8c76d2053e3ee200b3d80b9dec0c2c WatchSource:0}: Error finding container f780ebddbba70b732c603522b531da0e0a8c76d2053e3ee200b3d80b9dec0c2c: Status 404 returned error can't find the container with id f780ebddbba70b732c603522b531da0e0a8c76d2053e3ee200b3d80b9dec0c2c
Oct 30 13:08:38.630770 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.627303    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb"]
Oct 30 13:08:38.632631 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.632575    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:08:38.632719 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.632668    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:08:38.632773 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.632715    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:08:38.709422 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.709387    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv"]
Oct 30 13:08:38.710398 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.710376    4364 cpu_manager.go:414] "containerMap: removing container" podUID="1a0e173e-b428-45f5-b91a-095475c67cae" containerName="revoke-keys"
Oct 30 13:08:38.710474 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.710401    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a0e173e-b428-45f5-b91a-095475c67cae" containerName="revoke-keys"
Oct 30 13:08:38.710639 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.710617    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a0e173e-b428-45f5-b91a-095475c67cae" containerName="revoke-keys"
Oct 30 13:08:38.714713 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.714697    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv"
Oct 30 13:08:38.740930 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.737260    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv"]
Oct 30 13:08:38.758521 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.758489    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq"]
Oct 30 13:08:38.763024 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.763004    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq"
Oct 30 13:08:38.784637 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.783893    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq"]
Oct 30 13:08:38.812639 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.811712    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a6cf8c7-ad37-46bc-a392-90de84af768e-credential-socket\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv\" (UID: \"4a6cf8c7-ad37-46bc-a392-90de84af768e\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv"
Oct 30 13:08:38.812639 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.811769    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a6cf8c7-ad37-46bc-a392-90de84af768e-istio-data\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv\" (UID: \"4a6cf8c7-ad37-46bc-a392-90de84af768e\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv"
Oct 30 13:08:38.812639 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.811815    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a6cf8c7-ad37-46bc-a392-90de84af768e-workload-certs\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv\" (UID: \"4a6cf8c7-ad37-46bc-a392-90de84af768e\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv"
Oct 30 13:08:38.812639 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.811843    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a6cf8c7-ad37-46bc-a392-90de84af768e-istio-token\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv\" (UID: \"4a6cf8c7-ad37-46bc-a392-90de84af768e\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv"
Oct 30 13:08:38.812639 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.811874    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a6cf8c7-ad37-46bc-a392-90de84af768e-istio-envoy\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv\" (UID: \"4a6cf8c7-ad37-46bc-a392-90de84af768e\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv"
Oct 30 13:08:38.812639 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.811916    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmg87\" (UniqueName: \"kubernetes.io/projected/4a6cf8c7-ad37-46bc-a392-90de84af768e-kube-api-access-tmg87\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv\" (UID: \"4a6cf8c7-ad37-46bc-a392-90de84af768e\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv"
Oct 30 13:08:38.812639 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.812079    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a6cf8c7-ad37-46bc-a392-90de84af768e-istiod-ca-cert\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv\" (UID: \"4a6cf8c7-ad37-46bc-a392-90de84af768e\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv"
Oct 30 13:08:38.812639 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.812123    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a6cf8c7-ad37-46bc-a392-90de84af768e-workload-socket\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv\" (UID: \"4a6cf8c7-ad37-46bc-a392-90de84af768e\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv"
Oct 30 13:08:38.812639 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.812191    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a6cf8c7-ad37-46bc-a392-90de84af768e-istio-podinfo\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv\" (UID: \"4a6cf8c7-ad37-46bc-a392-90de84af768e\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv"
Oct 30 13:08:38.814104 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.814084    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l"]
Oct 30 13:08:38.822378 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:08:38.820785    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6ba17d2_cc66_444e_9976_36496fe9db21.slice/crio-92bd143314802ad478f01ebd07ff80df31f8a63fc704d248d3d50b6b7f84cba5 WatchSource:0}: Error finding container 92bd143314802ad478f01ebd07ff80df31f8a63fc704d248d3d50b6b7f84cba5: Status 404 returned error can't find the container with id 92bd143314802ad478f01ebd07ff80df31f8a63fc704d248d3d50b6b7f84cba5
Oct 30 13:08:38.823442 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.823412    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:08:38.823556 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.823484    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:08:38.823556 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.823529    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:08:38.913492 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.913457    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d08eaa5-399d-4d90-8a47-3c060abfb552-credential-socket\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq\" (UID: \"1d08eaa5-399d-4d90-8a47-3c060abfb552\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq"
Oct 30 13:08:38.913650 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.913539    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmg87\" (UniqueName: \"kubernetes.io/projected/4a6cf8c7-ad37-46bc-a392-90de84af768e-kube-api-access-tmg87\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv\" (UID: \"4a6cf8c7-ad37-46bc-a392-90de84af768e\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv"
Oct 30 13:08:38.913650 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.913577    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d08eaa5-399d-4d90-8a47-3c060abfb552-istio-data\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq\" (UID: \"1d08eaa5-399d-4d90-8a47-3c060abfb552\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq"
Oct 30 13:08:38.913774 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.913656    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a6cf8c7-ad37-46bc-a392-90de84af768e-istio-podinfo\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv\" (UID: \"4a6cf8c7-ad37-46bc-a392-90de84af768e\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv"
Oct 30 13:08:38.913774 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.913690    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d08eaa5-399d-4d90-8a47-3c060abfb552-istio-podinfo\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq\" (UID: \"1d08eaa5-399d-4d90-8a47-3c060abfb552\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq"
Oct 30 13:08:38.913774 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.913713    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d08eaa5-399d-4d90-8a47-3c060abfb552-istiod-ca-cert\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq\" (UID: \"1d08eaa5-399d-4d90-8a47-3c060abfb552\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq"
Oct 30 13:08:38.913922 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.913870    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d08eaa5-399d-4d90-8a47-3c060abfb552-istio-envoy\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq\" (UID: \"1d08eaa5-399d-4d90-8a47-3c060abfb552\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq"
Oct 30 13:08:38.913993 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.913975    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a6cf8c7-ad37-46bc-a392-90de84af768e-istio-envoy\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv\" (UID: \"4a6cf8c7-ad37-46bc-a392-90de84af768e\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv"
Oct 30 13:08:38.914079 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.914063    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a6cf8c7-ad37-46bc-a392-90de84af768e-workload-certs\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv\" (UID: \"4a6cf8c7-ad37-46bc-a392-90de84af768e\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv"
Oct 30 13:08:38.914218 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.914193    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a6cf8c7-ad37-46bc-a392-90de84af768e-istio-token\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv\" (UID: \"4a6cf8c7-ad37-46bc-a392-90de84af768e\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv"
Oct 30 13:08:38.914312 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.914264    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a6cf8c7-ad37-46bc-a392-90de84af768e-istiod-ca-cert\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv\" (UID: \"4a6cf8c7-ad37-46bc-a392-90de84af768e\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv"
Oct 30 13:08:38.914374 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.914320    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a6cf8c7-ad37-46bc-a392-90de84af768e-workload-socket\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv\" (UID: \"4a6cf8c7-ad37-46bc-a392-90de84af768e\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv"
Oct 30 13:08:38.914427 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.914383    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d08eaa5-399d-4d90-8a47-3c060abfb552-istio-token\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq\" (UID: \"1d08eaa5-399d-4d90-8a47-3c060abfb552\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq"
Oct 30 13:08:38.914482 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.914444    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a6cf8c7-ad37-46bc-a392-90de84af768e-credential-socket\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv\" (UID: \"4a6cf8c7-ad37-46bc-a392-90de84af768e\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv"
Oct 30 13:08:38.914482 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.914469    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a6cf8c7-ad37-46bc-a392-90de84af768e-workload-certs\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv\" (UID: \"4a6cf8c7-ad37-46bc-a392-90de84af768e\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv"
Oct 30 13:08:38.914634 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.914489    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a6cf8c7-ad37-46bc-a392-90de84af768e-istio-data\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv\" (UID: \"4a6cf8c7-ad37-46bc-a392-90de84af768e\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv"
Oct 30 13:08:38.914634 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.914540    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d08eaa5-399d-4d90-8a47-3c060abfb552-workload-certs\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq\" (UID: \"1d08eaa5-399d-4d90-8a47-3c060abfb552\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq"
Oct 30 13:08:38.914634 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.914591    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccvfb\" (UniqueName: \"kubernetes.io/projected/1d08eaa5-399d-4d90-8a47-3c060abfb552-kube-api-access-ccvfb\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq\" (UID: \"1d08eaa5-399d-4d90-8a47-3c060abfb552\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq"
Oct 30 13:08:38.914799 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.914732    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d08eaa5-399d-4d90-8a47-3c060abfb552-workload-socket\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq\" (UID: \"1d08eaa5-399d-4d90-8a47-3c060abfb552\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq"
Oct 30 13:08:38.914858 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.914806    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a6cf8c7-ad37-46bc-a392-90de84af768e-istio-data\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv\" (UID: \"4a6cf8c7-ad37-46bc-a392-90de84af768e\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv"
Oct 30 13:08:38.915045 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.915017    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a6cf8c7-ad37-46bc-a392-90de84af768e-workload-socket\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv\" (UID: \"4a6cf8c7-ad37-46bc-a392-90de84af768e\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv"
Oct 30 13:08:38.915155 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.915134    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a6cf8c7-ad37-46bc-a392-90de84af768e-credential-socket\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv\" (UID: \"4a6cf8c7-ad37-46bc-a392-90de84af768e\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv"
Oct 30 13:08:38.915214 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.915179    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a6cf8c7-ad37-46bc-a392-90de84af768e-istiod-ca-cert\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv\" (UID: \"4a6cf8c7-ad37-46bc-a392-90de84af768e\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv"
Oct 30 13:08:38.916453 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.916430    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a6cf8c7-ad37-46bc-a392-90de84af768e-istio-podinfo\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv\" (UID: \"4a6cf8c7-ad37-46bc-a392-90de84af768e\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv"
Oct 30 13:08:38.917256 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.917237    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a6cf8c7-ad37-46bc-a392-90de84af768e-istio-envoy\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv\" (UID: \"4a6cf8c7-ad37-46bc-a392-90de84af768e\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv"
Oct 30 13:08:38.951637 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.948777    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr"]
Oct 30 13:08:38.951637 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.950966    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a6cf8c7-ad37-46bc-a392-90de84af768e-istio-token\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv\" (UID: \"4a6cf8c7-ad37-46bc-a392-90de84af768e\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv"
Oct 30 13:08:38.951637 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.951237    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmg87\" (UniqueName: \"kubernetes.io/projected/4a6cf8c7-ad37-46bc-a392-90de84af768e-kube-api-access-tmg87\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv\" (UID: \"4a6cf8c7-ad37-46bc-a392-90de84af768e\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv"
Oct 30 13:08:38.955166 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.955148    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr"
Oct 30 13:08:38.981033 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:38.981002    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr"]
Oct 30 13:08:39.016413 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:39.016380    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/747bdafd-1a41-4b0e-9547-f14fe568d3f1-istiod-ca-cert\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr\" (UID: \"747bdafd-1a41-4b0e-9547-f14fe568d3f1\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr"
Oct 30 13:08:39.016704 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:39.016664    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/747bdafd-1a41-4b0e-9547-f14fe568d3f1-istio-envoy\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr\" (UID: \"747bdafd-1a41-4b0e-9547-f14fe568d3f1\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr"
Oct 30 13:08:39.016800 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:39.016702    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/747bdafd-1a41-4b0e-9547-f14fe568d3f1-istio-data\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr\" (UID: \"747bdafd-1a41-4b0e-9547-f14fe568d3f1\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr"
Oct 30 13:08:39.016852 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:39.016803    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d08eaa5-399d-4d90-8a47-3c060abfb552-istio-token\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq\" (UID: \"1d08eaa5-399d-4d90-8a47-3c060abfb552\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq"
Oct 30 13:08:39.016918 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:39.016891    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/747bdafd-1a41-4b0e-9547-f14fe568d3f1-workload-certs\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr\" (UID: \"747bdafd-1a41-4b0e-9547-f14fe568d3f1\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr"
Oct 30 13:08:39.016969 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:39.016943    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d08eaa5-399d-4d90-8a47-3c060abfb552-workload-certs\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq\" (UID: \"1d08eaa5-399d-4d90-8a47-3c060abfb552\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq"
Oct 30 13:08:39.017022 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:39.016981    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/747bdafd-1a41-4b0e-9547-f14fe568d3f1-credential-socket\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr\" (UID: \"747bdafd-1a41-4b0e-9547-f14fe568d3f1\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr"
Oct 30 13:08:39.017022 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:39.017015    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccvfb\" (UniqueName: \"kubernetes.io/projected/1d08eaa5-399d-4d90-8a47-3c060abfb552-kube-api-access-ccvfb\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq\" (UID: \"1d08eaa5-399d-4d90-8a47-3c060abfb552\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq"
Oct 30 13:08:39.017093 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:39.017051    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d08eaa5-399d-4d90-8a47-3c060abfb552-workload-socket\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq\" (UID: \"1d08eaa5-399d-4d90-8a47-3c060abfb552\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq"
Oct 30 13:08:39.017136 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:39.017090    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d08eaa5-399d-4d90-8a47-3c060abfb552-credential-socket\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq\" (UID: \"1d08eaa5-399d-4d90-8a47-3c060abfb552\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq"
Oct 30 13:08:39.017136 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:39.017123    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/747bdafd-1a41-4b0e-9547-f14fe568d3f1-istio-token\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr\" (UID: \"747bdafd-1a41-4b0e-9547-f14fe568d3f1\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr"
Oct 30 13:08:39.017214 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:39.017167    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d08eaa5-399d-4d90-8a47-3c060abfb552-istio-data\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq\" (UID: \"1d08eaa5-399d-4d90-8a47-3c060abfb552\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq"
Oct 30 13:08:39.017250 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:39.017221    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/747bdafd-1a41-4b0e-9547-f14fe568d3f1-istio-podinfo\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr\" (UID: \"747bdafd-1a41-4b0e-9547-f14fe568d3f1\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr"
Oct 30 13:08:39.017292 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:39.017272    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/747bdafd-1a41-4b0e-9547-f14fe568d3f1-workload-socket\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr\" (UID: \"747bdafd-1a41-4b0e-9547-f14fe568d3f1\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr"
Oct 30 13:08:39.017329 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:39.017316    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d08eaa5-399d-4d90-8a47-3c060abfb552-istio-podinfo\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq\" (UID: \"1d08eaa5-399d-4d90-8a47-3c060abfb552\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq"
Oct 30 13:08:39.017369 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:39.017339    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d08eaa5-399d-4d90-8a47-3c060abfb552-istiod-ca-cert\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq\" (UID: \"1d08eaa5-399d-4d90-8a47-3c060abfb552\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq"
Oct 30 13:08:39.017408 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:39.017370    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d08eaa5-399d-4d90-8a47-3c060abfb552-istio-envoy\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq\" (UID: \"1d08eaa5-399d-4d90-8a47-3c060abfb552\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq"
Oct 30 13:08:39.017458 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:39.017406    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sr82m\" (UniqueName: \"kubernetes.io/projected/747bdafd-1a41-4b0e-9547-f14fe568d3f1-kube-api-access-sr82m\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr\" (UID: \"747bdafd-1a41-4b0e-9547-f14fe568d3f1\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr"
Oct 30 13:08:39.018451 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:39.018418    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d08eaa5-399d-4d90-8a47-3c060abfb552-workload-certs\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq\" (UID: \"1d08eaa5-399d-4d90-8a47-3c060abfb552\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq"
Oct 30 13:08:39.018976 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:39.018952    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d08eaa5-399d-4d90-8a47-3c060abfb552-workload-socket\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq\" (UID: \"1d08eaa5-399d-4d90-8a47-3c060abfb552\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq"
Oct 30 13:08:39.019085 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:39.018985    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d08eaa5-399d-4d90-8a47-3c060abfb552-credential-socket\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq\" (UID: \"1d08eaa5-399d-4d90-8a47-3c060abfb552\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq"
Oct 30 13:08:39.019484 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:39.019464    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d08eaa5-399d-4d90-8a47-3c060abfb552-istio-data\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq\" (UID: \"1d08eaa5-399d-4d90-8a47-3c060abfb552\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq"
Oct 30 13:08:39.019983 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:39.019941    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d08eaa5-399d-4d90-8a47-3c060abfb552-istiod-ca-cert\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq\" (UID: \"1d08eaa5-399d-4d90-8a47-3c060abfb552\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq"
Oct 30 13:08:39.022075 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:39.022047    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d08eaa5-399d-4d90-8a47-3c060abfb552-istio-podinfo\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq\" (UID: \"1d08eaa5-399d-4d90-8a47-3c060abfb552\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq"
Oct 30 13:08:39.024127 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:39.024101    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d08eaa5-399d-4d90-8a47-3c060abfb552-istio-envoy\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq\" (UID: \"1d08eaa5-399d-4d90-8a47-3c060abfb552\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq"
Oct 30 13:08:39.029297 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:39.029154    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv"
Oct 30 13:08:39.045254 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:39.045202    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d08eaa5-399d-4d90-8a47-3c060abfb552-istio-token\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq\" (UID: \"1d08eaa5-399d-4d90-8a47-3c060abfb552\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq"
Oct 30 13:08:39.047023 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:39.046993    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccvfb\" (UniqueName: \"kubernetes.io/projected/1d08eaa5-399d-4d90-8a47-3c060abfb552-kube-api-access-ccvfb\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq\" (UID: \"1d08eaa5-399d-4d90-8a47-3c060abfb552\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq"
Oct 30 13:08:39.087472 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:39.087071    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq"
Oct 30 13:08:39.123752 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:39.120438    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/747bdafd-1a41-4b0e-9547-f14fe568d3f1-workload-socket\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr\" (UID: \"747bdafd-1a41-4b0e-9547-f14fe568d3f1\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr"
Oct 30 13:08:39.123752 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:39.120520    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sr82m\" (UniqueName: \"kubernetes.io/projected/747bdafd-1a41-4b0e-9547-f14fe568d3f1-kube-api-access-sr82m\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr\" (UID: \"747bdafd-1a41-4b0e-9547-f14fe568d3f1\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr"
Oct 30 13:08:39.123752 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:39.120635    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/747bdafd-1a41-4b0e-9547-f14fe568d3f1-istiod-ca-cert\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr\" (UID: \"747bdafd-1a41-4b0e-9547-f14fe568d3f1\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr"
Oct 30 13:08:39.123752 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:39.120760    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/747bdafd-1a41-4b0e-9547-f14fe568d3f1-istio-envoy\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr\" (UID: \"747bdafd-1a41-4b0e-9547-f14fe568d3f1\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr"
Oct 30 13:08:39.123752 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:39.120793    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/747bdafd-1a41-4b0e-9547-f14fe568d3f1-istio-data\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr\" (UID: \"747bdafd-1a41-4b0e-9547-f14fe568d3f1\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr"
Oct 30 13:08:39.123752 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:39.120908    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/747bdafd-1a41-4b0e-9547-f14fe568d3f1-workload-certs\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr\" (UID: \"747bdafd-1a41-4b0e-9547-f14fe568d3f1\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr"
Oct 30 13:08:39.123752 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:39.120977    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/747bdafd-1a41-4b0e-9547-f14fe568d3f1-credential-socket\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr\" (UID: \"747bdafd-1a41-4b0e-9547-f14fe568d3f1\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr"
Oct 30 13:08:39.123752 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:39.121082    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/747bdafd-1a41-4b0e-9547-f14fe568d3f1-istio-token\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr\" (UID: \"747bdafd-1a41-4b0e-9547-f14fe568d3f1\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr"
Oct 30 13:08:39.123752 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:39.121149    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/747bdafd-1a41-4b0e-9547-f14fe568d3f1-istio-podinfo\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr\" (UID: \"747bdafd-1a41-4b0e-9547-f14fe568d3f1\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr"
Oct 30 13:08:39.126644 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:39.125077    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/747bdafd-1a41-4b0e-9547-f14fe568d3f1-istio-podinfo\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr\" (UID: \"747bdafd-1a41-4b0e-9547-f14fe568d3f1\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr"
Oct 30 13:08:39.126644 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:39.126125    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/747bdafd-1a41-4b0e-9547-f14fe568d3f1-istiod-ca-cert\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr\" (UID: \"747bdafd-1a41-4b0e-9547-f14fe568d3f1\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr"
Oct 30 13:08:39.139637 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:39.127355    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/747bdafd-1a41-4b0e-9547-f14fe568d3f1-credential-socket\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr\" (UID: \"747bdafd-1a41-4b0e-9547-f14fe568d3f1\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr"
Oct 30 13:08:39.139637 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:39.127636    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/747bdafd-1a41-4b0e-9547-f14fe568d3f1-istio-data\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr\" (UID: \"747bdafd-1a41-4b0e-9547-f14fe568d3f1\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr"
Oct 30 13:08:39.139637 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:39.127925    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/747bdafd-1a41-4b0e-9547-f14fe568d3f1-workload-certs\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr\" (UID: \"747bdafd-1a41-4b0e-9547-f14fe568d3f1\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr"
Oct 30 13:08:39.139637 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:39.129127    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/747bdafd-1a41-4b0e-9547-f14fe568d3f1-workload-socket\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr\" (UID: \"747bdafd-1a41-4b0e-9547-f14fe568d3f1\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr"
Oct 30 13:08:39.142918 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:39.142859    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/747bdafd-1a41-4b0e-9547-f14fe568d3f1-istio-envoy\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr\" (UID: \"747bdafd-1a41-4b0e-9547-f14fe568d3f1\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr"
Oct 30 13:08:39.151754 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:39.151684    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sr82m\" (UniqueName: \"kubernetes.io/projected/747bdafd-1a41-4b0e-9547-f14fe568d3f1-kube-api-access-sr82m\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr\" (UID: \"747bdafd-1a41-4b0e-9547-f14fe568d3f1\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr"
Oct 30 13:08:39.154537 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:39.154510    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/747bdafd-1a41-4b0e-9547-f14fe568d3f1-istio-token\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr\" (UID: \"747bdafd-1a41-4b0e-9547-f14fe568d3f1\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr"
Oct 30 13:08:39.312921 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:39.312670    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr"
Oct 30 13:08:39.383088 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:39.381457    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l" event={"ID":"c6ba17d2-cc66-444e-9976-36496fe9db21","Type":"ContainerStarted","Data":"a3caecf61da2f0dafb8d3c999ac8b7beec600121f9c98611295dabfe46df175c"}
Oct 30 13:08:39.383088 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:39.381522    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l" event={"ID":"c6ba17d2-cc66-444e-9976-36496fe9db21","Type":"ContainerStarted","Data":"92bd143314802ad478f01ebd07ff80df31f8a63fc704d248d3d50b6b7f84cba5"}
Oct 30 13:08:39.396336 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:39.385914    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ceb91f-lw2hx" event={"ID":"1a0e173e-b428-45f5-b91a-095475c67cae","Type":"ContainerDied","Data":"ec3d4f79ce4fd132e14807c7b8e4335c62db26db0c21329401e16cef5c9597ec"}
Oct 30 13:08:39.396336 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:39.385984    4364 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec3d4f79ce4fd132e14807c7b8e4335c62db26db0c21329401e16cef5c9597ec"
Oct 30 13:08:39.396336 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:39.386075    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ceb91f-lw2hx"
Oct 30 13:08:39.396336 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:39.387400    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv"]
Oct 30 13:08:39.416212 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:39.400900    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb" event={"ID":"c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3","Type":"ContainerStarted","Data":"87e110aab645f150d2dab72a28f0fca46758a8173e6706024ef4a4dad396c4ab"}
Oct 30 13:08:39.416212 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:39.400959    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb" event={"ID":"c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3","Type":"ContainerStarted","Data":"f780ebddbba70b732c603522b531da0e0a8c76d2053e3ee200b3d80b9dec0c2c"}
Oct 30 13:08:39.475054 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:39.475013    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:08:39.475287 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:39.475268    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:08:39.475461 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:39.475441    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:08:39.497472 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:39.497352    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l"
Oct 30 13:08:39.498176 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:39.498152    4364 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 13:08:39.498410 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:39.498369    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l" podUID="c6ba17d2-cc66-444e-9976-36496fe9db21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 13:08:39.522975 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:39.522855    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l" podStartSLOduration=1.5228306109999998 podStartE2EDuration="1.522830611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:39.516209793 +0000 UTC m=+1018.682574442" watchObservedRunningTime="2026-10-30 13:08:39.522830611 +0000 UTC m=+1018.689195261"
Oct 30 13:08:39.652774 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:39.652718    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb" podStartSLOduration=2.652703152 podStartE2EDuration="2.652703152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:39.650953676 +0000 UTC m=+1018.817318325" watchObservedRunningTime="2026-10-30 13:08:39.652703152 +0000 UTC m=+1018.819067799"
Oct 30 13:08:39.655645 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:39.655513    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:08:39.655645 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:39.655618    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:08:39.655842 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:39.655664    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:08:39.661546 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:39.661521    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq"]
Oct 30 13:08:39.864082 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:39.861180    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr"]
Oct 30 13:08:39.874432 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:39.873770    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:08:39.874432 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:39.873856    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:08:39.874432 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:39.873895    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:08:40.282786 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:40.282742    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb"
Oct 30 13:08:40.284582 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:40.284557    4364 patch_prober.go:29] interesting pod/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 13:08:40.284813 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:40.284779    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb" podUID="c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 13:08:40.467328 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:40.464488    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv" event={"ID":"4a6cf8c7-ad37-46bc-a392-90de84af768e","Type":"ContainerStarted","Data":"333bb2cccbf13ddbeef909b155668362790223696316a0995f2e56778f2d9542"}
Oct 30 13:08:40.467328 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:40.464554    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv" event={"ID":"4a6cf8c7-ad37-46bc-a392-90de84af768e","Type":"ContainerStarted","Data":"fb5b7ebba6e8caa16fe61ecf2939128a1ef7fb73661838e6c5ce947a2f61c093"}
Oct 30 13:08:40.468374 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:40.467889    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr" event={"ID":"747bdafd-1a41-4b0e-9547-f14fe568d3f1","Type":"ContainerStarted","Data":"c6b047c72d2a3839a86bbecb3a07abaaed62689d2473a04811aa1cc928573f11"}
Oct 30 13:08:40.468374 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:40.467947    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr" event={"ID":"747bdafd-1a41-4b0e-9547-f14fe568d3f1","Type":"ContainerStarted","Data":"2c20e055bc19ee5e57034cb9ef4538b8648212618f5b3febd4b6394be57239f7"}
Oct 30 13:08:40.470675 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:40.470651    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq" event={"ID":"1d08eaa5-399d-4d90-8a47-3c060abfb552","Type":"ContainerStarted","Data":"b8c165e33e34b0ca2ad872c8938134b53338fda62cd9c2439b8af09d8650e51f"}
Oct 30 13:08:40.471225 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:40.471208    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq" event={"ID":"1d08eaa5-399d-4d90-8a47-3c060abfb552","Type":"ContainerStarted","Data":"758bf5ec1acece63a57e5263a3ab5c90c755c5fe6f70a7bbe911d373fc64e556"}
Oct 30 13:08:40.487294 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:40.486920    4364 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 13:08:40.487294 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:40.486972    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l" podUID="c6ba17d2-cc66-444e-9976-36496fe9db21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 13:08:40.564364 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:40.564283    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv" podStartSLOduration=2.564260399 podStartE2EDuration="2.564260399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:40.515021184 +0000 UTC m=+1019.681385845" watchObservedRunningTime="2026-10-30 13:08:40.564260399 +0000 UTC m=+1019.730625053"
Oct 30 13:08:40.590457 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:40.590395    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq" podStartSLOduration=2.590373414 podStartE2EDuration="2.590373414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:40.576168541 +0000 UTC m=+1019.742533194" watchObservedRunningTime="2026-10-30 13:08:40.590373414 +0000 UTC m=+1019.756738066"
Oct 30 13:08:41.063621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:41.062799    4364 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 13:08:41.063621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:41.062877    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv" podUID="4a6cf8c7-ad37-46bc-a392-90de84af768e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 13:08:41.087135 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:41.087102    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv"
Oct 30 13:08:41.116706 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:41.116678    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq"
Oct 30 13:08:41.134382 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:41.132380    4364 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 13:08:41.134382 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:41.132452    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq" podUID="1d08eaa5-399d-4d90-8a47-3c060abfb552" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 13:08:41.326488 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:41.305737    4364 patch_prober.go:29] interesting pod/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 13:08:41.326488 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:41.305808    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb" podUID="c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 13:08:41.381280 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:41.381246    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr"
Oct 30 13:08:41.402766 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:41.402708    4364 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 13:08:41.406623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:41.403034    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr" podUID="747bdafd-1a41-4b0e-9547-f14fe568d3f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 13:08:41.514623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:41.490901    4364 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 13:08:41.514623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:41.490971    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l" podUID="c6ba17d2-cc66-444e-9976-36496fe9db21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 13:08:42.037624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:42.013152    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz" podUID="a568f1ce-602d-4069-a41c-9fe1a7635c5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:42.156020 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:42.064693    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz" podUID="bc5caf69-a649-47e5-8910-9b8f331dfd4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 13:08:42.156549 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:42.064759    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4" podUID="0cbe6af7-d884-4613-90b8-192e5ffdb027" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 13:08:42.156878 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:42.099332    4364 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 13:08:42.173621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:42.157044    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv" podUID="4a6cf8c7-ad37-46bc-a392-90de84af768e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 13:08:42.174764 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:42.130867    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh" podUID="d41ac9e5-eeed-4e07-a7fa-05d3260630b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:42.189679 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:42.126935    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc" podUID="8911b790-331d-490c-876b-647e6d2fcd14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:42.217850 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:42.217800    4364 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 13:08:42.218175 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:42.218123    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq" podUID="1d08eaa5-399d-4d90-8a47-3c060abfb552" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 13:08:42.312824 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:42.312714    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6" podUID="b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 13:08:42.338168 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:42.338110    4364 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 13:08:42.338646 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:42.338507    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr" podUID="747bdafd-1a41-4b0e-9547-f14fe568d3f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 13:08:42.425638 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:42.411955    4364 patch_prober.go:29] interesting pod/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 13:08:42.425638 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:42.412023    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb" podUID="c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 13:08:42.425638 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:08:42.418893    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:08:42.524874 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:42.524819    4364 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 13:08:42.525470 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:42.525422    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l" podUID="c6ba17d2-cc66-444e-9976-36496fe9db21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 13:08:43.091619 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:43.065874    4364 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 13:08:43.091619 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:43.090681    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv" podUID="4a6cf8c7-ad37-46bc-a392-90de84af768e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 13:08:43.091619 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:43.091406    4364 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 13:08:43.091619 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:43.091444    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq" podUID="1d08eaa5-399d-4d90-8a47-3c060abfb552" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 13:08:43.161623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:43.153144    4364 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz" podUID="a568f1ce-602d-4069-a41c-9fe1a7635c5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:43.190627 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:43.175518    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz" podUID="bc5caf69-a649-47e5-8910-9b8f331dfd4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:43.190627 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:43.175747    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh" podUID="d41ac9e5-eeed-4e07-a7fa-05d3260630b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:43.190627 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:43.176049    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4" podUID="0cbe6af7-d884-4613-90b8-192e5ffdb027" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 13:08:43.190627 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:43.177411    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz" podUID="a568f1ce-602d-4069-a41c-9fe1a7635c5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 30 13:08:43.190627 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:43.178562    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz"
Oct 30 13:08:43.190627 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:43.178651    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh"
Oct 30 13:08:43.190627 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:43.178706    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4"
Oct 30 13:08:43.192036 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:43.192010    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz"
Oct 30 13:08:43.193529 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:43.192901    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc" podUID="8911b790-331d-490c-876b-647e6d2fcd14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:43.219620 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:43.194166    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc"
Oct 30 13:08:43.256519 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:43.256458    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr" podStartSLOduration=5.256438749 podStartE2EDuration="5.256438749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:40.653957643 +0000 UTC m=+1019.820322292" watchObservedRunningTime="2026-10-30 13:08:43.256438749 +0000 UTC m=+1022.422803397"
Oct 30 13:08:43.319835 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:43.301977    4364 patch_prober.go:29] interesting pod/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 13:08:43.330890 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:43.321795    4364 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 13:08:43.404743 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:43.404694    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr" podUID="747bdafd-1a41-4b0e-9547-f14fe568d3f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 13:08:43.472502 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:43.331111    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb" podUID="c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 13:08:43.472502 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:43.321479    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6" podUID="b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:43.472502 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:43.467764    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6"
Oct 30 13:08:43.555633 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:43.508824    4364 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 13:08:43.579807 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:43.579754    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l" podUID="c6ba17d2-cc66-444e-9976-36496fe9db21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 13:08:44.029801 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:44.029756    4364 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 13:08:44.030104 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:44.030061    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv" podUID="4a6cf8c7-ad37-46bc-a392-90de84af768e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 13:08:44.102645 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:44.090950    4364 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 13:08:44.102645 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:44.091006    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq" podUID="1d08eaa5-399d-4d90-8a47-3c060abfb552" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 13:08:44.210624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:44.193081    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz" podUID="bc5caf69-a649-47e5-8910-9b8f331dfd4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:44.210624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:44.193883    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz" podUID="a568f1ce-602d-4069-a41c-9fe1a7635c5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 30 13:08:44.210624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:44.194203    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4" podUID="0cbe6af7-d884-4613-90b8-192e5ffdb027" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:44.210624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:44.194454    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh" podUID="d41ac9e5-eeed-4e07-a7fa-05d3260630b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 30 13:08:44.210624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:44.194731    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc" podUID="8911b790-331d-490c-876b-647e6d2fcd14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:44.287114 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:44.287009    4364 patch_prober.go:29] interesting pod/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 13:08:44.287394 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:44.287353    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb" podUID="c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 13:08:44.342672 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:44.342269    4364 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 13:08:44.342672 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:44.342326    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr" podUID="747bdafd-1a41-4b0e-9547-f14fe568d3f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 13:08:44.483620 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:44.482721    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6" podUID="b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:44.501622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:44.487996    4364 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 13:08:44.501622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:44.488062    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l" podUID="c6ba17d2-cc66-444e-9976-36496fe9db21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 13:08:44.572679 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:44.570741    4364 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4" podUID="0cbe6af7-d884-4613-90b8-192e5ffdb027" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:45.085442 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:45.085405    4364 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 13:08:45.248042 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:45.248004    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv" podUID="4a6cf8c7-ad37-46bc-a392-90de84af768e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 13:08:45.248278 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:45.208219    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz" podUID="bc5caf69-a649-47e5-8910-9b8f331dfd4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 13:08:45.265660 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:45.208177    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc" podUID="8911b790-331d-490c-876b-647e6d2fcd14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:45.265660 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:45.179176    4364 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 13:08:45.265660 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:45.251485    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq" podUID="1d08eaa5-399d-4d90-8a47-3c060abfb552" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 13:08:45.265660 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:45.208281    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh" podUID="d41ac9e5-eeed-4e07-a7fa-05d3260630b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 30 13:08:45.265660 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:45.208262    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4" podUID="0cbe6af7-d884-4613-90b8-192e5ffdb027" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 13:08:45.265660 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:45.208241    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz" podUID="a568f1ce-602d-4069-a41c-9fe1a7635c5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 30 13:08:45.311372 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:45.309804    4364 patch_prober.go:29] interesting pod/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 13:08:45.311372 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:45.309871    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb" podUID="c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 13:08:45.321642 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:45.319976    4364 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 13:08:45.321642 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:45.320034    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr" podUID="747bdafd-1a41-4b0e-9547-f14fe568d3f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 13:08:45.491372 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:45.491319    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6" podUID="b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:45.492379 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:45.492351    4364 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 13:08:45.492564 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:45.492543    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l" podUID="c6ba17d2-cc66-444e-9976-36496fe9db21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 13:08:46.071730 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:46.071695    4364 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 13:08:46.072442 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:46.072405    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv" podUID="4a6cf8c7-ad37-46bc-a392-90de84af768e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 13:08:46.091625 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:46.090134    4364 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 13:08:46.091625 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:46.090225    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq" podUID="1d08eaa5-399d-4d90-8a47-3c060abfb552" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 13:08:46.422708 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:46.249172    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz" podUID="bc5caf69-a649-47e5-8910-9b8f331dfd4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:46.422708 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:46.267403    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz" podUID="a568f1ce-602d-4069-a41c-9fe1a7635c5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 30 13:08:46.422708 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:46.269272    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4" podUID="0cbe6af7-d884-4613-90b8-192e5ffdb027" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 13:08:46.422708 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:46.275585    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc" podUID="8911b790-331d-490c-876b-647e6d2fcd14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded"
Oct 30 13:08:46.422708 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:46.283584    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh" podUID="d41ac9e5-eeed-4e07-a7fa-05d3260630b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 30 13:08:46.422708 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:46.324634    4364 patch_prober.go:29] interesting pod/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 13:08:46.422708 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:46.324717    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb" podUID="c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 13:08:46.422708 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:46.355096    4364 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 13:08:46.422708 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:46.355154    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr" podUID="747bdafd-1a41-4b0e-9547-f14fe568d3f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 13:08:46.517308 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:46.517262    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6" podUID="b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:46.602917 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:46.569768    4364 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 13:08:46.602917 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:46.569850    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l" podUID="c6ba17d2-cc66-444e-9976-36496fe9db21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 13:08:47.078630 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:47.047838    4364 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 13:08:47.078630 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:47.074930    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv" podUID="4a6cf8c7-ad37-46bc-a392-90de84af768e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 13:08:47.148651 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:47.096230    4364 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 13:08:47.148651 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:47.096310    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq" podUID="1d08eaa5-399d-4d90-8a47-3c060abfb552" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 13:08:47.253620 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:47.253364    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz" podUID="bc5caf69-a649-47e5-8910-9b8f331dfd4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 13:08:47.373682 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:47.336080    4364 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh" podUID="d41ac9e5-eeed-4e07-a7fa-05d3260630b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:47.434633 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:47.339351    4364 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 13:08:47.459719 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:47.340027    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc" podUID="8911b790-331d-490c-876b-647e6d2fcd14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded"
Oct 30 13:08:47.463618 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:47.340058    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz" podUID="a568f1ce-602d-4069-a41c-9fe1a7635c5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 30 13:08:47.463618 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:47.340100    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4" podUID="0cbe6af7-d884-4613-90b8-192e5ffdb027" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 13:08:47.463618 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:47.340136    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh" podUID="d41ac9e5-eeed-4e07-a7fa-05d3260630b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:47.463618 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:47.459740    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr" podUID="747bdafd-1a41-4b0e-9547-f14fe568d3f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 13:08:47.463618 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:47.340432    4364 patch_prober.go:29] interesting pod/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 13:08:47.463618 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:47.461757    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb" podUID="c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 13:08:47.617629 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:47.491794    4364 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 13:08:47.617931 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:47.617889    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l" podUID="c6ba17d2-cc66-444e-9976-36496fe9db21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 13:08:47.634672 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:47.555561    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6" podUID="b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:48.066682 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:48.066262    4364 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 13:08:48.066682 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:48.066419    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv" podUID="4a6cf8c7-ad37-46bc-a392-90de84af768e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 13:08:48.089050 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:48.089012    4364 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 13:08:48.089641 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:48.089588    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq" podUID="1d08eaa5-399d-4d90-8a47-3c060abfb552" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 13:08:48.176291 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:48.176248    4364 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz" podUID="bc5caf69-a649-47e5-8910-9b8f331dfd4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": context deadline exceeded"
Oct 30 13:08:48.310646 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:48.309775    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz" podUID="bc5caf69-a649-47e5-8910-9b8f331dfd4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:48.368141 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:48.368101    4364 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 13:08:48.376677 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:48.376639    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr" podUID="747bdafd-1a41-4b0e-9547-f14fe568d3f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 13:08:48.403734 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:48.400881    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6"
Oct 30 13:08:48.519255 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:48.476621    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh" podUID="d41ac9e5-eeed-4e07-a7fa-05d3260630b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:48.545624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:48.481428    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc" podUID="8911b790-331d-490c-876b-647e6d2fcd14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:48.560987 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:48.481480    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz" podUID="a568f1ce-602d-4069-a41c-9fe1a7635c5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 30 13:08:48.590380 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:48.482286    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4" podUID="0cbe6af7-d884-4613-90b8-192e5ffdb027" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 13:08:48.618674 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:48.501559    4364 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 13:08:48.618989 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:48.618952    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l" podUID="c6ba17d2-cc66-444e-9976-36496fe9db21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 13:08:49.038682 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:49.034356    4364 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 13:08:49.038682 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:49.034417    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv" podUID="4a6cf8c7-ad37-46bc-a392-90de84af768e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 13:08:49.105635 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:49.103443    4364 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 13:08:49.105635 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:49.103506    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq" podUID="1d08eaa5-399d-4d90-8a47-3c060abfb552" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 13:08:49.160797 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:49.160730    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4"
Oct 30 13:08:49.161225 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:49.161196    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb"
Oct 30 13:08:49.163989 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:49.163965    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb"
Oct 30 13:08:49.167572 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:49.167550    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb"
Oct 30 13:08:49.168517 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:49.168499    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz"
Oct 30 13:08:49.292371 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:49.292290    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz"
Oct 30 13:08:49.317086 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:49.317038    4364 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 13:08:49.317385 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:49.317340    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr" podUID="747bdafd-1a41-4b0e-9547-f14fe568d3f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 13:08:49.446099 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:49.442075    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh"
Oct 30 13:08:49.460728 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:49.460699    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc"
Oct 30 13:08:49.503406 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:49.503021    4364 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 13:08:49.503406 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:49.503077    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l" podUID="c6ba17d2-cc66-444e-9976-36496fe9db21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 13:08:49.724842 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:49.724802    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ace0686d-ljqxb"]
Oct 30 13:08:49.750993 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:49.750961    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ace0686d-ljqxb"
Oct 30 13:08:49.772434 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:49.772403    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:08:49.772853 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:49.772833    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\""
Oct 30 13:08:49.849914 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:49.849888    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ace0686d-ljqxb"]
Oct 30 13:08:49.862782 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:49.862729    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmt65\" (UniqueName: \"kubernetes.io/projected/8e4db309-7b4f-4752-84c5-64c248011b6a-kube-api-access-qmt65\") pod \"maas-api-revoke-sub-ace0686d-ljqxb\" (UID: \"8e4db309-7b4f-4752-84c5-64c248011b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ace0686d-ljqxb"
Oct 30 13:08:49.863033 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:49.862791    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e4db309-7b4f-4752-84c5-64c248011b6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ace0686d-ljqxb\" (UID: \"8e4db309-7b4f-4752-84c5-64c248011b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ace0686d-ljqxb"
Oct 30 13:08:49.863033 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:49.862888    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e4db309-7b4f-4752-84c5-64c248011b6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ace0686d-ljqxb\" (UID: \"8e4db309-7b4f-4752-84c5-64c248011b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ace0686d-ljqxb"
Oct 30 13:08:49.964482 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:49.964425    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmt65\" (UniqueName: \"kubernetes.io/projected/8e4db309-7b4f-4752-84c5-64c248011b6a-kube-api-access-qmt65\") pod \"maas-api-revoke-sub-ace0686d-ljqxb\" (UID: \"8e4db309-7b4f-4752-84c5-64c248011b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ace0686d-ljqxb"
Oct 30 13:08:49.964722 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:49.964505    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e4db309-7b4f-4752-84c5-64c248011b6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ace0686d-ljqxb\" (UID: \"8e4db309-7b4f-4752-84c5-64c248011b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ace0686d-ljqxb"
Oct 30 13:08:49.964722 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:49.964651    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e4db309-7b4f-4752-84c5-64c248011b6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ace0686d-ljqxb\" (UID: \"8e4db309-7b4f-4752-84c5-64c248011b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ace0686d-ljqxb"
Oct 30 13:08:49.965678 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:49.965649    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e4db309-7b4f-4752-84c5-64c248011b6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ace0686d-ljqxb\" (UID: \"8e4db309-7b4f-4752-84c5-64c248011b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ace0686d-ljqxb"
Oct 30 13:08:49.984441 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:49.984334    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e4db309-7b4f-4752-84c5-64c248011b6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ace0686d-ljqxb\" (UID: \"8e4db309-7b4f-4752-84c5-64c248011b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ace0686d-ljqxb"
Oct 30 13:08:49.995460 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:49.995398    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmt65\" (UniqueName: \"kubernetes.io/projected/8e4db309-7b4f-4752-84c5-64c248011b6a-kube-api-access-qmt65\") pod \"maas-api-revoke-sub-ace0686d-ljqxb\" (UID: \"8e4db309-7b4f-4752-84c5-64c248011b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ace0686d-ljqxb"
Oct 30 13:08:50.030850 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:50.030814    4364 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 13:08:50.031147 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:50.031116    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv" podUID="4a6cf8c7-ad37-46bc-a392-90de84af768e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 13:08:50.087815 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:50.087778    4364 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 13:08:50.088021 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:50.087841    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq" podUID="1d08eaa5-399d-4d90-8a47-3c060abfb552" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 13:08:50.145410 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:50.145368    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ace0686d-ljqxb"
Oct 30 13:08:50.500164 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:50.500125    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l"
Oct 30 13:08:50.502012 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:50.501369    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l"
Oct 30 13:08:50.502977 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:50.502958    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l"
Oct 30 13:08:50.519994 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:50.519960    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ace0686d-ljqxb"]
Oct 30 13:08:50.520922 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:08:50.520865    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e4db309_7b4f_4752_84c5_64c248011b6a.slice/crio-e1cea4cc0adf892e30623687cb7cec9bf84b2d87cdcb036c69a391aea2848e73 WatchSource:0}: Error finding container e1cea4cc0adf892e30623687cb7cec9bf84b2d87cdcb036c69a391aea2848e73: Status 404 returned error can't find the container with id e1cea4cc0adf892e30623687cb7cec9bf84b2d87cdcb036c69a391aea2848e73
Oct 30 13:08:50.541690 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:50.541218    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr"
Oct 30 13:08:50.542269 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:50.541858    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr"
Oct 30 13:08:50.546211 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:50.546115    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr"
Oct 30 13:08:51.239069 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:51.239026    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ace0686d-ljqxb" event={"ID":"8e4db309-7b4f-4752-84c5-64c248011b6a","Type":"ContainerStarted","Data":"7610e96aa9718ac6ce70b64cfcc98ed375dbac57c14cdfebf0b741a9925ef2d9"}
Oct 30 13:08:51.239585 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:51.239095    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ace0686d-ljqxb" event={"ID":"8e4db309-7b4f-4752-84c5-64c248011b6a","Type":"ContainerStarted","Data":"e1cea4cc0adf892e30623687cb7cec9bf84b2d87cdcb036c69a391aea2848e73"}
Oct 30 13:08:51.286245 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:51.286115    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ace0686d-ljqxb" podStartSLOduration=2.286100008 podStartE2EDuration="2.286100008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:51.284044068 +0000 UTC m=+1030.450408717" watchObservedRunningTime="2026-10-30 13:08:51.286100008 +0000 UTC m=+1030.452464655"
Oct 30 13:08:51.344613 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:51.344566    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq"
Oct 30 13:08:51.344996 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:51.344972    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq"
Oct 30 13:08:51.345896 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:51.345875    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq"
Oct 30 13:08:51.397913 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:51.397882    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv"
Oct 30 13:08:51.398352 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:51.398315    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv"
Oct 30 13:08:51.399836 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:51.399792    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv"
Oct 30 13:08:52.249074 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:52.249039    4364 generic.go:356] "Generic (PLEG): container finished" podID="8e4db309-7b4f-4752-84c5-64c248011b6a" containerID="7610e96aa9718ac6ce70b64cfcc98ed375dbac57c14cdfebf0b741a9925ef2d9" exitCode=0
Oct 30 13:08:52.249547 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:52.249122    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ace0686d-ljqxb" event={"ID":"8e4db309-7b4f-4752-84c5-64c248011b6a","Type":"ContainerDied","Data":"7610e96aa9718ac6ce70b64cfcc98ed375dbac57c14cdfebf0b741a9925ef2d9"}
Oct 30 13:08:53.400975 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:53.400949    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v"]
Oct 30 13:08:53.405347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:53.405324    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v"
Oct 30 13:08:53.432050 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:53.432014    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v"]
Oct 30 13:08:53.432923 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:53.432899    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ace0686d-ljqxb"
Oct 30 13:08:53.513474 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:53.513388    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc6e612c-e20e-4573-a608-edbf4b7de570-workload-socket\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v\" (UID: \"cc6e612c-e20e-4573-a608-edbf4b7de570\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v"
Oct 30 13:08:53.513673 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:53.513486    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llgpx\" (UniqueName: \"kubernetes.io/projected/cc6e612c-e20e-4573-a608-edbf4b7de570-kube-api-access-llgpx\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v\" (UID: \"cc6e612c-e20e-4573-a608-edbf4b7de570\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v"
Oct 30 13:08:53.513673 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:53.513554    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc6e612c-e20e-4573-a608-edbf4b7de570-istio-envoy\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v\" (UID: \"cc6e612c-e20e-4573-a608-edbf4b7de570\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v"
Oct 30 13:08:53.513673 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:53.513642    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc6e612c-e20e-4573-a608-edbf4b7de570-istio-token\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v\" (UID: \"cc6e612c-e20e-4573-a608-edbf4b7de570\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v"
Oct 30 13:08:53.513901 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:53.513687    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc6e612c-e20e-4573-a608-edbf4b7de570-istio-podinfo\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v\" (UID: \"cc6e612c-e20e-4573-a608-edbf4b7de570\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v"
Oct 30 13:08:53.513901 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:53.513719    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc6e612c-e20e-4573-a608-edbf4b7de570-workload-certs\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v\" (UID: \"cc6e612c-e20e-4573-a608-edbf4b7de570\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v"
Oct 30 13:08:53.513901 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:53.513822    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc6e612c-e20e-4573-a608-edbf4b7de570-istiod-ca-cert\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v\" (UID: \"cc6e612c-e20e-4573-a608-edbf4b7de570\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v"
Oct 30 13:08:53.513901 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:53.513864    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc6e612c-e20e-4573-a608-edbf4b7de570-istio-data\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v\" (UID: \"cc6e612c-e20e-4573-a608-edbf4b7de570\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v"
Oct 30 13:08:53.513901 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:53.513891    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc6e612c-e20e-4573-a608-edbf4b7de570-credential-socket\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v\" (UID: \"cc6e612c-e20e-4573-a608-edbf4b7de570\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v"
Oct 30 13:08:53.614931 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:53.614894    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e4db309-7b4f-4752-84c5-64c248011b6a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e4db309-7b4f-4752-84c5-64c248011b6a-maas-api-openshift-ca\") pod \"8e4db309-7b4f-4752-84c5-64c248011b6a\" (UID: \"8e4db309-7b4f-4752-84c5-64c248011b6a\") "
Oct 30 13:08:53.615110 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:53.615028    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e4db309-7b4f-4752-84c5-64c248011b6a-kube-api-access-qmt65\" (UniqueName: \"kubernetes.io/projected/8e4db309-7b4f-4752-84c5-64c248011b6a-kube-api-access-qmt65\") pod \"8e4db309-7b4f-4752-84c5-64c248011b6a\" (UID: \"8e4db309-7b4f-4752-84c5-64c248011b6a\") "
Oct 30 13:08:53.615110 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:53.615090    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e4db309-7b4f-4752-84c5-64c248011b6a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e4db309-7b4f-4752-84c5-64c248011b6a-maas-api-xks-ca\") pod \"8e4db309-7b4f-4752-84c5-64c248011b6a\" (UID: \"8e4db309-7b4f-4752-84c5-64c248011b6a\") "
Oct 30 13:08:53.615331 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:53.615311    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e4db309-7b4f-4752-84c5-64c248011b6a-maas-api-openshift-ca" pod "8e4db309-7b4f-4752-84c5-64c248011b6a" (UID: "8e4db309-7b4f-4752-84c5-64c248011b6a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:08:53.615567 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:53.615545    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc6e612c-e20e-4573-a608-edbf4b7de570-istiod-ca-cert\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v\" (UID: \"cc6e612c-e20e-4573-a608-edbf4b7de570\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v"
Oct 30 13:08:53.615656 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:53.615594    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc6e612c-e20e-4573-a608-edbf4b7de570-istio-data\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v\" (UID: \"cc6e612c-e20e-4573-a608-edbf4b7de570\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v"
Oct 30 13:08:53.615713 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:53.615661    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc6e612c-e20e-4573-a608-edbf4b7de570-credential-socket\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v\" (UID: \"cc6e612c-e20e-4573-a608-edbf4b7de570\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v"
Oct 30 13:08:53.615812 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:53.615794    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc6e612c-e20e-4573-a608-edbf4b7de570-workload-socket\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v\" (UID: \"cc6e612c-e20e-4573-a608-edbf4b7de570\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v"
Oct 30 13:08:53.615908 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:53.615891    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llgpx\" (UniqueName: \"kubernetes.io/projected/cc6e612c-e20e-4573-a608-edbf4b7de570-kube-api-access-llgpx\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v\" (UID: \"cc6e612c-e20e-4573-a608-edbf4b7de570\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v"
Oct 30 13:08:53.615989 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:53.615973    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc6e612c-e20e-4573-a608-edbf4b7de570-istio-envoy\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v\" (UID: \"cc6e612c-e20e-4573-a608-edbf4b7de570\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v"
Oct 30 13:08:53.616049 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:53.616029    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc6e612c-e20e-4573-a608-edbf4b7de570-istio-token\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v\" (UID: \"cc6e612c-e20e-4573-a608-edbf4b7de570\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v"
Oct 30 13:08:53.616109 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:53.616079    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc6e612c-e20e-4573-a608-edbf4b7de570-istio-podinfo\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v\" (UID: \"cc6e612c-e20e-4573-a608-edbf4b7de570\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v"
Oct 30 13:08:53.616167 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:53.616113    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc6e612c-e20e-4573-a608-edbf4b7de570-workload-certs\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v\" (UID: \"cc6e612c-e20e-4573-a608-edbf4b7de570\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v"
Oct 30 13:08:53.616167 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:53.616121    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc6e612c-e20e-4573-a608-edbf4b7de570-istio-data\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v\" (UID: \"cc6e612c-e20e-4573-a608-edbf4b7de570\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v"
Oct 30 13:08:53.616269 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:53.616253    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e4db309-7b4f-4752-84c5-64c248011b6a-maas-api-openshift-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:08:53.616416 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:53.616389    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc6e612c-e20e-4573-a608-edbf4b7de570-credential-socket\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v\" (UID: \"cc6e612c-e20e-4573-a608-edbf4b7de570\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v"
Oct 30 13:08:53.616658 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:53.616632    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc6e612c-e20e-4573-a608-edbf4b7de570-istiod-ca-cert\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v\" (UID: \"cc6e612c-e20e-4573-a608-edbf4b7de570\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v"
Oct 30 13:08:53.616759 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:53.616686    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc6e612c-e20e-4573-a608-edbf4b7de570-workload-certs\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v\" (UID: \"cc6e612c-e20e-4573-a608-edbf4b7de570\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v"
Oct 30 13:08:53.616823 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:53.616809    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc6e612c-e20e-4573-a608-edbf4b7de570-workload-socket\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v\" (UID: \"cc6e612c-e20e-4573-a608-edbf4b7de570\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v"
Oct 30 13:08:53.617483 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:53.617452    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e4db309-7b4f-4752-84c5-64c248011b6a-maas-api-xks-ca" pod "8e4db309-7b4f-4752-84c5-64c248011b6a" (UID: "8e4db309-7b4f-4752-84c5-64c248011b6a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:08:53.617826 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:53.617808    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e4db309-7b4f-4752-84c5-64c248011b6a-kube-api-access-qmt65" pod "8e4db309-7b4f-4752-84c5-64c248011b6a" (UID: "8e4db309-7b4f-4752-84c5-64c248011b6a"). InnerVolumeSpecName "kube-api-access-qmt65". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:08:53.618608 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:53.618573    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc6e612c-e20e-4573-a608-edbf4b7de570-istio-envoy\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v\" (UID: \"cc6e612c-e20e-4573-a608-edbf4b7de570\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v"
Oct 30 13:08:53.618814 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:53.618799    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc6e612c-e20e-4573-a608-edbf4b7de570-istio-podinfo\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v\" (UID: \"cc6e612c-e20e-4573-a608-edbf4b7de570\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v"
Oct 30 13:08:53.644761 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:53.644735    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llgpx\" (UniqueName: \"kubernetes.io/projected/cc6e612c-e20e-4573-a608-edbf4b7de570-kube-api-access-llgpx\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v\" (UID: \"cc6e612c-e20e-4573-a608-edbf4b7de570\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v"
Oct 30 13:08:53.646793 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:53.646776    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc6e612c-e20e-4573-a608-edbf4b7de570-istio-token\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v\" (UID: \"cc6e612c-e20e-4573-a608-edbf4b7de570\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v"
Oct 30 13:08:53.717718 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:53.717661    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmt65\" (UniqueName: \"kubernetes.io/projected/8e4db309-7b4f-4752-84c5-64c248011b6a-kube-api-access-qmt65\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:08:53.717718 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:53.717695    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e4db309-7b4f-4752-84c5-64c248011b6a-maas-api-xks-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:08:53.756626 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:53.747916    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v"
Oct 30 13:08:54.394664 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:54.393177    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ace0686d-ljqxb" event={"ID":"8e4db309-7b4f-4752-84c5-64c248011b6a","Type":"ContainerDied","Data":"e1cea4cc0adf892e30623687cb7cec9bf84b2d87cdcb036c69a391aea2848e73"}
Oct 30 13:08:54.394664 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:54.393242    4364 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1cea4cc0adf892e30623687cb7cec9bf84b2d87cdcb036c69a391aea2848e73"
Oct 30 13:08:54.394664 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:54.393318    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ace0686d-ljqxb"
Oct 30 13:08:54.537622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:54.529736    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v"]
Oct 30 13:08:54.565079 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:54.562753    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:08:54.565079 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:54.562858    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:08:54.565079 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:54.562903    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:08:55.201622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:55.200693    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc" podUID="8911b790-331d-490c-876b-647e6d2fcd14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded"
Oct 30 13:08:55.264708 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:55.258391    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4" podUID="0cbe6af7-d884-4613-90b8-192e5ffdb027" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:55.264708 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:55.258873    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh" podUID="d41ac9e5-eeed-4e07-a7fa-05d3260630b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:55.307686 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:55.304165    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz" podUID="a568f1ce-602d-4069-a41c-9fe1a7635c5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:55.489064 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:55.487891    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6" podUID="b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 13:08:55.505503 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:55.504235    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v" event={"ID":"cc6e612c-e20e-4573-a608-edbf4b7de570","Type":"ContainerStarted","Data":"3eac5a3d4c71d8b90fd92739254e773b6047468c3b747ed5a18b998a5d26bf7a"}
Oct 30 13:08:55.505503 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:55.504309    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v" event={"ID":"cc6e612c-e20e-4573-a608-edbf4b7de570","Type":"ContainerStarted","Data":"34bbc56dbe8bd5b95938341aae4c077bd2bde23244088a3bc33350c38dc39e19"}
Oct 30 13:08:55.604068 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:08:55.603962    4364 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc6e612c_e20e_4573_a608_edbf4b7de570.slice/crio-3eac5a3d4c71d8b90fd92739254e773b6047468c3b747ed5a18b998a5d26bf7a.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 13:08:55.633322 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:55.627921    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v" podStartSLOduration=2.627901062 podStartE2EDuration="2.627901062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:55.584236334 +0000 UTC m=+1034.750600984" watchObservedRunningTime="2026-10-30 13:08:55.627901062 +0000 UTC m=+1034.794265710"
Oct 30 13:08:55.751440 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:55.751356    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v"
Oct 30 13:08:56.204626 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:56.200976    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc" podUID="8911b790-331d-490c-876b-647e6d2fcd14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:56.205678 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:56.204975    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc"
Oct 30 13:08:56.261747 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:56.261699    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz" podUID="bc5caf69-a649-47e5-8910-9b8f331dfd4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:56.262468 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:56.261991    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh" podUID="d41ac9e5-eeed-4e07-a7fa-05d3260630b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 30 13:08:56.263762 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:56.262019    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4" podUID="0cbe6af7-d884-4613-90b8-192e5ffdb027" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 13:08:56.264672 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:56.263930    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh"
Oct 30 13:08:56.265277 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:56.265263    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4"
Oct 30 13:08:56.314630 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:56.312729    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz" podUID="a568f1ce-602d-4069-a41c-9fe1a7635c5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:56.314630 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:56.313213    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz"
Oct 30 13:08:56.583173 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:56.510428    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6" podUID="b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 13:08:56.583173 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:56.511166    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6"
Oct 30 13:08:56.966755 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:56.966717    4364 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 13:08:56.984803 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:56.984753    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v" podUID="cc6e612c-e20e-4573-a608-edbf4b7de570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 13:08:57.204671 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:57.201796    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc" podUID="8911b790-331d-490c-876b-647e6d2fcd14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:57.297685 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:57.270762    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4" podUID="0cbe6af7-d884-4613-90b8-192e5ffdb027" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:57.297685 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:57.296086    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz" podUID="bc5caf69-a649-47e5-8910-9b8f331dfd4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:57.297685 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:57.296485    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh" podUID="d41ac9e5-eeed-4e07-a7fa-05d3260630b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:57.297685 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:57.297542    4364 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh" podUID="d41ac9e5-eeed-4e07-a7fa-05d3260630b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": context deadline exceeded"
Oct 30 13:08:57.322102 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:57.315692    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz" podUID="a568f1ce-602d-4069-a41c-9fe1a7635c5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 30 13:08:57.550410 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:57.550295    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6" podUID="b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:57.600624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:57.554727    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz"
Oct 30 13:08:57.794450 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:57.794410    4364 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 13:08:57.794751 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:57.794719    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v" podUID="cc6e612c-e20e-4573-a608-edbf4b7de570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 13:08:58.345671 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:58.248900    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc" podUID="8911b790-331d-490c-876b-647e6d2fcd14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:58.345671 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:58.292218    4364 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz" podUID="bc5caf69-a649-47e5-8910-9b8f331dfd4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": context deadline exceeded"
Oct 30 13:08:58.384713 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:58.356844    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz" podUID="a568f1ce-602d-4069-a41c-9fe1a7635c5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:58.407625 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:58.386976    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz" podUID="bc5caf69-a649-47e5-8910-9b8f331dfd4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:58.441508 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:58.441445    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh" podUID="d41ac9e5-eeed-4e07-a7fa-05d3260630b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:58.511939 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:58.511883    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4" podUID="0cbe6af7-d884-4613-90b8-192e5ffdb027" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 13:08:58.552893 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:58.550875    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6" podUID="b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:58.858396 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:58.855417    4364 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 13:08:58.858396 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:58.855497    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v" podUID="cc6e612c-e20e-4573-a608-edbf4b7de570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 13:08:59.249783 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:59.249731    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc" podUID="8911b790-331d-490c-876b-647e6d2fcd14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded"
Oct 30 13:08:59.359763 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:59.359712    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz" podUID="a568f1ce-602d-4069-a41c-9fe1a7635c5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:59.415620 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:59.414768    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz" podUID="bc5caf69-a649-47e5-8910-9b8f331dfd4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:59.480814 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:59.480754    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh" podUID="d41ac9e5-eeed-4e07-a7fa-05d3260630b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:59.522657 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:59.522536    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4" podUID="0cbe6af7-d884-4613-90b8-192e5ffdb027" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:08:59.553761 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:59.553712    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6" podUID="b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 13:08:59.766782 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:59.766737    4364 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 13:08:59.767075 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:59.767034    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v" podUID="cc6e612c-e20e-4573-a608-edbf4b7de570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 13:08:59.901627 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:08:59.896743    4364 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6" podUID="b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:09:00.410658 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:00.403788    4364 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc" podUID="8911b790-331d-490c-876b-647e6d2fcd14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": context deadline exceeded"
Oct 30 13:09:00.454623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:00.415196    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz" podUID="a568f1ce-602d-4069-a41c-9fe1a7635c5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:09:00.573382 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:00.415258    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz" podUID="bc5caf69-a649-47e5-8910-9b8f331dfd4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 13:09:00.573920 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:00.416577    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc" podUID="8911b790-331d-490c-876b-647e6d2fcd14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:09:00.574307 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:00.485094    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh" podUID="d41ac9e5-eeed-4e07-a7fa-05d3260630b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 30 13:09:00.574704 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:00.534684    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4" podUID="0cbe6af7-d884-4613-90b8-192e5ffdb027" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:09:00.578623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:00.561722    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6" podUID="b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:09:00.788249 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:00.788156    4364 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 13:09:00.788510 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:00.788477    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v" podUID="cc6e612c-e20e-4573-a608-edbf4b7de570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 13:09:01.577650 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:01.577586    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz" podUID="a568f1ce-602d-4069-a41c-9fe1a7635c5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 30 13:09:01.590057 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:01.590017    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6" podUID="b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 13:09:01.590591 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:01.590561    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz" podUID="bc5caf69-a649-47e5-8910-9b8f331dfd4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 13:09:01.591024 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:01.591000    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc" podUID="8911b790-331d-490c-876b-647e6d2fcd14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded"
Oct 30 13:09:01.591436 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:01.591411    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh" podUID="d41ac9e5-eeed-4e07-a7fa-05d3260630b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 30 13:09:01.591847 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:01.591824    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4" podUID="0cbe6af7-d884-4613-90b8-192e5ffdb027" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 13:09:01.874621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:01.871081    4364 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 13:09:01.874621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:01.871146    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v" podUID="cc6e612c-e20e-4573-a608-edbf4b7de570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 13:09:02.140687 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:02.138945    4364 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:09:02.140687 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:02.139028    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww" podUID="3f1549da-ef30-4001-9eec-35b1040de3b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:09:02.169182 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:02.169148    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz"
Oct 30 13:09:02.244617 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:02.244175    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6"
Oct 30 13:09:02.353180 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:02.353154    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc"
Oct 30 13:09:02.353388 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:02.353377    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz"
Oct 30 13:09:02.357196 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:02.357157    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh"
Oct 30 13:09:02.362711 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:02.362686    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4"
Oct 30 13:09:02.749101 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:02.749066    4364 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 13:09:02.749625 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:02.749127    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v" podUID="cc6e612c-e20e-4573-a608-edbf4b7de570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 13:09:03.748395 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:03.748361    4364 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 13:09:03.748593 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:03.748424    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v" podUID="cc6e612c-e20e-4573-a608-edbf4b7de570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 13:09:04.748996 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:04.748959    4364 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 13:09:04.751284 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:04.749013    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v" podUID="cc6e612c-e20e-4573-a608-edbf4b7de570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 13:09:05.748748 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:05.748710    4364 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 13:09:05.748932 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:05.748774    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v" podUID="cc6e612c-e20e-4573-a608-edbf4b7de570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 13:09:06.748525 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:06.748491    4364 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 13:09:06.748905 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:06.748557    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v" podUID="cc6e612c-e20e-4573-a608-edbf4b7de570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 13:09:07.753065 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:07.753034    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v"
Oct 30 13:09:07.753458 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:07.753433    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v"
Oct 30 13:09:07.754047 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:07.754028    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v"
Oct 30 13:09:11.505556 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:09:11.505518    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:09:14.228863 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:14.228824    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aff624f9-44trc"]
Oct 30 13:09:14.229442 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:14.229428    4364 cpu_manager.go:414] "containerMap: removing container" podUID="8e4db309-7b4f-4752-84c5-64c248011b6a" containerName="revoke-keys"
Oct 30 13:09:14.229442 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:14.229442    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e4db309-7b4f-4752-84c5-64c248011b6a" containerName="revoke-keys"
Oct 30 13:09:14.229579 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:14.229570    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e4db309-7b4f-4752-84c5-64c248011b6a" containerName="revoke-keys"
Oct 30 13:09:14.234399 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:14.234384    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aff624f9-44trc"
Oct 30 13:09:14.253167 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:14.253140    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aff624f9-44trc"]
Oct 30 13:09:14.260062 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:14.260035    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:09:14.260278 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:14.260261    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\""
Oct 30 13:09:14.306270 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:14.306236    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2v8t\" (UniqueName: \"kubernetes.io/projected/8f075f88-c8cd-4215-be7b-be7f2da241be-kube-api-access-z2v8t\") pod \"maas-api-revoke-sub-aff624f9-44trc\" (UID: \"8f075f88-c8cd-4215-be7b-be7f2da241be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aff624f9-44trc"
Oct 30 13:09:14.306414 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:14.306303    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f075f88-c8cd-4215-be7b-be7f2da241be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aff624f9-44trc\" (UID: \"8f075f88-c8cd-4215-be7b-be7f2da241be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aff624f9-44trc"
Oct 30 13:09:14.306549 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:14.306527    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f075f88-c8cd-4215-be7b-be7f2da241be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aff624f9-44trc\" (UID: \"8f075f88-c8cd-4215-be7b-be7f2da241be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aff624f9-44trc"
Oct 30 13:09:14.408267 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:14.408223    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2v8t\" (UniqueName: \"kubernetes.io/projected/8f075f88-c8cd-4215-be7b-be7f2da241be-kube-api-access-z2v8t\") pod \"maas-api-revoke-sub-aff624f9-44trc\" (UID: \"8f075f88-c8cd-4215-be7b-be7f2da241be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aff624f9-44trc"
Oct 30 13:09:14.408443 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:14.408277    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f075f88-c8cd-4215-be7b-be7f2da241be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aff624f9-44trc\" (UID: \"8f075f88-c8cd-4215-be7b-be7f2da241be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aff624f9-44trc"
Oct 30 13:09:14.408443 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:14.408415    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f075f88-c8cd-4215-be7b-be7f2da241be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aff624f9-44trc\" (UID: \"8f075f88-c8cd-4215-be7b-be7f2da241be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aff624f9-44trc"
Oct 30 13:09:14.409019 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:14.408998    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f075f88-c8cd-4215-be7b-be7f2da241be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aff624f9-44trc\" (UID: \"8f075f88-c8cd-4215-be7b-be7f2da241be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aff624f9-44trc"
Oct 30 13:09:14.410780 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:14.410758    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f075f88-c8cd-4215-be7b-be7f2da241be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aff624f9-44trc\" (UID: \"8f075f88-c8cd-4215-be7b-be7f2da241be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aff624f9-44trc"
Oct 30 13:09:14.433897 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:14.433877    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2v8t\" (UniqueName: \"kubernetes.io/projected/8f075f88-c8cd-4215-be7b-be7f2da241be-kube-api-access-z2v8t\") pod \"maas-api-revoke-sub-aff624f9-44trc\" (UID: \"8f075f88-c8cd-4215-be7b-be7f2da241be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aff624f9-44trc"
Oct 30 13:09:14.549008 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:14.548942    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aff624f9-44trc"
Oct 30 13:09:14.855220 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:09:14.855191    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f075f88_c8cd_4215_be7b_be7f2da241be.slice/crio-1a928a82c9bc6fd0363c139b47ec015bb40fc78ece3b837e4d740726bbf2c653 WatchSource:0}: Error finding container 1a928a82c9bc6fd0363c139b47ec015bb40fc78ece3b837e4d740726bbf2c653: Status 404 returned error can't find the container with id 1a928a82c9bc6fd0363c139b47ec015bb40fc78ece3b837e4d740726bbf2c653
Oct 30 13:09:14.855666 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:14.855645    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aff624f9-44trc"]
Oct 30 13:09:15.722495 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:15.722455    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aff624f9-44trc" event={"ID":"8f075f88-c8cd-4215-be7b-be7f2da241be","Type":"ContainerStarted","Data":"9dd06988e349d811dd5a71a037240fbdb54d3abcc357d962888f58f162e973a7"}
Oct 30 13:09:15.722966 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:15.722517    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aff624f9-44trc" event={"ID":"8f075f88-c8cd-4215-be7b-be7f2da241be","Type":"ContainerStarted","Data":"1a928a82c9bc6fd0363c139b47ec015bb40fc78ece3b837e4d740726bbf2c653"}
Oct 30 13:09:15.771077 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:15.771028    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aff624f9-44trc" podStartSLOduration=1.771011444 podStartE2EDuration="1.771011444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:15.768928099 +0000 UTC m=+1054.935292748" watchObservedRunningTime="2026-10-30 13:09:15.771011444 +0000 UTC m=+1054.937376121"
Oct 30 13:09:16.732162 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:16.732122    4364 generic.go:356] "Generic (PLEG): container finished" podID="8f075f88-c8cd-4215-be7b-be7f2da241be" containerID="9dd06988e349d811dd5a71a037240fbdb54d3abcc357d962888f58f162e973a7" exitCode=0
Oct 30 13:09:16.732708 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:16.732209    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aff624f9-44trc" event={"ID":"8f075f88-c8cd-4215-be7b-be7f2da241be","Type":"ContainerDied","Data":"9dd06988e349d811dd5a71a037240fbdb54d3abcc357d962888f58f162e973a7"}
Oct 30 13:09:17.981232 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:17.981207    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aff624f9-44trc"
Oct 30 13:09:18.095562 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:18.095487    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f075f88-c8cd-4215-be7b-be7f2da241be-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f075f88-c8cd-4215-be7b-be7f2da241be-maas-api-xks-ca\") pod \"8f075f88-c8cd-4215-be7b-be7f2da241be\" (UID: \"8f075f88-c8cd-4215-be7b-be7f2da241be\") "
Oct 30 13:09:18.095716 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:18.095594    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f075f88-c8cd-4215-be7b-be7f2da241be-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f075f88-c8cd-4215-be7b-be7f2da241be-maas-api-openshift-ca\") pod \"8f075f88-c8cd-4215-be7b-be7f2da241be\" (UID: \"8f075f88-c8cd-4215-be7b-be7f2da241be\") "
Oct 30 13:09:18.095716 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:18.095652    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f075f88-c8cd-4215-be7b-be7f2da241be-kube-api-access-z2v8t\" (UniqueName: \"kubernetes.io/projected/8f075f88-c8cd-4215-be7b-be7f2da241be-kube-api-access-z2v8t\") pod \"8f075f88-c8cd-4215-be7b-be7f2da241be\" (UID: \"8f075f88-c8cd-4215-be7b-be7f2da241be\") "
Oct 30 13:09:18.096007 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:18.095975    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f075f88-c8cd-4215-be7b-be7f2da241be-maas-api-openshift-ca" pod "8f075f88-c8cd-4215-be7b-be7f2da241be" (UID: "8f075f88-c8cd-4215-be7b-be7f2da241be"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:09:18.096187 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:18.096173    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f075f88-c8cd-4215-be7b-be7f2da241be-maas-api-openshift-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:09:18.097637 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:18.097609    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f075f88-c8cd-4215-be7b-be7f2da241be-maas-api-xks-ca" pod "8f075f88-c8cd-4215-be7b-be7f2da241be" (UID: "8f075f88-c8cd-4215-be7b-be7f2da241be"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:09:18.097717 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:18.097695    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f075f88-c8cd-4215-be7b-be7f2da241be-kube-api-access-z2v8t" pod "8f075f88-c8cd-4215-be7b-be7f2da241be" (UID: "8f075f88-c8cd-4215-be7b-be7f2da241be"). InnerVolumeSpecName "kube-api-access-z2v8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:09:18.197177 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:18.197153    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2v8t\" (UniqueName: \"kubernetes.io/projected/8f075f88-c8cd-4215-be7b-be7f2da241be-kube-api-access-z2v8t\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:09:18.197177 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:18.197177    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f075f88-c8cd-4215-be7b-be7f2da241be-maas-api-xks-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:09:18.748895 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:18.748863    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aff624f9-44trc"
Oct 30 13:09:18.749060 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:18.748886    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aff624f9-44trc" event={"ID":"8f075f88-c8cd-4215-be7b-be7f2da241be","Type":"ContainerDied","Data":"1a928a82c9bc6fd0363c139b47ec015bb40fc78ece3b837e4d740726bbf2c653"}
Oct 30 13:09:18.749060 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:18.748945    4364 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a928a82c9bc6fd0363c139b47ec015bb40fc78ece3b837e4d740726bbf2c653"
Oct 30 13:09:18.953904 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:18.953877    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr"]
Oct 30 13:09:18.954486 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:18.954472    4364 cpu_manager.go:414] "containerMap: removing container" podUID="8f075f88-c8cd-4215-be7b-be7f2da241be" containerName="revoke-keys"
Oct 30 13:09:18.954532 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:18.954487    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f075f88-c8cd-4215-be7b-be7f2da241be" containerName="revoke-keys"
Oct 30 13:09:18.954631 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:18.954621    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f075f88-c8cd-4215-be7b-be7f2da241be" containerName="revoke-keys"
Oct 30 13:09:18.958290 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:18.958269    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr"
Oct 30 13:09:18.975422 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:18.975401    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-cb58b8\"/\"default-dockercfg-z4nxf\""
Oct 30 13:09:18.988922 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:18.988896    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-cb58b8\"/\"distinct-2-w2-cb58b8-kserve-self-signed-certs\""
Oct 30 13:09:18.989250 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:18.988942    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-cb58b8\"/\"openshift-service-ca.crt\""
Oct 30 13:09:18.989250 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:18.989040    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-cb58b8\"/\"kube-root-ca.crt\""
Oct 30 13:09:19.001556 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:19.001346    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr"]
Oct 30 13:09:19.064758 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:19.064736    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:09:19.107556 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:19.107528    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/446c12df-de12-4c7c-8c71-dfed93cc33f1-home\") pod \"distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr\" (UID: \"446c12df-de12-4c7c-8c71-dfed93cc33f1\") " pod="e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr"
Oct 30 13:09:19.107686 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:19.107572    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/446c12df-de12-4c7c-8c71-dfed93cc33f1-tmp-dir\") pod \"distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr\" (UID: \"446c12df-de12-4c7c-8c71-dfed93cc33f1\") " pod="e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr"
Oct 30 13:09:19.107686 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:19.107632    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5qtj\" (UniqueName: \"kubernetes.io/projected/446c12df-de12-4c7c-8c71-dfed93cc33f1-kube-api-access-b5qtj\") pod \"distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr\" (UID: \"446c12df-de12-4c7c-8c71-dfed93cc33f1\") " pod="e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr"
Oct 30 13:09:19.107799 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:19.107719    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/446c12df-de12-4c7c-8c71-dfed93cc33f1-dshm\") pod \"distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr\" (UID: \"446c12df-de12-4c7c-8c71-dfed93cc33f1\") " pod="e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr"
Oct 30 13:09:19.107860 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:19.107784    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/446c12df-de12-4c7c-8c71-dfed93cc33f1-tls-certs\") pod \"distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr\" (UID: \"446c12df-de12-4c7c-8c71-dfed93cc33f1\") " pod="e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr"
Oct 30 13:09:19.107860 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:19.107847    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/446c12df-de12-4c7c-8c71-dfed93cc33f1-model-cache\") pod \"distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr\" (UID: \"446c12df-de12-4c7c-8c71-dfed93cc33f1\") " pod="e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr"
Oct 30 13:09:19.210621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:19.210569    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/446c12df-de12-4c7c-8c71-dfed93cc33f1-home\") pod \"distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr\" (UID: \"446c12df-de12-4c7c-8c71-dfed93cc33f1\") " pod="e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr"
Oct 30 13:09:19.214503 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:19.214480    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/446c12df-de12-4c7c-8c71-dfed93cc33f1-tmp-dir\") pod \"distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr\" (UID: \"446c12df-de12-4c7c-8c71-dfed93cc33f1\") " pod="e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr"
Oct 30 13:09:19.214729 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:19.214708    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5qtj\" (UniqueName: \"kubernetes.io/projected/446c12df-de12-4c7c-8c71-dfed93cc33f1-kube-api-access-b5qtj\") pod \"distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr\" (UID: \"446c12df-de12-4c7c-8c71-dfed93cc33f1\") " pod="e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr"
Oct 30 13:09:19.214886 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:19.214871    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/446c12df-de12-4c7c-8c71-dfed93cc33f1-dshm\") pod \"distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr\" (UID: \"446c12df-de12-4c7c-8c71-dfed93cc33f1\") " pod="e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr"
Oct 30 13:09:19.214984 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:19.214946    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/446c12df-de12-4c7c-8c71-dfed93cc33f1-tmp-dir\") pod \"distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr\" (UID: \"446c12df-de12-4c7c-8c71-dfed93cc33f1\") " pod="e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr"
Oct 30 13:09:19.215774 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:19.214977    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/446c12df-de12-4c7c-8c71-dfed93cc33f1-tls-certs\") pod \"distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr\" (UID: \"446c12df-de12-4c7c-8c71-dfed93cc33f1\") " pod="e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr"
Oct 30 13:09:19.215991 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:19.215964    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/446c12df-de12-4c7c-8c71-dfed93cc33f1-model-cache\") pod \"distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr\" (UID: \"446c12df-de12-4c7c-8c71-dfed93cc33f1\") " pod="e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr"
Oct 30 13:09:19.216515 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:19.214362    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/446c12df-de12-4c7c-8c71-dfed93cc33f1-home\") pod \"distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr\" (UID: \"446c12df-de12-4c7c-8c71-dfed93cc33f1\") " pod="e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr"
Oct 30 13:09:19.222618 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:19.216407    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/446c12df-de12-4c7c-8c71-dfed93cc33f1-model-cache\") pod \"distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr\" (UID: \"446c12df-de12-4c7c-8c71-dfed93cc33f1\") " pod="e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr"
Oct 30 13:09:19.222766 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:19.222288    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/446c12df-de12-4c7c-8c71-dfed93cc33f1-tls-certs\") pod \"distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr\" (UID: \"446c12df-de12-4c7c-8c71-dfed93cc33f1\") " pod="e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr"
Oct 30 13:09:19.251667 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:19.251346    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/446c12df-de12-4c7c-8c71-dfed93cc33f1-dshm\") pod \"distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr\" (UID: \"446c12df-de12-4c7c-8c71-dfed93cc33f1\") " pod="e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr"
Oct 30 13:09:19.273023 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:19.272992    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5qtj\" (UniqueName: \"kubernetes.io/projected/446c12df-de12-4c7c-8c71-dfed93cc33f1-kube-api-access-b5qtj\") pod \"distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr\" (UID: \"446c12df-de12-4c7c-8c71-dfed93cc33f1\") " pod="e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr"
Oct 30 13:09:19.276953 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:19.275634    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr"
Oct 30 13:09:19.693223 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:09:19.693190    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod446c12df_de12_4c7c_8c71_dfed93cc33f1.slice/crio-a64639c9ca43dfafe0f4c3d8421170673c0e87851640016f1d1e2ac1eb9468a9 WatchSource:0}: Error finding container a64639c9ca43dfafe0f4c3d8421170673c0e87851640016f1d1e2ac1eb9468a9: Status 404 returned error can't find the container with id a64639c9ca43dfafe0f4c3d8421170673c0e87851640016f1d1e2ac1eb9468a9
Oct 30 13:09:19.693890 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:19.693870    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr"]
Oct 30 13:09:19.757196 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:19.757166    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr" event={"ID":"446c12df-de12-4c7c-8c71-dfed93cc33f1","Type":"ContainerStarted","Data":"a64639c9ca43dfafe0f4c3d8421170673c0e87851640016f1d1e2ac1eb9468a9"}
Oct 30 13:09:20.837623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:20.837548    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr" event={"ID":"446c12df-de12-4c7c-8c71-dfed93cc33f1","Type":"ContainerStarted","Data":"50462199db7e93b6faff758547a9ab3723af306bc5074114c36b245f146f0f08"}
Oct 30 13:09:20.915623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:20.909702    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr" podStartSLOduration=2.909682504 podStartE2EDuration="2.909682504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:20.908056046 +0000 UTC m=+1060.074420701" watchObservedRunningTime="2026-10-30 13:09:20.909682504 +0000 UTC m=+1060.076047156"
Oct 30 13:09:21.280622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:21.279678    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr"
Oct 30 13:09:22.181622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:22.180721    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz" podUID="bc5caf69-a649-47e5-8910-9b8f331dfd4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:09:22.250630 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:22.249743    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc" podUID="8911b790-331d-490c-876b-647e6d2fcd14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:09:22.284622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:22.283672    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4" podUID="0cbe6af7-d884-4613-90b8-192e5ffdb027" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:09:22.320620 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:22.319672    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz" podUID="a568f1ce-602d-4069-a41c-9fe1a7635c5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:09:22.477786 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:22.477681    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh" podUID="d41ac9e5-eeed-4e07-a7fa-05d3260630b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:09:22.589724 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:22.589679    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6" podUID="b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:09:23.094622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:23.093564    4364 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz" podUID="a568f1ce-602d-4069-a41c-9fe1a7635c5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:09:23.338250 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:23.279992    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz" podUID="bc5caf69-a649-47e5-8910-9b8f331dfd4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 13:09:23.338250 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:23.309767    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4" podUID="0cbe6af7-d884-4613-90b8-192e5ffdb027" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:09:23.410622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:23.410515    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc" podUID="8911b790-331d-490c-876b-647e6d2fcd14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded"
Oct 30 13:09:23.427226 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:23.427196    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz"
Oct 30 13:09:23.588490 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:23.532367    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc"
Oct 30 13:09:23.588490 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:23.532411    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb"]
Oct 30 13:09:23.588490 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:23.460829    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz" podUID="a568f1ce-602d-4069-a41c-9fe1a7635c5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 30 13:09:23.588490 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:23.532240    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh" podUID="d41ac9e5-eeed-4e07-a7fa-05d3260630b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 30 13:09:23.621620 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:23.591022    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6" podUID="b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:09:24.117661 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:24.110007    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4"
Oct 30 13:09:24.117661 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:24.110236    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz"
Oct 30 13:09:24.117661 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:24.110389    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh"
Oct 30 13:09:24.117661 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:24.110463    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6"
Oct 30 13:09:24.177109 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:24.141480    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb"
Oct 30 13:09:24.288985 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:24.288691    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz" podUID="bc5caf69-a649-47e5-8910-9b8f331dfd4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:09:24.414674 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:24.413679    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc" podUID="8911b790-331d-490c-876b-647e6d2fcd14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded"
Oct 30 13:09:24.485625 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:24.476415    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4" podUID="0cbe6af7-d884-4613-90b8-192e5ffdb027" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:09:24.563622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:24.535679    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz" podUID="a568f1ce-602d-4069-a41c-9fe1a7635c5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 30 13:09:24.575621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:24.569133    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh" podUID="d41ac9e5-eeed-4e07-a7fa-05d3260630b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:09:24.667672 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:24.573665    4364 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4" podUID="0cbe6af7-d884-4613-90b8-192e5ffdb027" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": context deadline exceeded"
Oct 30 13:09:24.667672 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:24.612656    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb"]
Oct 30 13:09:24.721199 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:24.700808    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6" podUID="b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:09:24.806623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:24.734049    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59c001b5-b0f7-4898-ba90-aa8a0b96a6e7-istio-token\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb\" (UID: \"59c001b5-b0f7-4898-ba90-aa8a0b96a6e7\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb"
Oct 30 13:09:24.806623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:24.734160    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59c001b5-b0f7-4898-ba90-aa8a0b96a6e7-istio-podinfo\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb\" (UID: \"59c001b5-b0f7-4898-ba90-aa8a0b96a6e7\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb"
Oct 30 13:09:24.806623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:24.734194    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59c001b5-b0f7-4898-ba90-aa8a0b96a6e7-istio-envoy\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb\" (UID: \"59c001b5-b0f7-4898-ba90-aa8a0b96a6e7\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb"
Oct 30 13:09:24.806623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:24.734262    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59c001b5-b0f7-4898-ba90-aa8a0b96a6e7-istiod-ca-cert\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb\" (UID: \"59c001b5-b0f7-4898-ba90-aa8a0b96a6e7\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb"
Oct 30 13:09:24.806623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:24.734487    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59c001b5-b0f7-4898-ba90-aa8a0b96a6e7-workload-socket\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb\" (UID: \"59c001b5-b0f7-4898-ba90-aa8a0b96a6e7\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb"
Oct 30 13:09:24.889624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:24.811438    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdn7s\" (UniqueName: \"kubernetes.io/projected/59c001b5-b0f7-4898-ba90-aa8a0b96a6e7-kube-api-access-mdn7s\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb\" (UID: \"59c001b5-b0f7-4898-ba90-aa8a0b96a6e7\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb"
Oct 30 13:09:24.889624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:24.811529    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59c001b5-b0f7-4898-ba90-aa8a0b96a6e7-istio-data\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb\" (UID: \"59c001b5-b0f7-4898-ba90-aa8a0b96a6e7\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb"
Oct 30 13:09:24.889624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:24.811568    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59c001b5-b0f7-4898-ba90-aa8a0b96a6e7-workload-certs\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb\" (UID: \"59c001b5-b0f7-4898-ba90-aa8a0b96a6e7\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb"
Oct 30 13:09:24.889624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:24.811678    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59c001b5-b0f7-4898-ba90-aa8a0b96a6e7-credential-socket\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb\" (UID: \"59c001b5-b0f7-4898-ba90-aa8a0b96a6e7\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb"
Oct 30 13:09:24.897072 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:24.894306    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc"
Oct 30 13:09:24.920693 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:24.920130    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdn7s\" (UniqueName: \"kubernetes.io/projected/59c001b5-b0f7-4898-ba90-aa8a0b96a6e7-kube-api-access-mdn7s\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb\" (UID: \"59c001b5-b0f7-4898-ba90-aa8a0b96a6e7\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb"
Oct 30 13:09:24.920693 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:24.920210    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59c001b5-b0f7-4898-ba90-aa8a0b96a6e7-istio-data\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb\" (UID: \"59c001b5-b0f7-4898-ba90-aa8a0b96a6e7\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb"
Oct 30 13:09:24.920693 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:24.920248    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59c001b5-b0f7-4898-ba90-aa8a0b96a6e7-workload-certs\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb\" (UID: \"59c001b5-b0f7-4898-ba90-aa8a0b96a6e7\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb"
Oct 30 13:09:24.920693 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:24.920318    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59c001b5-b0f7-4898-ba90-aa8a0b96a6e7-credential-socket\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb\" (UID: \"59c001b5-b0f7-4898-ba90-aa8a0b96a6e7\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb"
Oct 30 13:09:24.920693 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:24.920383    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59c001b5-b0f7-4898-ba90-aa8a0b96a6e7-istio-token\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb\" (UID: \"59c001b5-b0f7-4898-ba90-aa8a0b96a6e7\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb"
Oct 30 13:09:24.920693 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:24.920503    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59c001b5-b0f7-4898-ba90-aa8a0b96a6e7-istio-podinfo\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb\" (UID: \"59c001b5-b0f7-4898-ba90-aa8a0b96a6e7\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb"
Oct 30 13:09:24.920693 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:24.920534    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59c001b5-b0f7-4898-ba90-aa8a0b96a6e7-istio-envoy\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb\" (UID: \"59c001b5-b0f7-4898-ba90-aa8a0b96a6e7\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb"
Oct 30 13:09:24.922035 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:24.922008    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59c001b5-b0f7-4898-ba90-aa8a0b96a6e7-istio-data\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb\" (UID: \"59c001b5-b0f7-4898-ba90-aa8a0b96a6e7\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb"
Oct 30 13:09:24.922256 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:24.922235    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59c001b5-b0f7-4898-ba90-aa8a0b96a6e7-workload-certs\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb\" (UID: \"59c001b5-b0f7-4898-ba90-aa8a0b96a6e7\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb"
Oct 30 13:09:24.922542 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:24.922471    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59c001b5-b0f7-4898-ba90-aa8a0b96a6e7-credential-socket\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb\" (UID: \"59c001b5-b0f7-4898-ba90-aa8a0b96a6e7\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb"
Oct 30 13:09:24.922982 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:24.922960    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59c001b5-b0f7-4898-ba90-aa8a0b96a6e7-istiod-ca-cert\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb\" (UID: \"59c001b5-b0f7-4898-ba90-aa8a0b96a6e7\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb"
Oct 30 13:09:24.923103 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:24.923083    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59c001b5-b0f7-4898-ba90-aa8a0b96a6e7-workload-socket\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb\" (UID: \"59c001b5-b0f7-4898-ba90-aa8a0b96a6e7\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb"
Oct 30 13:09:24.923461 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:24.923444    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59c001b5-b0f7-4898-ba90-aa8a0b96a6e7-workload-socket\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb\" (UID: \"59c001b5-b0f7-4898-ba90-aa8a0b96a6e7\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb"
Oct 30 13:09:24.924797 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:24.924097    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59c001b5-b0f7-4898-ba90-aa8a0b96a6e7-istiod-ca-cert\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb\" (UID: \"59c001b5-b0f7-4898-ba90-aa8a0b96a6e7\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb"
Oct 30 13:09:24.936639 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:24.936614    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59c001b5-b0f7-4898-ba90-aa8a0b96a6e7-istio-podinfo\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb\" (UID: \"59c001b5-b0f7-4898-ba90-aa8a0b96a6e7\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb"
Oct 30 13:09:24.980828 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:24.980800    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz"
Oct 30 13:09:25.006100 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.006048    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdn7s\" (UniqueName: \"kubernetes.io/projected/59c001b5-b0f7-4898-ba90-aa8a0b96a6e7-kube-api-access-mdn7s\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb\" (UID: \"59c001b5-b0f7-4898-ba90-aa8a0b96a6e7\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb"
Oct 30 13:09:25.062171 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.062128    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59c001b5-b0f7-4898-ba90-aa8a0b96a6e7-istio-envoy\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb\" (UID: \"59c001b5-b0f7-4898-ba90-aa8a0b96a6e7\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb"
Oct 30 13:09:25.087925 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.087894    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59c001b5-b0f7-4898-ba90-aa8a0b96a6e7-istio-token\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb\" (UID: \"59c001b5-b0f7-4898-ba90-aa8a0b96a6e7\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb"
Oct 30 13:09:25.088388 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.088364    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45"]
Oct 30 13:09:25.094755 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.094733    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4"
Oct 30 13:09:25.094976 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.094958    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz"
Oct 30 13:09:25.095112 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.095098    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45"]
Oct 30 13:09:25.095245 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.095232    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4"]
Oct 30 13:09:25.097442 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.097422    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45"
Oct 30 13:09:25.102163 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.102136    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh"
Oct 30 13:09:25.102301 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.102285    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4"]
Oct 30 13:09:25.102494 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.102478    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4"
Oct 30 13:09:25.131291 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.126646    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6"
Oct 30 13:09:25.157571 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.157542    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b96befc8-d310-489e-a714-2a06ccd4455c-credential-socket\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4\" (UID: \"b96befc8-d310-489e-a714-2a06ccd4455c\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4"
Oct 30 13:09:25.157769 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.157575    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b835314b-3c09-4143-966b-928a86200e6c-istio-data\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45\" (UID: \"b835314b-3c09-4143-966b-928a86200e6c\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45"
Oct 30 13:09:25.157769 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.157631    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b96befc8-d310-489e-a714-2a06ccd4455c-istiod-ca-cert\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4\" (UID: \"b96befc8-d310-489e-a714-2a06ccd4455c\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4"
Oct 30 13:09:25.158078 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.158055    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b835314b-3c09-4143-966b-928a86200e6c-istio-envoy\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45\" (UID: \"b835314b-3c09-4143-966b-928a86200e6c\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45"
Oct 30 13:09:25.158144 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.158096    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b96befc8-d310-489e-a714-2a06ccd4455c-istio-token\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4\" (UID: \"b96befc8-d310-489e-a714-2a06ccd4455c\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4"
Oct 30 13:09:25.158218 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.158183    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b835314b-3c09-4143-966b-928a86200e6c-workload-certs\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45\" (UID: \"b835314b-3c09-4143-966b-928a86200e6c\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45"
Oct 30 13:09:25.158320 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.158299    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b96befc8-d310-489e-a714-2a06ccd4455c-istio-envoy\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4\" (UID: \"b96befc8-d310-489e-a714-2a06ccd4455c\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4"
Oct 30 13:09:25.158446 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.158342    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvt6r\" (UniqueName: \"kubernetes.io/projected/b835314b-3c09-4143-966b-928a86200e6c-kube-api-access-rvt6r\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45\" (UID: \"b835314b-3c09-4143-966b-928a86200e6c\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45"
Oct 30 13:09:25.158446 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.158401    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b96befc8-d310-489e-a714-2a06ccd4455c-workload-socket\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4\" (UID: \"b96befc8-d310-489e-a714-2a06ccd4455c\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4"
Oct 30 13:09:25.158446 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.158428    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b96befc8-d310-489e-a714-2a06ccd4455c-istio-podinfo\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4\" (UID: \"b96befc8-d310-489e-a714-2a06ccd4455c\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4"
Oct 30 13:09:25.158670 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.158477    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4gp2\" (UniqueName: \"kubernetes.io/projected/b96befc8-d310-489e-a714-2a06ccd4455c-kube-api-access-n4gp2\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4\" (UID: \"b96befc8-d310-489e-a714-2a06ccd4455c\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4"
Oct 30 13:09:25.158670 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.158500    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b835314b-3c09-4143-966b-928a86200e6c-credential-socket\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45\" (UID: \"b835314b-3c09-4143-966b-928a86200e6c\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45"
Oct 30 13:09:25.158670 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.158562    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b96befc8-d310-489e-a714-2a06ccd4455c-istio-data\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4\" (UID: \"b96befc8-d310-489e-a714-2a06ccd4455c\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4"
Oct 30 13:09:25.158670 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.158611    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b835314b-3c09-4143-966b-928a86200e6c-istio-token\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45\" (UID: \"b835314b-3c09-4143-966b-928a86200e6c\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45"
Oct 30 13:09:25.158670 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.158649    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b96befc8-d310-489e-a714-2a06ccd4455c-workload-certs\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4\" (UID: \"b96befc8-d310-489e-a714-2a06ccd4455c\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4"
Oct 30 13:09:25.158944 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.158752    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b835314b-3c09-4143-966b-928a86200e6c-istiod-ca-cert\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45\" (UID: \"b835314b-3c09-4143-966b-928a86200e6c\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45"
Oct 30 13:09:25.158944 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.158895    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b835314b-3c09-4143-966b-928a86200e6c-istio-podinfo\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45\" (UID: \"b835314b-3c09-4143-966b-928a86200e6c\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45"
Oct 30 13:09:25.159046 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.158951    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b835314b-3c09-4143-966b-928a86200e6c-workload-socket\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45\" (UID: \"b835314b-3c09-4143-966b-928a86200e6c\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45"
Oct 30 13:09:25.237163 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.237087    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb"
Oct 30 13:09:25.260487 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.260454    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b835314b-3c09-4143-966b-928a86200e6c-workload-certs\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45\" (UID: \"b835314b-3c09-4143-966b-928a86200e6c\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45"
Oct 30 13:09:25.260624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.260576    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b96befc8-d310-489e-a714-2a06ccd4455c-istio-envoy\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4\" (UID: \"b96befc8-d310-489e-a714-2a06ccd4455c\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4"
Oct 30 13:09:25.260687 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.260632    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvt6r\" (UniqueName: \"kubernetes.io/projected/b835314b-3c09-4143-966b-928a86200e6c-kube-api-access-rvt6r\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45\" (UID: \"b835314b-3c09-4143-966b-928a86200e6c\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45"
Oct 30 13:09:25.260729 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.260692    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b96befc8-d310-489e-a714-2a06ccd4455c-workload-socket\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4\" (UID: \"b96befc8-d310-489e-a714-2a06ccd4455c\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4"
Oct 30 13:09:25.260729 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.260719    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b96befc8-d310-489e-a714-2a06ccd4455c-istio-podinfo\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4\" (UID: \"b96befc8-d310-489e-a714-2a06ccd4455c\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4"
Oct 30 13:09:25.260805 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.260767    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4gp2\" (UniqueName: \"kubernetes.io/projected/b96befc8-d310-489e-a714-2a06ccd4455c-kube-api-access-n4gp2\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4\" (UID: \"b96befc8-d310-489e-a714-2a06ccd4455c\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4"
Oct 30 13:09:25.260805 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.260797    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b835314b-3c09-4143-966b-928a86200e6c-credential-socket\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45\" (UID: \"b835314b-3c09-4143-966b-928a86200e6c\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45"
Oct 30 13:09:25.260877 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.260856    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b96befc8-d310-489e-a714-2a06ccd4455c-istio-data\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4\" (UID: \"b96befc8-d310-489e-a714-2a06ccd4455c\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4"
Oct 30 13:09:25.260916 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.260879    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b835314b-3c09-4143-966b-928a86200e6c-workload-certs\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45\" (UID: \"b835314b-3c09-4143-966b-928a86200e6c\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45"
Oct 30 13:09:25.261208 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.260888    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b835314b-3c09-4143-966b-928a86200e6c-istio-token\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45\" (UID: \"b835314b-3c09-4143-966b-928a86200e6c\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45"
Oct 30 13:09:25.261306 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.261244    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b96befc8-d310-489e-a714-2a06ccd4455c-workload-certs\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4\" (UID: \"b96befc8-d310-489e-a714-2a06ccd4455c\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4"
Oct 30 13:09:25.261306 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.261291    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b835314b-3c09-4143-966b-928a86200e6c-istiod-ca-cert\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45\" (UID: \"b835314b-3c09-4143-966b-928a86200e6c\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45"
Oct 30 13:09:25.261427 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.261332    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b835314b-3c09-4143-966b-928a86200e6c-istio-podinfo\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45\" (UID: \"b835314b-3c09-4143-966b-928a86200e6c\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45"
Oct 30 13:09:25.261427 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.261385    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b835314b-3c09-4143-966b-928a86200e6c-workload-socket\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45\" (UID: \"b835314b-3c09-4143-966b-928a86200e6c\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45"
Oct 30 13:09:25.261427 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.261420    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b835314b-3c09-4143-966b-928a86200e6c-credential-socket\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45\" (UID: \"b835314b-3c09-4143-966b-928a86200e6c\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45"
Oct 30 13:09:25.261623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.261486    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b96befc8-d310-489e-a714-2a06ccd4455c-credential-socket\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4\" (UID: \"b96befc8-d310-489e-a714-2a06ccd4455c\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4"
Oct 30 13:09:25.261623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.261515    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b835314b-3c09-4143-966b-928a86200e6c-istio-data\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45\" (UID: \"b835314b-3c09-4143-966b-928a86200e6c\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45"
Oct 30 13:09:25.261623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.261566    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b96befc8-d310-489e-a714-2a06ccd4455c-istiod-ca-cert\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4\" (UID: \"b96befc8-d310-489e-a714-2a06ccd4455c\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4"
Oct 30 13:09:25.261623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.261613    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b96befc8-d310-489e-a714-2a06ccd4455c-istio-data\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4\" (UID: \"b96befc8-d310-489e-a714-2a06ccd4455c\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4"
Oct 30 13:09:25.261839 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.261673    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b835314b-3c09-4143-966b-928a86200e6c-istio-envoy\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45\" (UID: \"b835314b-3c09-4143-966b-928a86200e6c\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45"
Oct 30 13:09:25.261839 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.261708    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b96befc8-d310-489e-a714-2a06ccd4455c-istio-token\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4\" (UID: \"b96befc8-d310-489e-a714-2a06ccd4455c\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4"
Oct 30 13:09:25.261839 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.261784    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b835314b-3c09-4143-966b-928a86200e6c-workload-socket\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45\" (UID: \"b835314b-3c09-4143-966b-928a86200e6c\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45"
Oct 30 13:09:25.262022 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.261184    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b96befc8-d310-489e-a714-2a06ccd4455c-workload-socket\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4\" (UID: \"b96befc8-d310-489e-a714-2a06ccd4455c\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4"
Oct 30 13:09:25.262323 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.262287    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b96befc8-d310-489e-a714-2a06ccd4455c-workload-certs\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4\" (UID: \"b96befc8-d310-489e-a714-2a06ccd4455c\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4"
Oct 30 13:09:25.262323 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.262288    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b96befc8-d310-489e-a714-2a06ccd4455c-istiod-ca-cert\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4\" (UID: \"b96befc8-d310-489e-a714-2a06ccd4455c\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4"
Oct 30 13:09:25.262667 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.262499    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b96befc8-d310-489e-a714-2a06ccd4455c-credential-socket\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4\" (UID: \"b96befc8-d310-489e-a714-2a06ccd4455c\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4"
Oct 30 13:09:25.263292 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.262935    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b835314b-3c09-4143-966b-928a86200e6c-istiod-ca-cert\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45\" (UID: \"b835314b-3c09-4143-966b-928a86200e6c\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45"
Oct 30 13:09:25.263292 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.263070    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b835314b-3c09-4143-966b-928a86200e6c-istio-data\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45\" (UID: \"b835314b-3c09-4143-966b-928a86200e6c\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45"
Oct 30 13:09:25.263292 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.263200    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b96befc8-d310-489e-a714-2a06ccd4455c-istio-envoy\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4\" (UID: \"b96befc8-d310-489e-a714-2a06ccd4455c\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4"
Oct 30 13:09:25.265715 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.265695    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b835314b-3c09-4143-966b-928a86200e6c-istio-envoy\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45\" (UID: \"b835314b-3c09-4143-966b-928a86200e6c\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45"
Oct 30 13:09:25.265826 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.265748    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b835314b-3c09-4143-966b-928a86200e6c-istio-podinfo\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45\" (UID: \"b835314b-3c09-4143-966b-928a86200e6c\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45"
Oct 30 13:09:25.265894 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.265869    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b96befc8-d310-489e-a714-2a06ccd4455c-istio-podinfo\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4\" (UID: \"b96befc8-d310-489e-a714-2a06ccd4455c\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4"
Oct 30 13:09:25.312860 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.312798    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4gp2\" (UniqueName: \"kubernetes.io/projected/b96befc8-d310-489e-a714-2a06ccd4455c-kube-api-access-n4gp2\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4\" (UID: \"b96befc8-d310-489e-a714-2a06ccd4455c\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4"
Oct 30 13:09:25.314160 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.314104    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b96befc8-d310-489e-a714-2a06ccd4455c-istio-token\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4\" (UID: \"b96befc8-d310-489e-a714-2a06ccd4455c\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4"
Oct 30 13:09:25.317740 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.316894    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b835314b-3c09-4143-966b-928a86200e6c-istio-token\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45\" (UID: \"b835314b-3c09-4143-966b-928a86200e6c\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45"
Oct 30 13:09:25.317740 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.317055    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvt6r\" (UniqueName: \"kubernetes.io/projected/b835314b-3c09-4143-966b-928a86200e6c-kube-api-access-rvt6r\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45\" (UID: \"b835314b-3c09-4143-966b-928a86200e6c\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45"
Oct 30 13:09:25.460534 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.460509    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45"
Oct 30 13:09:25.476905 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.476879    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4"
Oct 30 13:09:25.613306 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:09:25.613275    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59c001b5_b0f7_4898_ba90_aa8a0b96a6e7.slice/crio-7f352153d5ad5d4b6bbf89f4b703b649ab5261ba5a95bc9ac130890439deb7db WatchSource:0}: Error finding container 7f352153d5ad5d4b6bbf89f4b703b649ab5261ba5a95bc9ac130890439deb7db: Status 404 returned error can't find the container with id 7f352153d5ad5d4b6bbf89f4b703b649ab5261ba5a95bc9ac130890439deb7db
Oct 30 13:09:25.624938 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.616645    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:09:25.624938 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.616733    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:09:25.624938 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.616774    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:09:25.624938 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.616817    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb"]
Oct 30 13:09:25.797697 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:09:25.797662    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb835314b_3c09_4143_966b_928a86200e6c.slice/crio-6ee2a82403a88a42543d04dce16d1d3e9458f5f59c95d4cf0391d37d3f010e2b WatchSource:0}: Error finding container 6ee2a82403a88a42543d04dce16d1d3e9458f5f59c95d4cf0391d37d3f010e2b: Status 404 returned error can't find the container with id 6ee2a82403a88a42543d04dce16d1d3e9458f5f59c95d4cf0391d37d3f010e2b
Oct 30 13:09:25.798827 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.798804    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45"]
Oct 30 13:09:25.800435 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.800399    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:09:25.800536 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.800483    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:09:25.800536 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.800526    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:09:25.809159 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.809136    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4"]
Oct 30 13:09:25.811782 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:09:25.811750    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb96befc8_d310_489e_a714_2a06ccd4455c.slice/crio-207f96bf6c3b1141fc4088f4a4b9dbd4b0e9f735bac871e53eadd43b06e1c4bc WatchSource:0}: Error finding container 207f96bf6c3b1141fc4088f4a4b9dbd4b0e9f735bac871e53eadd43b06e1c4bc: Status 404 returned error can't find the container with id 207f96bf6c3b1141fc4088f4a4b9dbd4b0e9f735bac871e53eadd43b06e1c4bc
Oct 30 13:09:25.814231 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.814205    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:09:25.814324 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.814260    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:09:25.814324 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:25.814288    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:09:26.660265 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:26.659961    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb" event={"ID":"59c001b5-b0f7-4898-ba90-aa8a0b96a6e7","Type":"ContainerStarted","Data":"9388d1a63e7622e5bdab5846d099b319327e2a2d5686a27207ef54b1dd247a86"}
Oct 30 13:09:26.660715 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:26.660290    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb" event={"ID":"59c001b5-b0f7-4898-ba90-aa8a0b96a6e7","Type":"ContainerStarted","Data":"7f352153d5ad5d4b6bbf89f4b703b649ab5261ba5a95bc9ac130890439deb7db"}
Oct 30 13:09:26.661904 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:26.661877    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4" event={"ID":"b96befc8-d310-489e-a714-2a06ccd4455c","Type":"ContainerStarted","Data":"60e747d0babe56a27a485269a80fe2afb8a6a61328f0366e890dddd2656dd332"}
Oct 30 13:09:26.662015 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:26.661922    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4" event={"ID":"b96befc8-d310-489e-a714-2a06ccd4455c","Type":"ContainerStarted","Data":"207f96bf6c3b1141fc4088f4a4b9dbd4b0e9f735bac871e53eadd43b06e1c4bc"}
Oct 30 13:09:26.669279 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:26.669248    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45" event={"ID":"b835314b-3c09-4143-966b-928a86200e6c","Type":"ContainerStarted","Data":"e1aaa7f1ff3fb33a9df88cb64536a8fb57fb64d03cd81347e8e56bded80bf1a2"}
Oct 30 13:09:26.669424 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:26.669300    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45" event={"ID":"b835314b-3c09-4143-966b-928a86200e6c","Type":"ContainerStarted","Data":"6ee2a82403a88a42543d04dce16d1d3e9458f5f59c95d4cf0391d37d3f010e2b"}
Oct 30 13:09:26.762660 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:26.762398    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb" podStartSLOduration=3.762362751 podStartE2EDuration="3.762362751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:26.737010799 +0000 UTC m=+1065.903375448" watchObservedRunningTime="2026-10-30 13:09:26.762362751 +0000 UTC m=+1065.928727402"
Oct 30 13:09:26.805057 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:26.804995    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45" podStartSLOduration=3.804977981 podStartE2EDuration="3.804977981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:26.804490666 +0000 UTC m=+1065.970855315" watchObservedRunningTime="2026-10-30 13:09:26.804977981 +0000 UTC m=+1065.971342633"
Oct 30 13:09:26.862343 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:26.862180    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4" podStartSLOduration=2.862160238 podStartE2EDuration="2.862160238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:26.859357613 +0000 UTC m=+1066.025722274" watchObservedRunningTime="2026-10-30 13:09:26.862160238 +0000 UTC m=+1066.028524890"
Oct 30 13:09:27.247621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:27.246842    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb"
Oct 30 13:09:27.247621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:27.247097    4364 patch_prober.go:29] interesting pod/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 30 13:09:27.247621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:27.247164    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb" podUID="59c001b5-b0f7-4898-ba90-aa8a0b96a6e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 30 13:09:27.462815 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:27.462776    4364 patch_prober.go:29] interesting pod/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 13:09:27.463064 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:27.462856    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45" podUID="b835314b-3c09-4143-966b-928a86200e6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 13:09:27.463152 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:27.463136    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45"
Oct 30 13:09:27.477263 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:27.477242    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4"
Oct 30 13:09:27.478186 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:27.478167    4364 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 30 13:09:27.478279 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:27.478220    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4" podUID="b96befc8-d310-489e-a714-2a06ccd4455c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 30 13:09:28.251855 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:28.251820    4364 patch_prober.go:29] interesting pod/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 30 13:09:28.252926 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:28.252881    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb" podUID="59c001b5-b0f7-4898-ba90-aa8a0b96a6e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 30 13:09:28.461406 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:28.461372    4364 patch_prober.go:29] interesting pod/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 13:09:28.461681 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:28.461652    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45" podUID="b835314b-3c09-4143-966b-928a86200e6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 13:09:28.477591 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:28.477562    4364 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 30 13:09:28.477739 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:28.477631    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4" podUID="b96befc8-d310-489e-a714-2a06ccd4455c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 30 13:09:29.251105 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:29.251069    4364 patch_prober.go:29] interesting pod/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 30 13:09:29.251402 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:29.251372    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb" podUID="59c001b5-b0f7-4898-ba90-aa8a0b96a6e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 30 13:09:29.300867 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:29.300833    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr"
Oct 30 13:09:29.373913 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:29.373882    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr"
Oct 30 13:09:29.463042 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:29.463004    4364 patch_prober.go:29] interesting pod/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 13:09:29.463319 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:29.463291    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45" podUID="b835314b-3c09-4143-966b-928a86200e6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 13:09:29.478055 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:29.478025    4364 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 30 13:09:29.478186 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:29.478064    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4" podUID="b96befc8-d310-489e-a714-2a06ccd4455c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 30 13:09:29.977865 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:29.977833    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr"
Oct 30 13:09:30.466624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:30.445313    4364 patch_prober.go:29] interesting pod/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 30 13:09:30.467234 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:30.467202    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb" podUID="59c001b5-b0f7-4898-ba90-aa8a0b96a6e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 30 13:09:30.467379 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:30.466963    4364 patch_prober.go:29] interesting pod/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 13:09:30.467505 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:30.467485    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45" podUID="b835314b-3c09-4143-966b-928a86200e6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 13:09:30.478672 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:30.478306    4364 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 30 13:09:30.478672 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:30.478339    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4" podUID="b96befc8-d310-489e-a714-2a06ccd4455c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 30 13:09:31.241579 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:31.241492    4364 patch_prober.go:29] interesting pod/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 30 13:09:31.241916 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:31.241880    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb" podUID="59c001b5-b0f7-4898-ba90-aa8a0b96a6e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 30 13:09:31.462666 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:31.462588    4364 patch_prober.go:29] interesting pod/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 13:09:31.462867 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:31.462707    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45" podUID="b835314b-3c09-4143-966b-928a86200e6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 13:09:31.477372 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:31.477337    4364 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 30 13:09:31.477754 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:31.477383    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4" podUID="b96befc8-d310-489e-a714-2a06ccd4455c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 30 13:09:32.238129 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:32.238087    4364 patch_prober.go:29] interesting pod/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 30 13:09:32.238398 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:32.238144    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb" podUID="59c001b5-b0f7-4898-ba90-aa8a0b96a6e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 30 13:09:32.461528 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:32.461497    4364 patch_prober.go:29] interesting pod/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 13:09:32.461765 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:32.461559    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45" podUID="b835314b-3c09-4143-966b-928a86200e6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 13:09:32.477861 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:32.477833    4364 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 30 13:09:32.478823 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:32.478787    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4" podUID="b96befc8-d310-489e-a714-2a06ccd4455c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 30 13:09:33.625648 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:33.623659    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb"
Oct 30 13:09:33.625648 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:33.625240    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb"
Oct 30 13:09:33.626391 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:33.626370    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb"
Oct 30 13:09:33.668265 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:33.668236    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4"
Oct 30 13:09:33.670508 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:33.670183    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4"
Oct 30 13:09:33.671970 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:33.671114    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4"
Oct 30 13:09:34.035673 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:34.035580    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45"
Oct 30 13:09:34.036516 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:34.036487    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45"
Oct 30 13:09:34.037734 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:34.037715    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45"
Oct 30 13:09:39.151280 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:39.151244    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8752-967514"]
Oct 30 13:09:39.160851 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:39.160818    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8752-967514"
Oct 30 13:09:39.162987 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:39.162957    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8752-967514"]
Oct 30 13:09:39.207399 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:39.206371    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-nqkjx\""
Oct 30 13:09:39.228629 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:39.228582    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccbsk\" (UniqueName: \"kubernetes.io/projected/5216d0e5-7792-4c3c-8dc3-9adc32bebdc0-kube-api-access-ccbsk\") pod \"test-curl-8752-967514\" (UID: \"5216d0e5-7792-4c3c-8dc3-9adc32bebdc0\") " pod="opendatahub/test-curl-8752-967514"
Oct 30 13:09:39.330521 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:39.330487    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccbsk\" (UniqueName: \"kubernetes.io/projected/5216d0e5-7792-4c3c-8dc3-9adc32bebdc0-kube-api-access-ccbsk\") pod \"test-curl-8752-967514\" (UID: \"5216d0e5-7792-4c3c-8dc3-9adc32bebdc0\") " pod="opendatahub/test-curl-8752-967514"
Oct 30 13:09:39.355636 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:39.355588    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccbsk\" (UniqueName: \"kubernetes.io/projected/5216d0e5-7792-4c3c-8dc3-9adc32bebdc0-kube-api-access-ccbsk\") pod \"test-curl-8752-967514\" (UID: \"5216d0e5-7792-4c3c-8dc3-9adc32bebdc0\") " pod="opendatahub/test-curl-8752-967514"
Oct 30 13:09:39.521907 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:39.521851    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8752-967514"
Oct 30 13:09:39.826180 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:09:39.826144    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5216d0e5_7792_4c3c_8dc3_9adc32bebdc0.slice/crio-d689d7c58c82a2a5184ae51aa128e6cda0b56b63d69caa210f13b134f008b7c1 WatchSource:0}: Error finding container d689d7c58c82a2a5184ae51aa128e6cda0b56b63d69caa210f13b134f008b7c1: Status 404 returned error can't find the container with id d689d7c58c82a2a5184ae51aa128e6cda0b56b63d69caa210f13b134f008b7c1
Oct 30 13:09:39.827338 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:39.827313    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8752-967514"]
Oct 30 13:09:39.828365 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:39.828350    4364 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 13:09:39.982072 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:39.982035    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8752-967514" event={"ID":"5216d0e5-7792-4c3c-8dc3-9adc32bebdc0","Type":"ContainerStarted","Data":"d689d7c58c82a2a5184ae51aa128e6cda0b56b63d69caa210f13b134f008b7c1"}
Oct 30 13:09:40.991953 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:40.991916    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8752-967514" event={"ID":"5216d0e5-7792-4c3c-8dc3-9adc32bebdc0","Type":"ContainerStarted","Data":"32ebfdd8096018b41e2f1823ec967060e3cae947cd7aa11077ca982047efde92"}
Oct 30 13:09:41.034466 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:41.034420    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8752-967514" podStartSLOduration=1.373275882 podStartE2EDuration="2.034408366s" totalImagesPullingTime="661.132484ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:41.033570268 +0000 UTC m=+1080.199934917" watchObservedRunningTime="2026-10-30 13:09:41.034408366 +0000 UTC m=+1080.200773014"
Oct 30 13:09:41.505076 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:09:41.505037    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:09:41.908649 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:41.908584    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8752-967514"]
Oct 30 13:09:41.915625 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:41.915580    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8752-967514"]
Oct 30 13:09:42.067954 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:42.067928    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8752-1e157b"]
Oct 30 13:09:42.068661 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:42.068643    4364 cpu_manager.go:414] "containerMap: removing container" podUID="5216d0e5-7792-4c3c-8dc3-9adc32bebdc0" containerName="test-curl-8752-967514"
Oct 30 13:09:42.068717 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:42.068663    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5216d0e5-7792-4c3c-8dc3-9adc32bebdc0" containerName="test-curl-8752-967514"
Oct 30 13:09:42.068835 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:42.068824    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="5216d0e5-7792-4c3c-8dc3-9adc32bebdc0" containerName="test-curl-8752-967514"
Oct 30 13:09:42.072803 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:42.072784    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8752-1e157b"
Oct 30 13:09:42.087316 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:42.087293    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8752-1e157b"]
Oct 30 13:09:42.166640 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:42.166557    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khvrc\" (UniqueName: \"kubernetes.io/projected/2598810c-b732-4bfe-a3bf-cfe05206609a-kube-api-access-khvrc\") pod \"test-curl-8752-1e157b\" (UID: \"2598810c-b732-4bfe-a3bf-cfe05206609a\") " pod="opendatahub/test-curl-8752-1e157b"
Oct 30 13:09:42.268270 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:42.268245    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khvrc\" (UniqueName: \"kubernetes.io/projected/2598810c-b732-4bfe-a3bf-cfe05206609a-kube-api-access-khvrc\") pod \"test-curl-8752-1e157b\" (UID: \"2598810c-b732-4bfe-a3bf-cfe05206609a\") " pod="opendatahub/test-curl-8752-1e157b"
Oct 30 13:09:42.294095 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:42.294066    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khvrc\" (UniqueName: \"kubernetes.io/projected/2598810c-b732-4bfe-a3bf-cfe05206609a-kube-api-access-khvrc\") pod \"test-curl-8752-1e157b\" (UID: \"2598810c-b732-4bfe-a3bf-cfe05206609a\") " pod="opendatahub/test-curl-8752-1e157b"
Oct 30 13:09:42.387437 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:42.387411    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8752-1e157b"
Oct 30 13:09:42.743094 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:42.728403    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8752-1e157b"]
Oct 30 13:09:43.007067 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:43.006987    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8752-1e157b" event={"ID":"2598810c-b732-4bfe-a3bf-cfe05206609a","Type":"ContainerStarted","Data":"9234961608e3cede0bce726bc5597a047a135cfafe32d334f8d3a777291df779"}
Oct 30 13:09:43.007067 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:43.007026    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8752-1e157b" event={"ID":"2598810c-b732-4bfe-a3bf-cfe05206609a","Type":"ContainerStarted","Data":"47d246100e4657257c73dfa8572e96ab941145ee2c07b7dccea07d3bd901ff5f"}
Oct 30 13:09:43.007270 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:43.007040    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8752-967514" podUID="5216d0e5-7792-4c3c-8dc3-9adc32bebdc0" containerName="test-curl-8752-967514" containerID="cri-o://32ebfdd8096018b41e2f1823ec967060e3cae947cd7aa11077ca982047efde92" gracePeriod=2
Oct 30 13:09:43.057953 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:43.057908    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8752-1e157b" podStartSLOduration=1.057896858 podStartE2EDuration="1.057896858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:43.052160765 +0000 UTC m=+1082.218525414" watchObservedRunningTime="2026-10-30 13:09:43.057896858 +0000 UTC m=+1082.224261506"
Oct 30 13:09:44.052095 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:44.052068    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8752-1e157b"]
Oct 30 13:09:44.063526 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:44.063499    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8752-1e157b"]
Oct 30 13:09:44.151720 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:44.151693    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-d833bd/distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74"]
Oct 30 13:09:44.152331 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:44.152319    4364 cpu_manager.go:414] "containerMap: removing container" podUID="2598810c-b732-4bfe-a3bf-cfe05206609a" containerName="test-curl-8752-1e157b"
Oct 30 13:09:44.152386 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:44.152333    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2598810c-b732-4bfe-a3bf-cfe05206609a" containerName="test-curl-8752-1e157b"
Oct 30 13:09:44.152459 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:44.152450    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="2598810c-b732-4bfe-a3bf-cfe05206609a" containerName="test-curl-8752-1e157b"
Oct 30 13:09:44.156001 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:44.155984    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d833bd/distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74"
Oct 30 13:09:44.177711 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:44.177687    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-d833bd/distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74"]
Oct 30 13:09:44.186446 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:44.186427    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-d833bd\"/\"distinct-2-w3-d833bd-kserve-self-signed-certs\""
Oct 30 13:09:44.306761 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:44.306690    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f2b06ac-9bf8-4b57-bd22-85532576f9c1-tls-certs\") pod \"distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74\" (UID: \"4f2b06ac-9bf8-4b57-bd22-85532576f9c1\") " pod="e2e-models-e2e-worker-w3-d833bd/distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74"
Oct 30 13:09:44.306761 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:44.306733    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx8wb\" (UniqueName: \"kubernetes.io/projected/4f2b06ac-9bf8-4b57-bd22-85532576f9c1-kube-api-access-dx8wb\") pod \"distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74\" (UID: \"4f2b06ac-9bf8-4b57-bd22-85532576f9c1\") " pod="e2e-models-e2e-worker-w3-d833bd/distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74"
Oct 30 13:09:44.306761 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:44.306760    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f2b06ac-9bf8-4b57-bd22-85532576f9c1-model-cache\") pod \"distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74\" (UID: \"4f2b06ac-9bf8-4b57-bd22-85532576f9c1\") " pod="e2e-models-e2e-worker-w3-d833bd/distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74"
Oct 30 13:09:44.306956 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:44.306895    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f2b06ac-9bf8-4b57-bd22-85532576f9c1-dshm\") pod \"distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74\" (UID: \"4f2b06ac-9bf8-4b57-bd22-85532576f9c1\") " pod="e2e-models-e2e-worker-w3-d833bd/distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74"
Oct 30 13:09:44.307053 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:44.307033    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f2b06ac-9bf8-4b57-bd22-85532576f9c1-home\") pod \"distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74\" (UID: \"4f2b06ac-9bf8-4b57-bd22-85532576f9c1\") " pod="e2e-models-e2e-worker-w3-d833bd/distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74"
Oct 30 13:09:44.307099 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:44.307086    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f2b06ac-9bf8-4b57-bd22-85532576f9c1-tmp-dir\") pod \"distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74\" (UID: \"4f2b06ac-9bf8-4b57-bd22-85532576f9c1\") " pod="e2e-models-e2e-worker-w3-d833bd/distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74"
Oct 30 13:09:44.414623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:44.413696    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f2b06ac-9bf8-4b57-bd22-85532576f9c1-tls-certs\") pod \"distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74\" (UID: \"4f2b06ac-9bf8-4b57-bd22-85532576f9c1\") " pod="e2e-models-e2e-worker-w3-d833bd/distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74"
Oct 30 13:09:44.414623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:44.413770    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx8wb\" (UniqueName: \"kubernetes.io/projected/4f2b06ac-9bf8-4b57-bd22-85532576f9c1-kube-api-access-dx8wb\") pod \"distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74\" (UID: \"4f2b06ac-9bf8-4b57-bd22-85532576f9c1\") " pod="e2e-models-e2e-worker-w3-d833bd/distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74"
Oct 30 13:09:44.414623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:44.413816    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f2b06ac-9bf8-4b57-bd22-85532576f9c1-model-cache\") pod \"distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74\" (UID: \"4f2b06ac-9bf8-4b57-bd22-85532576f9c1\") " pod="e2e-models-e2e-worker-w3-d833bd/distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74"
Oct 30 13:09:44.414623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:44.413906    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f2b06ac-9bf8-4b57-bd22-85532576f9c1-dshm\") pod \"distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74\" (UID: \"4f2b06ac-9bf8-4b57-bd22-85532576f9c1\") " pod="e2e-models-e2e-worker-w3-d833bd/distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74"
Oct 30 13:09:44.414623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:44.414090    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f2b06ac-9bf8-4b57-bd22-85532576f9c1-home\") pod \"distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74\" (UID: \"4f2b06ac-9bf8-4b57-bd22-85532576f9c1\") " pod="e2e-models-e2e-worker-w3-d833bd/distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74"
Oct 30 13:09:44.414623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:44.414122    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f2b06ac-9bf8-4b57-bd22-85532576f9c1-tmp-dir\") pod \"distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74\" (UID: \"4f2b06ac-9bf8-4b57-bd22-85532576f9c1\") " pod="e2e-models-e2e-worker-w3-d833bd/distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74"
Oct 30 13:09:44.415034 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:44.414664    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f2b06ac-9bf8-4b57-bd22-85532576f9c1-tmp-dir\") pod \"distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74\" (UID: \"4f2b06ac-9bf8-4b57-bd22-85532576f9c1\") " pod="e2e-models-e2e-worker-w3-d833bd/distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74"
Oct 30 13:09:44.418635 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:44.415222    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f2b06ac-9bf8-4b57-bd22-85532576f9c1-model-cache\") pod \"distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74\" (UID: \"4f2b06ac-9bf8-4b57-bd22-85532576f9c1\") " pod="e2e-models-e2e-worker-w3-d833bd/distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74"
Oct 30 13:09:44.430631 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:44.425272    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f2b06ac-9bf8-4b57-bd22-85532576f9c1-home\") pod \"distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74\" (UID: \"4f2b06ac-9bf8-4b57-bd22-85532576f9c1\") " pod="e2e-models-e2e-worker-w3-d833bd/distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74"
Oct 30 13:09:44.459872 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:44.458101    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx8wb\" (UniqueName: \"kubernetes.io/projected/4f2b06ac-9bf8-4b57-bd22-85532576f9c1-kube-api-access-dx8wb\") pod \"distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74\" (UID: \"4f2b06ac-9bf8-4b57-bd22-85532576f9c1\") " pod="e2e-models-e2e-worker-w3-d833bd/distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74"
Oct 30 13:09:44.459872 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:44.459042    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f2b06ac-9bf8-4b57-bd22-85532576f9c1-tls-certs\") pod \"distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74\" (UID: \"4f2b06ac-9bf8-4b57-bd22-85532576f9c1\") " pod="e2e-models-e2e-worker-w3-d833bd/distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74"
Oct 30 13:09:44.484235 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:44.484198    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f2b06ac-9bf8-4b57-bd22-85532576f9c1-dshm\") pod \"distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74\" (UID: \"4f2b06ac-9bf8-4b57-bd22-85532576f9c1\") " pod="e2e-models-e2e-worker-w3-d833bd/distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74"
Oct 30 13:09:44.782866 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:44.771974    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d833bd/distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74"
Oct 30 13:09:45.023308 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:45.023275    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8752-1e157b" podUID="2598810c-b732-4bfe-a3bf-cfe05206609a" containerName="test-curl-8752-1e157b" containerID="cri-o://9234961608e3cede0bce726bc5597a047a135cfafe32d334f8d3a777291df779" gracePeriod=2
Oct 30 13:09:45.159205 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:45.158941    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-d833bd/distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74"]
Oct 30 13:09:45.213906 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:45.213885    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8752-967514"
Oct 30 13:09:45.358048 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:45.357966    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5216d0e5-7792-4c3c-8dc3-9adc32bebdc0-kube-api-access-ccbsk\" (UniqueName: \"kubernetes.io/projected/5216d0e5-7792-4c3c-8dc3-9adc32bebdc0-kube-api-access-ccbsk\") pod \"5216d0e5-7792-4c3c-8dc3-9adc32bebdc0\" (UID: \"5216d0e5-7792-4c3c-8dc3-9adc32bebdc0\") "
Oct 30 13:09:45.359887 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:45.359860    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5216d0e5-7792-4c3c-8dc3-9adc32bebdc0-kube-api-access-ccbsk" pod "5216d0e5-7792-4c3c-8dc3-9adc32bebdc0" (UID: "5216d0e5-7792-4c3c-8dc3-9adc32bebdc0"). InnerVolumeSpecName "kube-api-access-ccbsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:09:45.459923 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:45.459899    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccbsk\" (UniqueName: \"kubernetes.io/projected/5216d0e5-7792-4c3c-8dc3-9adc32bebdc0-kube-api-access-ccbsk\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:09:46.031761 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:46.031723    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d833bd/distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74" event={"ID":"4f2b06ac-9bf8-4b57-bd22-85532576f9c1","Type":"ContainerStarted","Data":"87b706902f5786f2505a8bb78db3558d561529d5e0d8ce0476a0bee04e885419"}
Oct 30 13:09:46.031935 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:46.031771    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d833bd/distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74" event={"ID":"4f2b06ac-9bf8-4b57-bd22-85532576f9c1","Type":"ContainerStarted","Data":"42075a37c9bf8bd2f71fd89de09f99af8d7df03e7ca8a33cfc4e1172ee8c7596"}
Oct 30 13:09:46.033185 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:46.033160    4364 generic.go:356] "Generic (PLEG): container finished" podID="5216d0e5-7792-4c3c-8dc3-9adc32bebdc0" containerID="32ebfdd8096018b41e2f1823ec967060e3cae947cd7aa11077ca982047efde92" exitCode=137
Oct 30 13:09:46.033313 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:46.033199    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8752-967514"
Oct 30 13:09:46.033313 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:46.033260    4364 scope.go:122] "RemoveContainer" containerID="32ebfdd8096018b41e2f1823ec967060e3cae947cd7aa11077ca982047efde92"
Oct 30 13:09:46.043663 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:46.043649    4364 scope.go:122] "RemoveContainer" containerID="32ebfdd8096018b41e2f1823ec967060e3cae947cd7aa11077ca982047efde92"
Oct 30 13:09:46.043914 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:09:46.043895    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32ebfdd8096018b41e2f1823ec967060e3cae947cd7aa11077ca982047efde92\": container with ID starting with 32ebfdd8096018b41e2f1823ec967060e3cae947cd7aa11077ca982047efde92 not found: ID does not exist" containerID="32ebfdd8096018b41e2f1823ec967060e3cae947cd7aa11077ca982047efde92"
Oct 30 13:09:46.043975 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:46.043918    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32ebfdd8096018b41e2f1823ec967060e3cae947cd7aa11077ca982047efde92"} err="failed to get container status \"32ebfdd8096018b41e2f1823ec967060e3cae947cd7aa11077ca982047efde92\": rpc error: code = NotFound desc = could not find container \"32ebfdd8096018b41e2f1823ec967060e3cae947cd7aa11077ca982047efde92\": container with ID starting with 32ebfdd8096018b41e2f1823ec967060e3cae947cd7aa11077ca982047efde92 not found: ID does not exist"
Oct 30 13:09:46.079163 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:46.079123    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-d833bd/distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74" podStartSLOduration=2.079111096 podStartE2EDuration="2.079111096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:46.076359537 +0000 UTC m=+1085.242724184" watchObservedRunningTime="2026-10-30 13:09:46.079111096 +0000 UTC m=+1085.245475744"
Oct 30 13:09:46.091923 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:09:46.091898    4364 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8752-967514\" is forbidden: User \"system:node:ip-10-0-137-6.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-6.ec2.internal' and this object" podUID="5216d0e5-7792-4c3c-8dc3-9adc32bebdc0" pod="opendatahub/test-curl-8752-967514"
Oct 30 13:09:46.778629 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:46.774629    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d833bd/distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74"
Oct 30 13:09:46.972406 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:46.972374    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5216d0e5-7792-4c3c-8dc3-9adc32bebdc0" path="/var/lib/kubelet/pods/5216d0e5-7792-4c3c-8dc3-9adc32bebdc0/volumes"
Oct 30 13:09:47.261888 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:47.261864    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8752-1e157b"
Oct 30 13:09:47.301074 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:47.301025    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2598810c-b732-4bfe-a3bf-cfe05206609a-kube-api-access-khvrc\" (UniqueName: \"kubernetes.io/projected/2598810c-b732-4bfe-a3bf-cfe05206609a-kube-api-access-khvrc\") pod \"2598810c-b732-4bfe-a3bf-cfe05206609a\" (UID: \"2598810c-b732-4bfe-a3bf-cfe05206609a\") "
Oct 30 13:09:47.303014 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:47.302977    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2598810c-b732-4bfe-a3bf-cfe05206609a-kube-api-access-khvrc" pod "2598810c-b732-4bfe-a3bf-cfe05206609a" (UID: "2598810c-b732-4bfe-a3bf-cfe05206609a"). InnerVolumeSpecName "kube-api-access-khvrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:09:47.405419 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:47.404835    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khvrc\" (UniqueName: \"kubernetes.io/projected/2598810c-b732-4bfe-a3bf-cfe05206609a-kube-api-access-khvrc\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:09:48.051255 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:48.051221    4364 generic.go:356] "Generic (PLEG): container finished" podID="2598810c-b732-4bfe-a3bf-cfe05206609a" containerID="9234961608e3cede0bce726bc5597a047a135cfafe32d334f8d3a777291df779" exitCode=137
Oct 30 13:09:48.051704 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:48.051275    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8752-1e157b"
Oct 30 13:09:48.051704 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:48.051317    4364 scope.go:122] "RemoveContainer" containerID="9234961608e3cede0bce726bc5597a047a135cfafe32d334f8d3a777291df779"
Oct 30 13:09:48.061382 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:48.061367    4364 scope.go:122] "RemoveContainer" containerID="9234961608e3cede0bce726bc5597a047a135cfafe32d334f8d3a777291df779"
Oct 30 13:09:48.061672 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:09:48.061649    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9234961608e3cede0bce726bc5597a047a135cfafe32d334f8d3a777291df779\": container with ID starting with 9234961608e3cede0bce726bc5597a047a135cfafe32d334f8d3a777291df779 not found: ID does not exist" containerID="9234961608e3cede0bce726bc5597a047a135cfafe32d334f8d3a777291df779"
Oct 30 13:09:48.061770 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:48.061679    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9234961608e3cede0bce726bc5597a047a135cfafe32d334f8d3a777291df779"} err="failed to get container status \"9234961608e3cede0bce726bc5597a047a135cfafe32d334f8d3a777291df779\": rpc error: code = NotFound desc = could not find container \"9234961608e3cede0bce726bc5597a047a135cfafe32d334f8d3a777291df779\": container with ID starting with 9234961608e3cede0bce726bc5597a047a135cfafe32d334f8d3a777291df779 not found: ID does not exist"
Oct 30 13:09:48.075099 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:09:48.075072    4364 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8752-1e157b\" is forbidden: User \"system:node:ip-10-0-137-6.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-6.ec2.internal' and this object" podUID="2598810c-b732-4bfe-a3bf-cfe05206609a" pod="opendatahub/test-curl-8752-1e157b"
Oct 30 13:09:49.010587 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:49.010546    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2598810c-b732-4bfe-a3bf-cfe05206609a" path="/var/lib/kubelet/pods/2598810c-b732-4bfe-a3bf-cfe05206609a/volumes"
Oct 30 13:09:49.828088 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:49.825696    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr" podUID="446c12df-de12-4c7c-8c71-dfed93cc33f1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:09:50.002624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:49.991422    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz" podUID="a568f1ce-602d-4069-a41c-9fe1a7635c5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:09:50.020906 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:50.020865    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz" podUID="bc5caf69-a649-47e5-8910-9b8f331dfd4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:09:50.086677 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:50.086217    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh" podUID="d41ac9e5-eeed-4e07-a7fa-05d3260630b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 30 13:09:50.425622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:50.425167    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc" podUID="8911b790-331d-490c-876b-647e6d2fcd14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:09:50.907187 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:50.905727    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr" podUID="446c12df-de12-4c7c-8c71-dfed93cc33f1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:09:50.907187 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:50.906137    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr"
Oct 30 13:09:51.086633 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:51.051619    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz" podUID="a568f1ce-602d-4069-a41c-9fe1a7635c5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 30 13:09:51.086633 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:51.052950    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz" podUID="bc5caf69-a649-47e5-8910-9b8f331dfd4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:09:51.086633 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:51.053301    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4" podUID="0cbe6af7-d884-4613-90b8-192e5ffdb027" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 13:09:51.266682 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:51.265694    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh" podUID="d41ac9e5-eeed-4e07-a7fa-05d3260630b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:09:51.266682 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:51.265805    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6" podUID="b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 13:09:51.428627 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:51.427690    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz"
Oct 30 13:09:51.456625 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:51.428856    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc" podUID="8911b790-331d-490c-876b-647e6d2fcd14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:09:51.456625 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:51.430968    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz"
Oct 30 13:09:51.592882 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:51.592801    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh"
Oct 30 13:09:51.596619 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:51.593126    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc"
Oct 30 13:09:51.952624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:51.928914    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr" podUID="446c12df-de12-4c7c-8c71-dfed93cc33f1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:09:52.052981 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:52.052258    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz" podUID="a568f1ce-602d-4069-a41c-9fe1a7635c5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:09:52.469084 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:52.469023    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz" podUID="bc5caf69-a649-47e5-8910-9b8f331dfd4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 13:09:52.637245 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:52.472397    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4" podUID="0cbe6af7-d884-4613-90b8-192e5ffdb027" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:09:52.652634 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:52.500646    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh" podUID="d41ac9e5-eeed-4e07-a7fa-05d3260630b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 30 13:09:52.849842 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:52.505854    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6" podUID="b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 13:09:52.919702 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:52.506181    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc" podUID="8911b790-331d-490c-876b-647e6d2fcd14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded"
Oct 30 13:09:52.920369 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:52.548982    4364 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:09:52.924658 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:52.920553    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq" podUID="1d08eaa5-399d-4d90-8a47-3c060abfb552" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:09:52.930626 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:52.637200    4364 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:09:52.962758 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:52.654844    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4"
Oct 30 13:09:53.136856 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:53.136795    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv" podUID="4a6cf8c7-ad37-46bc-a392-90de84af768e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:09:53.272553 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:53.272511    4364 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz" podUID="a568f1ce-602d-4069-a41c-9fe1a7635c5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:09:53.272889 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:53.272858    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz" podUID="a568f1ce-602d-4069-a41c-9fe1a7635c5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:09:53.273127 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:53.273110    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6"
Oct 30 13:09:53.279729 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:53.279684    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr" podUID="446c12df-de12-4c7c-8c71-dfed93cc33f1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:09:53.654634 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:53.653628    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz" podUID="bc5caf69-a649-47e5-8910-9b8f331dfd4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:09:53.789624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:53.748914    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc"
Oct 30 13:09:53.928826 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:53.928742    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4"
Oct 30 13:09:53.929042 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:53.809803    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh" podUID="d41ac9e5-eeed-4e07-a7fa-05d3260630b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:09:53.969624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:53.954863    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6" podUID="b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 13:09:54.012375 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:54.012329    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6"
Oct 30 13:09:54.304682 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:54.281701    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr" podUID="446c12df-de12-4c7c-8c71-dfed93cc33f1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded"
Oct 30 13:09:54.304682 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:54.303931    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz" podUID="a568f1ce-602d-4069-a41c-9fe1a7635c5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:09:54.390957 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:54.385197    4364 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:09:54.390957 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:54.385274    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v" podUID="cc6e612c-e20e-4573-a608-edbf4b7de570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:09:54.405377 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:54.405333    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc" podUID="8911b790-331d-490c-876b-647e6d2fcd14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:09:54.549472 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:54.549424    4364 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4" podUID="0cbe6af7-d884-4613-90b8-192e5ffdb027" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:09:54.659647 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:54.659582    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz" podUID="bc5caf69-a649-47e5-8910-9b8f331dfd4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:09:54.928571 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:54.928486    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5"]
Oct 30 13:09:54.998901 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:54.998859    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh" podUID="d41ac9e5-eeed-4e07-a7fa-05d3260630b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:09:55.273039 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:55.272958    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5"]
Oct 30 13:09:55.288012 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:55.273155    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5"
Oct 30 13:09:55.316031 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:55.315977    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr" podUID="446c12df-de12-4c7c-8c71-dfed93cc33f1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded"
Oct 30 13:09:55.318243 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:55.318204    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz" podUID="a568f1ce-602d-4069-a41c-9fe1a7635c5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:09:55.434505 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:55.394302    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-d833bd/distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74"
Oct 30 13:09:55.730622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:55.439896    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc" podUID="8911b790-331d-490c-876b-647e6d2fcd14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:09:55.741631 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:55.529302    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ee5a4af-eb64-470e-a42e-7d334cedce2d-credential-socket\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5\" (UID: \"8ee5a4af-eb64-470e-a42e-7d334cedce2d\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5"
Oct 30 13:09:55.741631 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:55.735003    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ee5a4af-eb64-470e-a42e-7d334cedce2d-workload-socket\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5\" (UID: \"8ee5a4af-eb64-470e-a42e-7d334cedce2d\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5"
Oct 30 13:09:55.741631 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:55.660709    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz" podUID="bc5caf69-a649-47e5-8910-9b8f331dfd4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 13:09:55.860864 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:55.860562    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ee5a4af-eb64-470e-a42e-7d334cedce2d-istio-podinfo\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5\" (UID: \"8ee5a4af-eb64-470e-a42e-7d334cedce2d\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5"
Oct 30 13:09:55.860864 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:55.860628    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ee5a4af-eb64-470e-a42e-7d334cedce2d-workload-certs\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5\" (UID: \"8ee5a4af-eb64-470e-a42e-7d334cedce2d\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5"
Oct 30 13:09:55.860864 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:55.860660    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ee5a4af-eb64-470e-a42e-7d334cedce2d-istiod-ca-cert\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5\" (UID: \"8ee5a4af-eb64-470e-a42e-7d334cedce2d\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5"
Oct 30 13:09:55.860864 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:55.860865    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ee5a4af-eb64-470e-a42e-7d334cedce2d-istio-data\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5\" (UID: \"8ee5a4af-eb64-470e-a42e-7d334cedce2d\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5"
Oct 30 13:09:55.869236 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:55.868007    4364 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w3-d833bd/distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74" podUID="4f2b06ac-9bf8-4b57-bd22-85532576f9c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:09:55.887549 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:55.871643    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ee5a4af-eb64-470e-a42e-7d334cedce2d-credential-socket\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5\" (UID: \"8ee5a4af-eb64-470e-a42e-7d334cedce2d\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5"
Oct 30 13:09:55.887549 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:55.871747    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xz6r\" (UniqueName: \"kubernetes.io/projected/8ee5a4af-eb64-470e-a42e-7d334cedce2d-kube-api-access-5xz6r\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5\" (UID: \"8ee5a4af-eb64-470e-a42e-7d334cedce2d\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5"
Oct 30 13:09:55.887549 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:55.871787    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ee5a4af-eb64-470e-a42e-7d334cedce2d-workload-socket\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5\" (UID: \"8ee5a4af-eb64-470e-a42e-7d334cedce2d\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5"
Oct 30 13:09:55.887549 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:55.871939    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ee5a4af-eb64-470e-a42e-7d334cedce2d-istio-token\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5\" (UID: \"8ee5a4af-eb64-470e-a42e-7d334cedce2d\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5"
Oct 30 13:09:55.887549 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:55.872016    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ee5a4af-eb64-470e-a42e-7d334cedce2d-istio-envoy\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5\" (UID: \"8ee5a4af-eb64-470e-a42e-7d334cedce2d\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5"
Oct 30 13:09:55.894857 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:55.894807    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ee5a4af-eb64-470e-a42e-7d334cedce2d-workload-socket\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5\" (UID: \"8ee5a4af-eb64-470e-a42e-7d334cedce2d\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5"
Oct 30 13:09:55.929331 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:55.736592    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc"
Oct 30 13:09:55.996124 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:55.993787    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ee5a4af-eb64-470e-a42e-7d334cedce2d-credential-socket\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5\" (UID: \"8ee5a4af-eb64-470e-a42e-7d334cedce2d\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5"
Oct 30 13:09:55.996124 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:55.995111    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ee5a4af-eb64-470e-a42e-7d334cedce2d-istio-podinfo\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5\" (UID: \"8ee5a4af-eb64-470e-a42e-7d334cedce2d\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5"
Oct 30 13:09:55.996124 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:55.995157    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ee5a4af-eb64-470e-a42e-7d334cedce2d-workload-certs\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5\" (UID: \"8ee5a4af-eb64-470e-a42e-7d334cedce2d\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5"
Oct 30 13:09:55.996124 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:55.995183    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ee5a4af-eb64-470e-a42e-7d334cedce2d-istiod-ca-cert\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5\" (UID: \"8ee5a4af-eb64-470e-a42e-7d334cedce2d\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5"
Oct 30 13:09:55.996124 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:55.995284    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ee5a4af-eb64-470e-a42e-7d334cedce2d-istio-data\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5\" (UID: \"8ee5a4af-eb64-470e-a42e-7d334cedce2d\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5"
Oct 30 13:09:55.996124 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:55.995380    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xz6r\" (UniqueName: \"kubernetes.io/projected/8ee5a4af-eb64-470e-a42e-7d334cedce2d-kube-api-access-5xz6r\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5\" (UID: \"8ee5a4af-eb64-470e-a42e-7d334cedce2d\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5"
Oct 30 13:09:55.996124 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:55.995465    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ee5a4af-eb64-470e-a42e-7d334cedce2d-istio-token\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5\" (UID: \"8ee5a4af-eb64-470e-a42e-7d334cedce2d\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5"
Oct 30 13:09:55.996124 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:55.995520    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ee5a4af-eb64-470e-a42e-7d334cedce2d-istio-envoy\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5\" (UID: \"8ee5a4af-eb64-470e-a42e-7d334cedce2d\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5"
Oct 30 13:09:56.009623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:56.004793    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ee5a4af-eb64-470e-a42e-7d334cedce2d-istiod-ca-cert\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5\" (UID: \"8ee5a4af-eb64-470e-a42e-7d334cedce2d\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5"
Oct 30 13:09:56.033727 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:56.025909    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh" podUID="d41ac9e5-eeed-4e07-a7fa-05d3260630b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 30 13:09:56.036767 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:56.036728    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4" podUID="0cbe6af7-d884-4613-90b8-192e5ffdb027" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 13:09:56.037289 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:56.037260    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6" podUID="b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 13:09:56.109993 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:56.064293    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ee5a4af-eb64-470e-a42e-7d334cedce2d-workload-certs\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5\" (UID: \"8ee5a4af-eb64-470e-a42e-7d334cedce2d\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5"
Oct 30 13:09:56.128124 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:56.097622    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ee5a4af-eb64-470e-a42e-7d334cedce2d-istio-data\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5\" (UID: \"8ee5a4af-eb64-470e-a42e-7d334cedce2d\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5"
Oct 30 13:09:56.254910 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:56.241768    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xz6r\" (UniqueName: \"kubernetes.io/projected/8ee5a4af-eb64-470e-a42e-7d334cedce2d-kube-api-access-5xz6r\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5\" (UID: \"8ee5a4af-eb64-470e-a42e-7d334cedce2d\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5"
Oct 30 13:09:56.255804 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:56.255675    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ee5a4af-eb64-470e-a42e-7d334cedce2d-istio-podinfo\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5\" (UID: \"8ee5a4af-eb64-470e-a42e-7d334cedce2d\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5"
Oct 30 13:09:56.524677 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:56.375717    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr" podUID="446c12df-de12-4c7c-8c71-dfed93cc33f1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:09:56.648792 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:56.648742    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz" podUID="a568f1ce-602d-4069-a41c-9fe1a7635c5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 30 13:09:56.662630 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:56.662444    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ee5a4af-eb64-470e-a42e-7d334cedce2d-istio-envoy\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5\" (UID: \"8ee5a4af-eb64-470e-a42e-7d334cedce2d\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5"
Oct 30 13:09:57.062728 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:57.062682    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh" podUID="d41ac9e5-eeed-4e07-a7fa-05d3260630b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 30 13:09:57.092874 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:57.088683    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6" podUID="b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:09:57.092874 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:57.089487    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6"
Oct 30 13:09:57.112624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:57.100661    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc" podUID="8911b790-331d-490c-876b-647e6d2fcd14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded"
Oct 30 13:09:57.177669 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:57.161132    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ee5a4af-eb64-470e-a42e-7d334cedce2d-istio-token\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5\" (UID: \"8ee5a4af-eb64-470e-a42e-7d334cedce2d\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5"
Oct 30 13:09:57.358685 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:57.357609    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5"
Oct 30 13:09:57.414656 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:57.411765    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4" podUID="0cbe6af7-d884-4613-90b8-192e5ffdb027" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 13:09:57.414656 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:57.412858    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4"
Oct 30 13:09:57.501802 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:57.501761    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc"
Oct 30 13:09:57.528993 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:57.528922    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr" podUID="446c12df-de12-4c7c-8c71-dfed93cc33f1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:09:57.557889 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:57.557774    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz" podUID="bc5caf69-a649-47e5-8910-9b8f331dfd4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:09:57.675378 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:57.675324    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz"
Oct 30 13:09:57.675551 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:57.675410    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4"
Oct 30 13:09:57.675551 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:57.675470    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh"
Oct 30 13:09:57.675551 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:57.675526    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr"
Oct 30 13:09:57.675783 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:57.675560    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6"
Oct 30 13:09:57.679221 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:57.679194    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f"]
Oct 30 13:09:57.687060 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:57.687037    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz"
Oct 30 13:09:57.687192 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:57.687072    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f"]
Oct 30 13:09:57.687968 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:57.687946    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f"
Oct 30 13:09:57.731282 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:57.731232    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-b31c2926-openshift-default-dockercfg-9pptm\""
Oct 30 13:09:57.827821 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:57.827785    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1acea1c-40f3-4414-af50-8281af1a3809-credential-socket\") pod \"e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f\" (UID: \"e1acea1c-40f3-4414-af50-8281af1a3809\") " pod="openshift-ingress/e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f"
Oct 30 13:09:57.828353 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:57.828327    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brn6x\" (UniqueName: \"kubernetes.io/projected/e1acea1c-40f3-4414-af50-8281af1a3809-kube-api-access-brn6x\") pod \"e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f\" (UID: \"e1acea1c-40f3-4414-af50-8281af1a3809\") " pod="openshift-ingress/e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f"
Oct 30 13:09:57.828479 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:57.828391    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1acea1c-40f3-4414-af50-8281af1a3809-istio-podinfo\") pod \"e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f\" (UID: \"e1acea1c-40f3-4414-af50-8281af1a3809\") " pod="openshift-ingress/e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f"
Oct 30 13:09:57.828575 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:57.828519    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1acea1c-40f3-4414-af50-8281af1a3809-istio-data\") pod \"e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f\" (UID: \"e1acea1c-40f3-4414-af50-8281af1a3809\") " pod="openshift-ingress/e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f"
Oct 30 13:09:57.828701 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:57.828686    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1acea1c-40f3-4414-af50-8281af1a3809-istio-token\") pod \"e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f\" (UID: \"e1acea1c-40f3-4414-af50-8281af1a3809\") " pod="openshift-ingress/e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f"
Oct 30 13:09:57.828994 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:57.828966    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1acea1c-40f3-4414-af50-8281af1a3809-workload-socket\") pod \"e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f\" (UID: \"e1acea1c-40f3-4414-af50-8281af1a3809\") " pod="openshift-ingress/e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f"
Oct 30 13:09:57.829942 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:57.829919    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1acea1c-40f3-4414-af50-8281af1a3809-istio-envoy\") pod \"e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f\" (UID: \"e1acea1c-40f3-4414-af50-8281af1a3809\") " pod="openshift-ingress/e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f"
Oct 30 13:09:57.830058 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:57.830040    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1acea1c-40f3-4414-af50-8281af1a3809-workload-certs\") pod \"e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f\" (UID: \"e1acea1c-40f3-4414-af50-8281af1a3809\") " pod="openshift-ingress/e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f"
Oct 30 13:09:57.830205 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:57.830188    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1acea1c-40f3-4414-af50-8281af1a3809-istiod-ca-cert\") pod \"e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f\" (UID: \"e1acea1c-40f3-4414-af50-8281af1a3809\") " pod="openshift-ingress/e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f"
Oct 30 13:09:57.931467 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:57.931363    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1acea1c-40f3-4414-af50-8281af1a3809-istiod-ca-cert\") pod \"e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f\" (UID: \"e1acea1c-40f3-4414-af50-8281af1a3809\") " pod="openshift-ingress/e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f"
Oct 30 13:09:57.931467 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:57.931441    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1acea1c-40f3-4414-af50-8281af1a3809-credential-socket\") pod \"e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f\" (UID: \"e1acea1c-40f3-4414-af50-8281af1a3809\") " pod="openshift-ingress/e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f"
Oct 30 13:09:57.931770 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:57.931524    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brn6x\" (UniqueName: \"kubernetes.io/projected/e1acea1c-40f3-4414-af50-8281af1a3809-kube-api-access-brn6x\") pod \"e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f\" (UID: \"e1acea1c-40f3-4414-af50-8281af1a3809\") " pod="openshift-ingress/e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f"
Oct 30 13:09:57.931770 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:57.931543    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1acea1c-40f3-4414-af50-8281af1a3809-istio-podinfo\") pod \"e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f\" (UID: \"e1acea1c-40f3-4414-af50-8281af1a3809\") " pod="openshift-ingress/e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f"
Oct 30 13:09:57.931770 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:57.931584    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1acea1c-40f3-4414-af50-8281af1a3809-istio-data\") pod \"e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f\" (UID: \"e1acea1c-40f3-4414-af50-8281af1a3809\") " pod="openshift-ingress/e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f"
Oct 30 13:09:57.931770 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:57.931716    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1acea1c-40f3-4414-af50-8281af1a3809-istio-token\") pod \"e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f\" (UID: \"e1acea1c-40f3-4414-af50-8281af1a3809\") " pod="openshift-ingress/e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f"
Oct 30 13:09:57.932428 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:57.931785    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1acea1c-40f3-4414-af50-8281af1a3809-workload-socket\") pod \"e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f\" (UID: \"e1acea1c-40f3-4414-af50-8281af1a3809\") " pod="openshift-ingress/e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f"
Oct 30 13:09:57.932428 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:57.931888    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1acea1c-40f3-4414-af50-8281af1a3809-istio-envoy\") pod \"e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f\" (UID: \"e1acea1c-40f3-4414-af50-8281af1a3809\") " pod="openshift-ingress/e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f"
Oct 30 13:09:57.932428 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:57.931949    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1acea1c-40f3-4414-af50-8281af1a3809-workload-certs\") pod \"e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f\" (UID: \"e1acea1c-40f3-4414-af50-8281af1a3809\") " pod="openshift-ingress/e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f"
Oct 30 13:09:57.932428 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:57.932209    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1acea1c-40f3-4414-af50-8281af1a3809-istiod-ca-cert\") pod \"e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f\" (UID: \"e1acea1c-40f3-4414-af50-8281af1a3809\") " pod="openshift-ingress/e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f"
Oct 30 13:09:57.932428 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:57.932356    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1acea1c-40f3-4414-af50-8281af1a3809-workload-certs\") pod \"e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f\" (UID: \"e1acea1c-40f3-4414-af50-8281af1a3809\") " pod="openshift-ingress/e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f"
Oct 30 13:09:57.932749 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:57.932504    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1acea1c-40f3-4414-af50-8281af1a3809-istio-data\") pod \"e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f\" (UID: \"e1acea1c-40f3-4414-af50-8281af1a3809\") " pod="openshift-ingress/e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f"
Oct 30 13:09:57.932866 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:57.932843    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1acea1c-40f3-4414-af50-8281af1a3809-workload-socket\") pod \"e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f\" (UID: \"e1acea1c-40f3-4414-af50-8281af1a3809\") " pod="openshift-ingress/e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f"
Oct 30 13:09:57.933200 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:57.933163    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1acea1c-40f3-4414-af50-8281af1a3809-credential-socket\") pod \"e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f\" (UID: \"e1acea1c-40f3-4414-af50-8281af1a3809\") " pod="openshift-ingress/e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f"
Oct 30 13:09:57.935069 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:57.935040    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1acea1c-40f3-4414-af50-8281af1a3809-istio-podinfo\") pod \"e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f\" (UID: \"e1acea1c-40f3-4414-af50-8281af1a3809\") " pod="openshift-ingress/e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f"
Oct 30 13:09:57.935397 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:57.935373    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1acea1c-40f3-4414-af50-8281af1a3809-istio-envoy\") pod \"e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f\" (UID: \"e1acea1c-40f3-4414-af50-8281af1a3809\") " pod="openshift-ingress/e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f"
Oct 30 13:09:57.960201 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:57.960171    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1acea1c-40f3-4414-af50-8281af1a3809-istio-token\") pod \"e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f\" (UID: \"e1acea1c-40f3-4414-af50-8281af1a3809\") " pod="openshift-ingress/e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f"
Oct 30 13:09:57.964814 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:57.964791    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brn6x\" (UniqueName: \"kubernetes.io/projected/e1acea1c-40f3-4414-af50-8281af1a3809-kube-api-access-brn6x\") pod \"e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f\" (UID: \"e1acea1c-40f3-4414-af50-8281af1a3809\") " pod="openshift-ingress/e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f"
Oct 30 13:09:57.984917 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:09:57.984884    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ee5a4af_eb64_470e_a42e_7d334cedce2d.slice/crio-6b0dab6b207a353230cc0c964fa2ace1ac1526bbd0208e268381f1484345b43f WatchSource:0}: Error finding container 6b0dab6b207a353230cc0c964fa2ace1ac1526bbd0208e268381f1484345b43f: Status 404 returned error can't find the container with id 6b0dab6b207a353230cc0c964fa2ace1ac1526bbd0208e268381f1484345b43f
Oct 30 13:09:57.986344 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:57.986311    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5"]
Oct 30 13:09:57.988151 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:57.988121    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:09:57.988225 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:57.988189    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:09:57.988276 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:57.988223    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:09:58.011223 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:58.011033    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f"
Oct 30 13:09:58.374655 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:58.374330    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f"]
Oct 30 13:09:58.389715 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:58.389657    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:09:58.389848 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:58.389745    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:09:58.389848 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:58.389787    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:09:58.704731 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:58.704450    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f" event={"ID":"e1acea1c-40f3-4414-af50-8281af1a3809","Type":"ContainerStarted","Data":"124e5d84e38fa49aa3459495b6a769c8f3f524823289aa598a634f59a58386e0"}
Oct 30 13:09:58.704731 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:58.704518    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f" event={"ID":"e1acea1c-40f3-4414-af50-8281af1a3809","Type":"ContainerStarted","Data":"169ace6c94b9365da9a880ae485d92672cd8d5c9b5bf5f67688a6cd7269458d9"}
Oct 30 13:09:58.708689 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:58.708192    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5" event={"ID":"8ee5a4af-eb64-470e-a42e-7d334cedce2d","Type":"ContainerStarted","Data":"6eb1fee14b03910c6dca350f0c270f6ff32beab86fbd388efc10ab386fca21bd"}
Oct 30 13:09:58.708689 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:58.708247    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5" event={"ID":"8ee5a4af-eb64-470e-a42e-7d334cedce2d","Type":"ContainerStarted","Data":"6b0dab6b207a353230cc0c964fa2ace1ac1526bbd0208e268381f1484345b43f"}
Oct 30 13:09:58.755278 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:58.755229    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f" podStartSLOduration=3.755214155 podStartE2EDuration="3.755214155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:58.752422639 +0000 UTC m=+1097.918787287" watchObservedRunningTime="2026-10-30 13:09:58.755214155 +0000 UTC m=+1097.921578875"
Oct 30 13:09:58.802432 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:58.802391    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5" podStartSLOduration=5.802378071 podStartE2EDuration="5.802378071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:58.800841688 +0000 UTC m=+1097.967206336" watchObservedRunningTime="2026-10-30 13:09:58.802378071 +0000 UTC m=+1097.968742720"
Oct 30 13:09:59.014723 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:59.013158    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f"
Oct 30 13:09:59.014723 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:59.013270    4364 patch_prober.go:29] interesting pod/e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 30 13:09:59.014723 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:59.013315    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f" podUID="e1acea1c-40f3-4414-af50-8281af1a3809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 30 13:09:59.411223 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:59.411184    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5"
Oct 30 13:09:59.412356 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:59.412330    4364 patch_prober.go:29] interesting pod/e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 13:09:59.412455 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:09:59.412380    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5" podUID="8ee5a4af-eb64-470e-a42e-7d334cedce2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 13:10:00.015873 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:00.015841    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f"
Oct 30 13:10:00.413053 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:00.413010    4364 patch_prober.go:29] interesting pod/e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 13:10:00.413465 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:00.413075    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5" podUID="8ee5a4af-eb64-470e-a42e-7d334cedce2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 13:10:00.727134 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:00.727046    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f"
Oct 30 13:10:00.727965 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:00.727946    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f"
Oct 30 13:10:01.411898 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:01.411863    4364 patch_prober.go:29] interesting pod/e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 13:10:01.412076 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:01.411912    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5" podUID="8ee5a4af-eb64-470e-a42e-7d334cedce2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 13:10:02.375818 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:02.375787    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b31c2926-6b889ddf99-7dt86"]
Oct 30 13:10:02.400335 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:02.400307    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b31c2926-6b889ddf99-7dt86"
Oct 30 13:10:02.411439 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:02.411409    4364 patch_prober.go:29] interesting pod/e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 13:10:02.411618 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:02.411474    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5" podUID="8ee5a4af-eb64-470e-a42e-7d334cedce2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 13:10:02.411763 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:02.411744    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b31c2926-6b889ddf99-7dt86"]
Oct 30 13:10:02.424359 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:02.424329    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-b31c2926\""
Oct 30 13:10:02.438409 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:02.438382    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-b31c2926-dockercfg-49blb\""
Oct 30 13:10:02.444333 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:02.444311    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89vgm\" (UniqueName: \"kubernetes.io/projected/6ef82918-510b-4cbc-b4a0-2b9fb48c2c08-kube-api-access-89vgm\") pod \"payload-pre-processing-e2e-ait-b31c2926-6b889ddf99-7dt86\" (UID: \"6ef82918-510b-4cbc-b4a0-2b9fb48c2c08\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b31c2926-6b889ddf99-7dt86"
Oct 30 13:10:02.444426 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:02.444372    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ef82918-510b-4cbc-b4a0-2b9fb48c2c08-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b31c2926-6b889ddf99-7dt86\" (UID: \"6ef82918-510b-4cbc-b4a0-2b9fb48c2c08\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b31c2926-6b889ddf99-7dt86"
Oct 30 13:10:02.603007 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:02.602956    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89vgm\" (UniqueName: \"kubernetes.io/projected/6ef82918-510b-4cbc-b4a0-2b9fb48c2c08-kube-api-access-89vgm\") pod \"payload-pre-processing-e2e-ait-b31c2926-6b889ddf99-7dt86\" (UID: \"6ef82918-510b-4cbc-b4a0-2b9fb48c2c08\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b31c2926-6b889ddf99-7dt86"
Oct 30 13:10:02.603202 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:02.603099    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ef82918-510b-4cbc-b4a0-2b9fb48c2c08-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b31c2926-6b889ddf99-7dt86\" (UID: \"6ef82918-510b-4cbc-b4a0-2b9fb48c2c08\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b31c2926-6b889ddf99-7dt86"
Oct 30 13:10:02.605631 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:02.604201    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ef82918-510b-4cbc-b4a0-2b9fb48c2c08-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b31c2926-6b889ddf99-7dt86\" (UID: \"6ef82918-510b-4cbc-b4a0-2b9fb48c2c08\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b31c2926-6b889ddf99-7dt86"
Oct 30 13:10:02.920486 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:02.920457    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89vgm\" (UniqueName: \"kubernetes.io/projected/6ef82918-510b-4cbc-b4a0-2b9fb48c2c08-kube-api-access-89vgm\") pod \"payload-pre-processing-e2e-ait-b31c2926-6b889ddf99-7dt86\" (UID: \"6ef82918-510b-4cbc-b4a0-2b9fb48c2c08\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b31c2926-6b889ddf99-7dt86"
Oct 30 13:10:03.024190 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:03.024154    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b31c2926-6b889ddf99-7dt86"
Oct 30 13:10:03.440693 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:03.428240    4364 patch_prober.go:29] interesting pod/e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 13:10:03.440693 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:03.428408    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5" podUID="8ee5a4af-eb64-470e-a42e-7d334cedce2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 13:10:03.440693 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:03.434106    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b31c2926-6b889ddf99-7dt86"]
Oct 30 13:10:03.865382 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:03.865282    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b31c2926-6b889ddf99-7dt86" event={"ID":"6ef82918-510b-4cbc-b4a0-2b9fb48c2c08","Type":"ContainerStarted","Data":"502b8735006063dff54700b1fd5d4aca178d9998d6d0cc2067434989feb20f64"}
Oct 30 13:10:03.865382 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:03.865347    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b31c2926-6b889ddf99-7dt86" event={"ID":"6ef82918-510b-4cbc-b4a0-2b9fb48c2c08","Type":"ContainerStarted","Data":"26e98b4ef0a7956f7b8df5ed0f8a30e82c833f9a7a7ac1638ada192298bd14cc"}
Oct 30 13:10:03.865627 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:03.865492    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b31c2926-6b889ddf99-7dt86"
Oct 30 13:10:03.924128 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:03.924077    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-b31c2926-6b889ddf99-7dt86" podStartSLOduration=1.924063044 podStartE2EDuration="1.924063044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:03.919547202 +0000 UTC m=+1103.085911851" watchObservedRunningTime="2026-10-30 13:10:03.924063044 +0000 UTC m=+1103.090427692"
Oct 30 13:10:04.411080 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:04.411049    4364 patch_prober.go:29] interesting pod/e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 13:10:04.411225 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:04.411100    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5" podUID="8ee5a4af-eb64-470e-a42e-7d334cedce2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 13:10:04.726468 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:04.712666    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-7e2137bc-59bcbfbcc6-r8q9v"]
Oct 30 13:10:04.956648 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:04.953177    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-7e2137bc-59bcbfbcc6-r8q9v"]
Oct 30 13:10:05.037776 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:05.000914    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-7e2137bc-59bcbfbcc6-r8q9v"
Oct 30 13:10:05.037776 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:05.004479    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc82b255-a820-45b5-af91-a9d4f7eb824f-plugins-config-volume\") pod \"payload-processing-e2e-migrate-7e2137bc-59bcbfbcc6-r8q9v\" (UID: \"dc82b255-a820-45b5-af91-a9d4f7eb824f\") " pod="openshift-ingress/payload-processing-e2e-migrate-7e2137bc-59bcbfbcc6-r8q9v"
Oct 30 13:10:05.037776 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:05.004693    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bznt5\" (UniqueName: \"kubernetes.io/projected/dc82b255-a820-45b5-af91-a9d4f7eb824f-kube-api-access-bznt5\") pod \"payload-processing-e2e-migrate-7e2137bc-59bcbfbcc6-r8q9v\" (UID: \"dc82b255-a820-45b5-af91-a9d4f7eb824f\") " pod="openshift-ingress/payload-processing-e2e-migrate-7e2137bc-59bcbfbcc6-r8q9v"
Oct 30 13:10:05.037776 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:05.027250    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-7e2137bc\""
Oct 30 13:10:05.042063 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:05.040806    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-7e2137bc-dockercfg-6nr5j\""
Oct 30 13:10:05.048690 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:05.048666    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-d833bd/distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74"
Oct 30 13:10:05.061301 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:05.061267    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d833bd/distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74"
Oct 30 13:10:05.106449 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:05.106407    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc82b255-a820-45b5-af91-a9d4f7eb824f-plugins-config-volume\") pod \"payload-processing-e2e-migrate-7e2137bc-59bcbfbcc6-r8q9v\" (UID: \"dc82b255-a820-45b5-af91-a9d4f7eb824f\") " pod="openshift-ingress/payload-processing-e2e-migrate-7e2137bc-59bcbfbcc6-r8q9v"
Oct 30 13:10:05.106702 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:05.106679    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bznt5\" (UniqueName: \"kubernetes.io/projected/dc82b255-a820-45b5-af91-a9d4f7eb824f-kube-api-access-bznt5\") pod \"payload-processing-e2e-migrate-7e2137bc-59bcbfbcc6-r8q9v\" (UID: \"dc82b255-a820-45b5-af91-a9d4f7eb824f\") " pod="openshift-ingress/payload-processing-e2e-migrate-7e2137bc-59bcbfbcc6-r8q9v"
Oct 30 13:10:05.107256 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:05.107233    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc82b255-a820-45b5-af91-a9d4f7eb824f-plugins-config-volume\") pod \"payload-processing-e2e-migrate-7e2137bc-59bcbfbcc6-r8q9v\" (UID: \"dc82b255-a820-45b5-af91-a9d4f7eb824f\") " pod="openshift-ingress/payload-processing-e2e-migrate-7e2137bc-59bcbfbcc6-r8q9v"
Oct 30 13:10:05.131890 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:05.131865    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bznt5\" (UniqueName: \"kubernetes.io/projected/dc82b255-a820-45b5-af91-a9d4f7eb824f-kube-api-access-bznt5\") pod \"payload-processing-e2e-migrate-7e2137bc-59bcbfbcc6-r8q9v\" (UID: \"dc82b255-a820-45b5-af91-a9d4f7eb824f\") " pod="openshift-ingress/payload-processing-e2e-migrate-7e2137bc-59bcbfbcc6-r8q9v"
Oct 30 13:10:05.355441 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:05.355368    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-7e2137bc-59bcbfbcc6-r8q9v"
Oct 30 13:10:05.422810 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:05.422771    4364 patch_prober.go:29] interesting pod/e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 13:10:05.423006 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:05.422832    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5" podUID="8ee5a4af-eb64-470e-a42e-7d334cedce2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 13:10:05.844900 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:05.844866    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-7e2137bc-59bcbfbcc6-r8q9v"]
Oct 30 13:10:05.845501 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:10:05.845470    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc82b255_a820_45b5_af91_a9d4f7eb824f.slice/crio-1a557e2c91ffc05257b1e671df6ca69888ee683b4da52e3fb7d2e0d171ffeb74 WatchSource:0}: Error finding container 1a557e2c91ffc05257b1e671df6ca69888ee683b4da52e3fb7d2e0d171ffeb74: Status 404 returned error can't find the container with id 1a557e2c91ffc05257b1e671df6ca69888ee683b4da52e3fb7d2e0d171ffeb74
Oct 30 13:10:06.045396 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:06.045357    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-7e2137bc-59bcbfbcc6-r8q9v" event={"ID":"dc82b255-a820-45b5-af91-a9d4f7eb824f","Type":"ContainerStarted","Data":"9e66c526fd24fda071a4826ccbf5987ac3e2d075fdbf46f0aedce763e2718ffb"}
Oct 30 13:10:06.045725 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:06.045688    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-7e2137bc-59bcbfbcc6-r8q9v"
Oct 30 13:10:06.045725 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:06.045725    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-7e2137bc-59bcbfbcc6-r8q9v" event={"ID":"dc82b255-a820-45b5-af91-a9d4f7eb824f","Type":"ContainerStarted","Data":"1a557e2c91ffc05257b1e671df6ca69888ee683b4da52e3fb7d2e0d171ffeb74"}
Oct 30 13:10:06.094777 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:06.094673    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-7e2137bc-59bcbfbcc6-r8q9v" podStartSLOduration=2.09465668 podStartE2EDuration="2.09465668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:06.089474718 +0000 UTC m=+1105.255839379" watchObservedRunningTime="2026-10-30 13:10:06.09465668 +0000 UTC m=+1105.261021329"
Oct 30 13:10:06.602021 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:06.601989    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5"
Oct 30 13:10:06.602463 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:06.602431    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5"
Oct 30 13:10:06.603172 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:06.603154    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5"
Oct 30 13:10:11.556486 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:10:11.556450    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:10:15.039108 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:15.039079    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b31c2926-6b889ddf99-7dt86"
Oct 30 13:10:17.055298 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:17.055271    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-7e2137bc-59bcbfbcc6-r8q9v"
Oct 30 13:10:18.617541 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:18.617502    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:10:19.548472 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:19.548430    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7e2137bc-6b578c886f-bhb5f"]
Oct 30 13:10:19.554897 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:19.554874    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7e2137bc-6b578c886f-bhb5f"
Oct 30 13:10:19.577625 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:19.574744    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7e2137bc-6b578c886f-bhb5f"]
Oct 30 13:10:19.578515 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:19.578489    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-7e2137bc-serving-cert\""
Oct 30 13:10:19.638553 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:19.638518    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/715b8a02-efaa-43ea-b8f0-478fb3e11e68-metrics-certs\") pod \"maas-api-e2e-migrate-7e2137bc-6b578c886f-bhb5f\" (UID: \"715b8a02-efaa-43ea-b8f0-478fb3e11e68\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7e2137bc-6b578c886f-bhb5f"
Oct 30 13:10:19.638998 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:19.638642    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4w2x\" (UniqueName: \"kubernetes.io/projected/715b8a02-efaa-43ea-b8f0-478fb3e11e68-kube-api-access-c4w2x\") pod \"maas-api-e2e-migrate-7e2137bc-6b578c886f-bhb5f\" (UID: \"715b8a02-efaa-43ea-b8f0-478fb3e11e68\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7e2137bc-6b578c886f-bhb5f"
Oct 30 13:10:19.638998 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:19.638785    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/715b8a02-efaa-43ea-b8f0-478fb3e11e68-maas-api-tls\") pod \"maas-api-e2e-migrate-7e2137bc-6b578c886f-bhb5f\" (UID: \"715b8a02-efaa-43ea-b8f0-478fb3e11e68\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7e2137bc-6b578c886f-bhb5f"
Oct 30 13:10:19.638998 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:19.638908    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/715b8a02-efaa-43ea-b8f0-478fb3e11e68-openshift-service-ca\") pod \"maas-api-e2e-migrate-7e2137bc-6b578c886f-bhb5f\" (UID: \"715b8a02-efaa-43ea-b8f0-478fb3e11e68\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7e2137bc-6b578c886f-bhb5f"
Oct 30 13:10:19.739986 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:19.739955    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/715b8a02-efaa-43ea-b8f0-478fb3e11e68-maas-api-tls\") pod \"maas-api-e2e-migrate-7e2137bc-6b578c886f-bhb5f\" (UID: \"715b8a02-efaa-43ea-b8f0-478fb3e11e68\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7e2137bc-6b578c886f-bhb5f"
Oct 30 13:10:19.740110 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:19.740043    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/715b8a02-efaa-43ea-b8f0-478fb3e11e68-openshift-service-ca\") pod \"maas-api-e2e-migrate-7e2137bc-6b578c886f-bhb5f\" (UID: \"715b8a02-efaa-43ea-b8f0-478fb3e11e68\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7e2137bc-6b578c886f-bhb5f"
Oct 30 13:10:19.740268 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:19.740252    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/715b8a02-efaa-43ea-b8f0-478fb3e11e68-metrics-certs\") pod \"maas-api-e2e-migrate-7e2137bc-6b578c886f-bhb5f\" (UID: \"715b8a02-efaa-43ea-b8f0-478fb3e11e68\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7e2137bc-6b578c886f-bhb5f"
Oct 30 13:10:19.740349 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:19.740329    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4w2x\" (UniqueName: \"kubernetes.io/projected/715b8a02-efaa-43ea-b8f0-478fb3e11e68-kube-api-access-c4w2x\") pod \"maas-api-e2e-migrate-7e2137bc-6b578c886f-bhb5f\" (UID: \"715b8a02-efaa-43ea-b8f0-478fb3e11e68\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7e2137bc-6b578c886f-bhb5f"
Oct 30 13:10:19.740984 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:19.740953    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/715b8a02-efaa-43ea-b8f0-478fb3e11e68-openshift-service-ca\") pod \"maas-api-e2e-migrate-7e2137bc-6b578c886f-bhb5f\" (UID: \"715b8a02-efaa-43ea-b8f0-478fb3e11e68\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7e2137bc-6b578c886f-bhb5f"
Oct 30 13:10:19.743031 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:19.743001    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/715b8a02-efaa-43ea-b8f0-478fb3e11e68-maas-api-tls\") pod \"maas-api-e2e-migrate-7e2137bc-6b578c886f-bhb5f\" (UID: \"715b8a02-efaa-43ea-b8f0-478fb3e11e68\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7e2137bc-6b578c886f-bhb5f"
Oct 30 13:10:19.743128 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:19.743084    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/715b8a02-efaa-43ea-b8f0-478fb3e11e68-metrics-certs\") pod \"maas-api-e2e-migrate-7e2137bc-6b578c886f-bhb5f\" (UID: \"715b8a02-efaa-43ea-b8f0-478fb3e11e68\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7e2137bc-6b578c886f-bhb5f"
Oct 30 13:10:19.767927 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:19.767908    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4w2x\" (UniqueName: \"kubernetes.io/projected/715b8a02-efaa-43ea-b8f0-478fb3e11e68-kube-api-access-c4w2x\") pod \"maas-api-e2e-migrate-7e2137bc-6b578c886f-bhb5f\" (UID: \"715b8a02-efaa-43ea-b8f0-478fb3e11e68\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7e2137bc-6b578c886f-bhb5f"
Oct 30 13:10:19.879621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:19.877085    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7e2137bc-6b578c886f-bhb5f"
Oct 30 13:10:20.790004 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:20.781340    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7e2137bc-6b578c886f-bhb5f"]
Oct 30 13:10:21.319990 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:21.319951    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7e2137bc-6b578c886f-bhb5f" event={"ID":"715b8a02-efaa-43ea-b8f0-478fb3e11e68","Type":"ContainerStarted","Data":"9721096e73f617f288d7a17a10568103a428a5444948e88dd83d8dd48479074c"}
Oct 30 13:10:21.933310 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:21.933244    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh" podUID="d41ac9e5-eeed-4e07-a7fa-05d3260630b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:10:21.952713 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:21.936109    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr" podUID="446c12df-de12-4c7c-8c71-dfed93cc33f1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:10:22.057953 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:22.057907    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74" podUID="4f2b06ac-9bf8-4b57-bd22-85532576f9c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:10:22.135788 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:22.135741    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc" podUID="8911b790-331d-490c-876b-647e6d2fcd14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:10:22.308836 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:22.308788    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz" podUID="a568f1ce-602d-4069-a41c-9fe1a7635c5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:10:22.440707 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:22.440667    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7e2137bc-6b578c886f-bhb5f" event={"ID":"715b8a02-efaa-43ea-b8f0-478fb3e11e68","Type":"ContainerStarted","Data":"8414ab250aad0980bee5ed5a6b18175164cdac6bc0887c8a48a1d348af660532"}
Oct 30 13:10:22.442260 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:22.442234    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7e2137bc-6b578c886f-bhb5f"
Oct 30 13:10:22.503188 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:22.503079    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7e2137bc-6b578c886f-bhb5f" podStartSLOduration=2.989439333 podStartE2EDuration="3.503058456s" totalImagesPullingTime="513.619123ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:22.500278856 +0000 UTC m=+1121.666643519" watchObservedRunningTime="2026-10-30 13:10:22.503058456 +0000 UTC m=+1121.669423107"
Oct 30 13:10:22.647020 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:22.646960    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz" podUID="bc5caf69-a649-47e5-8910-9b8f331dfd4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:10:22.697620 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:22.693692    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6" podUID="b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:10:22.735294 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:22.735250    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4" podUID="0cbe6af7-d884-4613-90b8-192e5ffdb027" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 13:10:23.076325 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:23.076277    4364 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz" podUID="a568f1ce-602d-4069-a41c-9fe1a7635c5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": context deadline exceeded"
Oct 30 13:10:23.139854 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:23.082881    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh" podUID="d41ac9e5-eeed-4e07-a7fa-05d3260630b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:10:23.165934 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:23.085045    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr" podUID="446c12df-de12-4c7c-8c71-dfed93cc33f1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded"
Oct 30 13:10:23.166787 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:23.101800    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74" podUID="4f2b06ac-9bf8-4b57-bd22-85532576f9c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded"
Oct 30 13:10:23.167214 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:23.139719    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc" podUID="8911b790-331d-490c-876b-647e6d2fcd14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:10:23.371694 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:23.351155    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz" podUID="a568f1ce-602d-4069-a41c-9fe1a7635c5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:10:23.423625 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:23.381164    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh"
Oct 30 13:10:23.425717 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:23.425691    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr"
Oct 30 13:10:23.450820 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:23.450789    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d833bd/distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74"
Oct 30 13:10:23.487873 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:23.487836    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc"
Oct 30 13:10:23.495708 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:23.488127    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz"
Oct 30 13:10:23.764061 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:23.746806    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz" podUID="bc5caf69-a649-47e5-8910-9b8f331dfd4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:10:23.764061 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:23.755939    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6" podUID="b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 13:10:23.767621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:23.764469    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4" podUID="0cbe6af7-d884-4613-90b8-192e5ffdb027" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:10:23.866641 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:23.765066    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz"
Oct 30 13:10:24.062847 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:24.062764    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6"
Oct 30 13:10:24.063133 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:24.063116    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4"
Oct 30 13:10:24.175176 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:24.175124    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr" podUID="446c12df-de12-4c7c-8c71-dfed93cc33f1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:10:24.218765 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:24.218713    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74" podUID="4f2b06ac-9bf8-4b57-bd22-85532576f9c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:10:24.220905 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:24.220858    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh" podUID="d41ac9e5-eeed-4e07-a7fa-05d3260630b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 30 13:10:24.255658 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:24.220789    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc" podUID="8911b790-331d-490c-876b-647e6d2fcd14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:10:24.386680 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:24.352224    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz" podUID="a568f1ce-602d-4069-a41c-9fe1a7635c5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:10:24.630633 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:24.627348    4364 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4" podUID="0cbe6af7-d884-4613-90b8-192e5ffdb027" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:10:24.775708 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:24.703653    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8"]
Oct 30 13:10:24.831849 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:24.831791    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6" podUID="b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:10:24.866508 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:24.866404    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4" podUID="0cbe6af7-d884-4613-90b8-192e5ffdb027" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 13:10:24.867231 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:24.867046    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8"
Oct 30 13:10:24.988956 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:24.988865    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz" podUID="bc5caf69-a649-47e5-8910-9b8f331dfd4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 13:10:25.241659 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:25.177708    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr" podUID="446c12df-de12-4c7c-8c71-dfed93cc33f1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded"
Oct 30 13:10:25.241659 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:25.224436    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh" podUID="d41ac9e5-eeed-4e07-a7fa-05d3260630b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:10:25.241659 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:25.224865    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74" podUID="4f2b06ac-9bf8-4b57-bd22-85532576f9c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded"
Oct 30 13:10:25.310924 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:25.310815    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc" podUID="8911b790-331d-490c-876b-647e6d2fcd14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:10:25.441625 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:25.440285    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz" podUID="a568f1ce-602d-4069-a41c-9fe1a7635c5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:10:25.492623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:25.490810    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8"]
Oct 30 13:10:25.492623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:25.491529    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe89b812-7b37-4eeb-8a92-07c436511948-workload-socket\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8\" (UID: \"fe89b812-7b37-4eeb-8a92-07c436511948\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8"
Oct 30 13:10:25.492623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:25.491565    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe89b812-7b37-4eeb-8a92-07c436511948-istio-token\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8\" (UID: \"fe89b812-7b37-4eeb-8a92-07c436511948\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8"
Oct 30 13:10:25.543653 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:25.517207    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe89b812-7b37-4eeb-8a92-07c436511948-istiod-ca-cert\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8\" (UID: \"fe89b812-7b37-4eeb-8a92-07c436511948\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8"
Oct 30 13:10:25.543653 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:25.517339    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe89b812-7b37-4eeb-8a92-07c436511948-istio-envoy\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8\" (UID: \"fe89b812-7b37-4eeb-8a92-07c436511948\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8"
Oct 30 13:10:25.543653 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:25.517509    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe89b812-7b37-4eeb-8a92-07c436511948-istio-data\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8\" (UID: \"fe89b812-7b37-4eeb-8a92-07c436511948\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8"
Oct 30 13:10:25.543653 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:25.517618    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe89b812-7b37-4eeb-8a92-07c436511948-credential-socket\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8\" (UID: \"fe89b812-7b37-4eeb-8a92-07c436511948\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8"
Oct 30 13:10:25.543653 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:25.517751    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe89b812-7b37-4eeb-8a92-07c436511948-istio-podinfo\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8\" (UID: \"fe89b812-7b37-4eeb-8a92-07c436511948\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8"
Oct 30 13:10:25.543653 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:25.517813    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfvss\" (UniqueName: \"kubernetes.io/projected/fe89b812-7b37-4eeb-8a92-07c436511948-kube-api-access-sfvss\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8\" (UID: \"fe89b812-7b37-4eeb-8a92-07c436511948\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8"
Oct 30 13:10:25.543653 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:25.517844    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe89b812-7b37-4eeb-8a92-07c436511948-workload-certs\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8\" (UID: \"fe89b812-7b37-4eeb-8a92-07c436511948\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8"
Oct 30 13:10:25.650750 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:25.620737    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe89b812-7b37-4eeb-8a92-07c436511948-istio-data\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8\" (UID: \"fe89b812-7b37-4eeb-8a92-07c436511948\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8"
Oct 30 13:10:25.651116 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:25.651084    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe89b812-7b37-4eeb-8a92-07c436511948-credential-socket\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8\" (UID: \"fe89b812-7b37-4eeb-8a92-07c436511948\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8"
Oct 30 13:10:25.657619 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:25.651364    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe89b812-7b37-4eeb-8a92-07c436511948-istio-podinfo\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8\" (UID: \"fe89b812-7b37-4eeb-8a92-07c436511948\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8"
Oct 30 13:10:25.657843 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:25.657817    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfvss\" (UniqueName: \"kubernetes.io/projected/fe89b812-7b37-4eeb-8a92-07c436511948-kube-api-access-sfvss\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8\" (UID: \"fe89b812-7b37-4eeb-8a92-07c436511948\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8"
Oct 30 13:10:25.657981 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:25.657964    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe89b812-7b37-4eeb-8a92-07c436511948-workload-certs\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8\" (UID: \"fe89b812-7b37-4eeb-8a92-07c436511948\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8"
Oct 30 13:10:25.683622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:25.658392    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe89b812-7b37-4eeb-8a92-07c436511948-workload-socket\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8\" (UID: \"fe89b812-7b37-4eeb-8a92-07c436511948\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8"
Oct 30 13:10:25.683872 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:25.683843    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe89b812-7b37-4eeb-8a92-07c436511948-istio-token\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8\" (UID: \"fe89b812-7b37-4eeb-8a92-07c436511948\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8"
Oct 30 13:10:25.684000 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:25.683982    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe89b812-7b37-4eeb-8a92-07c436511948-istiod-ca-cert\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8\" (UID: \"fe89b812-7b37-4eeb-8a92-07c436511948\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8"
Oct 30 13:10:25.684155 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:25.684136    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe89b812-7b37-4eeb-8a92-07c436511948-istio-envoy\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8\" (UID: \"fe89b812-7b37-4eeb-8a92-07c436511948\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8"
Oct 30 13:10:25.694625 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:25.686450    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe89b812-7b37-4eeb-8a92-07c436511948-workload-socket\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8\" (UID: \"fe89b812-7b37-4eeb-8a92-07c436511948\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8"
Oct 30 13:10:25.711963 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:25.624042    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe89b812-7b37-4eeb-8a92-07c436511948-istio-data\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8\" (UID: \"fe89b812-7b37-4eeb-8a92-07c436511948\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8"
Oct 30 13:10:25.712141 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:25.695073    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe89b812-7b37-4eeb-8a92-07c436511948-credential-socket\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8\" (UID: \"fe89b812-7b37-4eeb-8a92-07c436511948\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8"
Oct 30 13:10:25.712266 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:25.695313    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe89b812-7b37-4eeb-8a92-07c436511948-workload-certs\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8\" (UID: \"fe89b812-7b37-4eeb-8a92-07c436511948\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8"
Oct 30 13:10:25.795699 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:25.697226    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe89b812-7b37-4eeb-8a92-07c436511948-istiod-ca-cert\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8\" (UID: \"fe89b812-7b37-4eeb-8a92-07c436511948\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8"
Oct 30 13:10:25.822629 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:25.796941    4364 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-d833bd/distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74" podUID="4f2b06ac-9bf8-4b57-bd22-85532576f9c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:10:25.835799 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:25.835757    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6" podUID="b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:10:25.863078 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:25.863047    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe89b812-7b37-4eeb-8a92-07c436511948-istio-envoy\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8\" (UID: \"fe89b812-7b37-4eeb-8a92-07c436511948\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8"
Oct 30 13:10:25.875529 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:25.875484    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4" podUID="0cbe6af7-d884-4613-90b8-192e5ffdb027" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:10:25.964700 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:25.933501    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfvss\" (UniqueName: \"kubernetes.io/projected/fe89b812-7b37-4eeb-8a92-07c436511948-kube-api-access-sfvss\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8\" (UID: \"fe89b812-7b37-4eeb-8a92-07c436511948\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8"
Oct 30 13:10:26.288684 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:25.980318    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe89b812-7b37-4eeb-8a92-07c436511948-istio-podinfo\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8\" (UID: \"fe89b812-7b37-4eeb-8a92-07c436511948\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8"
Oct 30 13:10:26.288684 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:26.264861    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74" podUID="4f2b06ac-9bf8-4b57-bd22-85532576f9c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:10:26.364303 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:26.364253    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh" podUID="d41ac9e5-eeed-4e07-a7fa-05d3260630b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:10:26.365524 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:26.365490    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz" podUID="bc5caf69-a649-47e5-8910-9b8f331dfd4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 13:10:26.366273 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:26.366236    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc" podUID="8911b790-331d-490c-876b-647e6d2fcd14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:10:26.455977 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:26.455018    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe89b812-7b37-4eeb-8a92-07c436511948-istio-token\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8\" (UID: \"fe89b812-7b37-4eeb-8a92-07c436511948\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8"
Oct 30 13:10:26.455977 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:26.455120    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr" podUID="446c12df-de12-4c7c-8c71-dfed93cc33f1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:10:26.503846 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:26.503792    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz" podUID="a568f1ce-602d-4069-a41c-9fe1a7635c5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:10:27.316622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:26.685197    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8"
Oct 30 13:10:27.316622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:26.871946    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6" podUID="b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:10:27.316622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:27.016646    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4" podUID="0cbe6af7-d884-4613-90b8-192e5ffdb027" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 13:10:27.316622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:27.276706    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74" podUID="4f2b06ac-9bf8-4b57-bd22-85532576f9c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:10:27.395202 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:27.341740    4364 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh" podUID="d41ac9e5-eeed-4e07-a7fa-05d3260630b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:10:27.446655 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:27.402806    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh" podUID="d41ac9e5-eeed-4e07-a7fa-05d3260630b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:10:27.446655 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:27.403384    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc" podUID="8911b790-331d-490c-876b-647e6d2fcd14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:10:27.446655 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:27.403515    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz" podUID="bc5caf69-a649-47e5-8910-9b8f331dfd4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:10:27.463117 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:27.463071    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr" podUID="446c12df-de12-4c7c-8c71-dfed93cc33f1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:10:27.507009 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:27.506973    4364 patch_prober.go:29] interesting pod/console-744d9f8b78-x4qbn container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.133.0.18:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:10:27.507400 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:27.507367    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/console-744d9f8b78-x4qbn" podUID="4f7d7ad7-a4ec-42bb-b124-8b9ac9e7b1a7" containerName="console" probeResult="failure" output="Get \"https://10.133.0.18:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:10:27.823362 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:27.823301    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-n7v9l" podUID="d3bdb4e6-148f-45c7-9fdd-1f0846e21699" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:10:27.878290 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:27.872791    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-tvhvc" podUID="fda756b1-bfd0-426a-b036-13c90355172a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:10:27.888392 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:27.888361    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4"
Oct 30 13:10:27.888666 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:27.888646    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz"
Oct 30 13:10:27.888855 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:27.888840    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d833bd/distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74"
Oct 30 13:10:27.889210 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:27.889181    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr"
Oct 30 13:10:27.889593 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:27.889582    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz"
Oct 30 13:10:27.891835 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:27.889776    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh"
Oct 30 13:10:27.891835 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:27.889946    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6"
Oct 30 13:10:27.893993 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:27.893934    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc"
Oct 30 13:10:28.050369 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:28.050301    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8"]
Oct 30 13:10:28.058386 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:10:28.058355    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe89b812_7b37_4eeb_8a92_07c436511948.slice/crio-3fcb14336bda388d96c9382db877e88e3eb3ccd6537a13bc8eb432e06bcfa1b4 WatchSource:0}: Error finding container 3fcb14336bda388d96c9382db877e88e3eb3ccd6537a13bc8eb432e06bcfa1b4: Status 404 returned error can't find the container with id 3fcb14336bda388d96c9382db877e88e3eb3ccd6537a13bc8eb432e06bcfa1b4
Oct 30 13:10:28.061662 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:28.061624    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:10:28.061774 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:28.061706    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:10:28.061774 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:28.061747    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:10:28.914880 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:28.914841    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8" event={"ID":"fe89b812-7b37-4eeb-8a92-07c436511948","Type":"ContainerStarted","Data":"3492a594c7962768b036eda766bd56685da25d669c7a51c59d6964a6c22d878b"}
Oct 30 13:10:28.915370 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:28.914909    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8" event={"ID":"fe89b812-7b37-4eeb-8a92-07c436511948","Type":"ContainerStarted","Data":"3fcb14336bda388d96c9382db877e88e3eb3ccd6537a13bc8eb432e06bcfa1b4"}
Oct 30 13:10:29.012377 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:29.012325    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8" podStartSLOduration=6.012309857 podStartE2EDuration="6.012309857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:28.99823636 +0000 UTC m=+1128.164601008" watchObservedRunningTime="2026-10-30 13:10:29.012309857 +0000 UTC m=+1128.178674504"
Oct 30 13:10:29.823176 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:29.823091    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8"
Oct 30 13:10:29.824848 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:29.824819    4364 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 13:10:29.824987 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:29.824881    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8" podUID="fe89b812-7b37-4eeb-8a92-07c436511948" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 13:10:30.840622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:30.835703    4364 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 13:10:30.840622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:30.835780    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8" podUID="fe89b812-7b37-4eeb-8a92-07c436511948" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 13:10:30.865950 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:30.865903    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7e2137bc-6b578c886f-bhb5f"
Oct 30 13:10:31.823280 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:31.823247    4364 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 13:10:31.823461 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:31.823307    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8" podUID="fe89b812-7b37-4eeb-8a92-07c436511948" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 13:10:32.822623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:32.822580    4364 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 13:10:32.823021 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:32.822656    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8" podUID="fe89b812-7b37-4eeb-8a92-07c436511948" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 13:10:33.822513 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:33.822477    4364 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 13:10:33.822728 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:33.822540    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8" podUID="fe89b812-7b37-4eeb-8a92-07c436511948" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 13:10:34.823457 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:34.823420    4364 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 13:10:34.823862 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:34.823482    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8" podUID="fe89b812-7b37-4eeb-8a92-07c436511948" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 13:10:35.832654 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:35.832620    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8"
Oct 30 13:10:35.833072 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:35.833049    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8"
Oct 30 13:10:35.833701 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:35.833680    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8"
Oct 30 13:10:36.032470 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:36.032440    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec81d790-rl9dp"]
Oct 30 13:10:36.042308 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:36.042260    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec81d790-rl9dp"]
Oct 30 13:10:36.974845 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:36.974801    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ee15a09-210f-4eea-b962-6af016052d6c" path="/var/lib/kubelet/pods/5ee15a09-210f-4eea-b962-6af016052d6c/volumes"
Oct 30 13:10:38.784953 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:38.784924    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq"]
Oct 30 13:10:38.790838 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:38.790819    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq"
Oct 30 13:10:38.819614 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:38.819570    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgwnz\" (UniqueName: \"kubernetes.io/projected/d4826806-75a8-47d6-866b-415116519d9d-kube-api-access-xgwnz\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq\" (UID: \"d4826806-75a8-47d6-866b-415116519d9d\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq"
Oct 30 13:10:38.819749 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:38.819653    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4826806-75a8-47d6-866b-415116519d9d-istio-token\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq\" (UID: \"d4826806-75a8-47d6-866b-415116519d9d\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq"
Oct 30 13:10:38.819749 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:38.819728    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4826806-75a8-47d6-866b-415116519d9d-istio-envoy\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq\" (UID: \"d4826806-75a8-47d6-866b-415116519d9d\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq"
Oct 30 13:10:38.819878 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:38.819800    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4826806-75a8-47d6-866b-415116519d9d-workload-certs\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq\" (UID: \"d4826806-75a8-47d6-866b-415116519d9d\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq"
Oct 30 13:10:38.819878 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:38.819846    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4826806-75a8-47d6-866b-415116519d9d-istiod-ca-cert\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq\" (UID: \"d4826806-75a8-47d6-866b-415116519d9d\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq"
Oct 30 13:10:38.819993 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:38.819906    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4826806-75a8-47d6-866b-415116519d9d-credential-socket\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq\" (UID: \"d4826806-75a8-47d6-866b-415116519d9d\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq"
Oct 30 13:10:38.819993 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:38.819940    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4826806-75a8-47d6-866b-415116519d9d-istio-data\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq\" (UID: \"d4826806-75a8-47d6-866b-415116519d9d\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq"
Oct 30 13:10:38.820105 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:38.820002    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4826806-75a8-47d6-866b-415116519d9d-workload-socket\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq\" (UID: \"d4826806-75a8-47d6-866b-415116519d9d\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq"
Oct 30 13:10:38.820164 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:38.820139    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4826806-75a8-47d6-866b-415116519d9d-istio-podinfo\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq\" (UID: \"d4826806-75a8-47d6-866b-415116519d9d\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq"
Oct 30 13:10:38.823508 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:38.823483    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq"]
Oct 30 13:10:38.921289 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:38.921260    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4826806-75a8-47d6-866b-415116519d9d-istio-token\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq\" (UID: \"d4826806-75a8-47d6-866b-415116519d9d\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq"
Oct 30 13:10:38.921444 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:38.921307    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4826806-75a8-47d6-866b-415116519d9d-istio-envoy\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq\" (UID: \"d4826806-75a8-47d6-866b-415116519d9d\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq"
Oct 30 13:10:38.921444 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:38.921332    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4826806-75a8-47d6-866b-415116519d9d-workload-certs\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq\" (UID: \"d4826806-75a8-47d6-866b-415116519d9d\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq"
Oct 30 13:10:38.921444 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:38.921357    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4826806-75a8-47d6-866b-415116519d9d-istiod-ca-cert\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq\" (UID: \"d4826806-75a8-47d6-866b-415116519d9d\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq"
Oct 30 13:10:38.921444 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:38.921389    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4826806-75a8-47d6-866b-415116519d9d-credential-socket\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq\" (UID: \"d4826806-75a8-47d6-866b-415116519d9d\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq"
Oct 30 13:10:38.921444 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:38.921409    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4826806-75a8-47d6-866b-415116519d9d-istio-data\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq\" (UID: \"d4826806-75a8-47d6-866b-415116519d9d\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq"
Oct 30 13:10:38.921770 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:38.921445    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4826806-75a8-47d6-866b-415116519d9d-workload-socket\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq\" (UID: \"d4826806-75a8-47d6-866b-415116519d9d\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq"
Oct 30 13:10:38.921770 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:38.921562    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4826806-75a8-47d6-866b-415116519d9d-istio-podinfo\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq\" (UID: \"d4826806-75a8-47d6-866b-415116519d9d\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq"
Oct 30 13:10:38.921770 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:38.921711    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgwnz\" (UniqueName: \"kubernetes.io/projected/d4826806-75a8-47d6-866b-415116519d9d-kube-api-access-xgwnz\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq\" (UID: \"d4826806-75a8-47d6-866b-415116519d9d\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq"
Oct 30 13:10:38.922026 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:38.922002    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4826806-75a8-47d6-866b-415116519d9d-workload-certs\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq\" (UID: \"d4826806-75a8-47d6-866b-415116519d9d\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq"
Oct 30 13:10:38.922138 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:38.922062    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4826806-75a8-47d6-866b-415116519d9d-istio-data\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq\" (UID: \"d4826806-75a8-47d6-866b-415116519d9d\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq"
Oct 30 13:10:38.922213 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:38.922170    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4826806-75a8-47d6-866b-415116519d9d-workload-socket\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq\" (UID: \"d4826806-75a8-47d6-866b-415116519d9d\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq"
Oct 30 13:10:38.922326 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:38.922302    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4826806-75a8-47d6-866b-415116519d9d-istiod-ca-cert\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq\" (UID: \"d4826806-75a8-47d6-866b-415116519d9d\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq"
Oct 30 13:10:38.922416 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:38.922378    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4826806-75a8-47d6-866b-415116519d9d-credential-socket\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq\" (UID: \"d4826806-75a8-47d6-866b-415116519d9d\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq"
Oct 30 13:10:38.923826 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:38.923803    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4826806-75a8-47d6-866b-415116519d9d-istio-envoy\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq\" (UID: \"d4826806-75a8-47d6-866b-415116519d9d\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq"
Oct 30 13:10:38.924320 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:38.924302    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4826806-75a8-47d6-866b-415116519d9d-istio-podinfo\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq\" (UID: \"d4826806-75a8-47d6-866b-415116519d9d\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq"
Oct 30 13:10:38.946395 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:38.946369    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4826806-75a8-47d6-866b-415116519d9d-istio-token\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq\" (UID: \"d4826806-75a8-47d6-866b-415116519d9d\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq"
Oct 30 13:10:38.946526 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:38.946509    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgwnz\" (UniqueName: \"kubernetes.io/projected/d4826806-75a8-47d6-866b-415116519d9d-kube-api-access-xgwnz\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq\" (UID: \"d4826806-75a8-47d6-866b-415116519d9d\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq"
Oct 30 13:10:39.105808 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:39.105743    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq"
Oct 30 13:10:39.427324 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:10:39.427295    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4826806_75a8_47d6_866b_415116519d9d.slice/crio-e693417bc6d5c9edf796d38661cf128b9edf5229d0a252f30ad3906a71e89847 WatchSource:0}: Error finding container e693417bc6d5c9edf796d38661cf128b9edf5229d0a252f30ad3906a71e89847: Status 404 returned error can't find the container with id e693417bc6d5c9edf796d38661cf128b9edf5229d0a252f30ad3906a71e89847
Oct 30 13:10:39.428832 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:39.428810    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq"]
Oct 30 13:10:39.429970 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:39.429940    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:10:39.430039 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:39.430003    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:10:39.430039 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:39.430031    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:10:40.023982 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:40.023943    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq" event={"ID":"d4826806-75a8-47d6-866b-415116519d9d","Type":"ContainerStarted","Data":"1b14ca834bdf6d9785e98377ab7a04e200530a0de03e2c2a2622fc357ba80a6d"}
Oct 30 13:10:40.024394 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:40.024012    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq" event={"ID":"d4826806-75a8-47d6-866b-415116519d9d","Type":"ContainerStarted","Data":"e693417bc6d5c9edf796d38661cf128b9edf5229d0a252f30ad3906a71e89847"}
Oct 30 13:10:40.104622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:40.104095    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq" podStartSLOduration=2.104076934 podStartE2EDuration="2.104076934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:40.088740012 +0000 UTC m=+1139.255104661" watchObservedRunningTime="2026-10-30 13:10:40.104076934 +0000 UTC m=+1139.270441583"
Oct 30 13:10:40.106713 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:40.106669    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq"
Oct 30 13:10:40.108122 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:40.108085    4364 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 30 13:10:40.108229 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:40.108142    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq" podUID="d4826806-75a8-47d6-866b-415116519d9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 30 13:10:41.106165 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:41.106129    4364 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 30 13:10:41.106534 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:41.106222    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq" podUID="d4826806-75a8-47d6-866b-415116519d9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 30 13:10:41.528681 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:10:41.528576    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:10:42.107027 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:42.106993    4364 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 30 13:10:42.107453 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:42.107047    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq" podUID="d4826806-75a8-47d6-866b-415116519d9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 30 13:10:43.106115 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:43.106073    4364 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 30 13:10:43.106312 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:43.106138    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq" podUID="d4826806-75a8-47d6-866b-415116519d9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 30 13:10:44.123691 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:44.116999    4364 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 30 13:10:44.123691 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:44.117066    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq" podUID="d4826806-75a8-47d6-866b-415116519d9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 30 13:10:45.107681 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:45.107645    4364 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 30 13:10:45.107873 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:45.107706    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq" podUID="d4826806-75a8-47d6-866b-415116519d9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 30 13:10:46.033778 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:46.033741    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6192dbad-6mrgv"]
Oct 30 13:10:46.047986 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:46.047955    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6192dbad-6mrgv"]
Oct 30 13:10:46.106690 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:46.106656    4364 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 30 13:10:46.106827 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:46.106711    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq" podUID="d4826806-75a8-47d6-866b-415116519d9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 30 13:10:46.972403 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:46.972359    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcbcc50f-e0b1-4260-9f8e-73822649fd7f" path="/var/lib/kubelet/pods/dcbcc50f-e0b1-4260-9f8e-73822649fd7f/volumes"
Oct 30 13:10:47.111306 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:47.111277    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq"
Oct 30 13:10:47.111716 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:47.111693    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq"
Oct 30 13:10:47.112330 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:47.112312    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq"
Oct 30 13:10:50.037080 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:50.037047    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbd8b596-j22lj"]
Oct 30 13:10:50.050373 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:50.050343    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbd8b596-j22lj"]
Oct 30 13:10:50.972378 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:50.972345    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c25ebdb3-1e2a-465b-9afe-68a89715f718" path="/var/lib/kubelet/pods/c25ebdb3-1e2a-465b-9afe-68a89715f718/volumes"
Oct 30 13:10:54.797625 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:54.760742    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh"]
Oct 30 13:10:55.030630 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:55.030062    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4" podUID="0cbe6af7-d884-4613-90b8-192e5ffdb027" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:10:55.114674 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:55.111337    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh"
Oct 30 13:10:55.190636 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:55.185995    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr" podUID="446c12df-de12-4c7c-8c71-dfed93cc33f1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:10:55.240620 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:55.223702    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d369e1-ee25-47fe-a808-cb0470669962-workload-socket\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh\" (UID: \"42d369e1-ee25-47fe-a808-cb0470669962\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh"
Oct 30 13:10:55.240620 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:55.223804    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8zz9\" (UniqueName: \"kubernetes.io/projected/42d369e1-ee25-47fe-a808-cb0470669962-kube-api-access-p8zz9\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh\" (UID: \"42d369e1-ee25-47fe-a808-cb0470669962\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh"
Oct 30 13:10:55.240620 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:55.223962    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42d369e1-ee25-47fe-a808-cb0470669962-istiod-ca-cert\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh\" (UID: \"42d369e1-ee25-47fe-a808-cb0470669962\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh"
Oct 30 13:10:55.240620 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:55.223994    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42d369e1-ee25-47fe-a808-cb0470669962-workload-certs\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh\" (UID: \"42d369e1-ee25-47fe-a808-cb0470669962\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh"
Oct 30 13:10:55.240620 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:55.224032    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42d369e1-ee25-47fe-a808-cb0470669962-istio-podinfo\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh\" (UID: \"42d369e1-ee25-47fe-a808-cb0470669962\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh"
Oct 30 13:10:55.240620 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:55.224285    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42d369e1-ee25-47fe-a808-cb0470669962-istio-envoy\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh\" (UID: \"42d369e1-ee25-47fe-a808-cb0470669962\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh"
Oct 30 13:10:55.240620 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:55.224375    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d369e1-ee25-47fe-a808-cb0470669962-credential-socket\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh\" (UID: \"42d369e1-ee25-47fe-a808-cb0470669962\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh"
Oct 30 13:10:55.240620 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:55.224423    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42d369e1-ee25-47fe-a808-cb0470669962-istio-data\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh\" (UID: \"42d369e1-ee25-47fe-a808-cb0470669962\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh"
Oct 30 13:10:55.240620 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:55.224461    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42d369e1-ee25-47fe-a808-cb0470669962-istio-token\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh\" (UID: \"42d369e1-ee25-47fe-a808-cb0470669962\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh"
Oct 30 13:10:55.328700 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:55.328666    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d369e1-ee25-47fe-a808-cb0470669962-workload-socket\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh\" (UID: \"42d369e1-ee25-47fe-a808-cb0470669962\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh"
Oct 30 13:10:55.328981 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:55.328952    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8zz9\" (UniqueName: \"kubernetes.io/projected/42d369e1-ee25-47fe-a808-cb0470669962-kube-api-access-p8zz9\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh\" (UID: \"42d369e1-ee25-47fe-a808-cb0470669962\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh"
Oct 30 13:10:55.329280 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:55.329258    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42d369e1-ee25-47fe-a808-cb0470669962-istiod-ca-cert\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh\" (UID: \"42d369e1-ee25-47fe-a808-cb0470669962\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh"
Oct 30 13:10:55.329426 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:55.329409    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42d369e1-ee25-47fe-a808-cb0470669962-workload-certs\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh\" (UID: \"42d369e1-ee25-47fe-a808-cb0470669962\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh"
Oct 30 13:10:55.329550 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:55.329533    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42d369e1-ee25-47fe-a808-cb0470669962-istio-podinfo\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh\" (UID: \"42d369e1-ee25-47fe-a808-cb0470669962\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh"
Oct 30 13:10:55.349056 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:55.349024    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42d369e1-ee25-47fe-a808-cb0470669962-istio-envoy\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh\" (UID: \"42d369e1-ee25-47fe-a808-cb0470669962\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh"
Oct 30 13:10:55.349292 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:55.349270    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d369e1-ee25-47fe-a808-cb0470669962-credential-socket\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh\" (UID: \"42d369e1-ee25-47fe-a808-cb0470669962\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh"
Oct 30 13:10:55.349467 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:55.349448    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42d369e1-ee25-47fe-a808-cb0470669962-istio-data\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh\" (UID: \"42d369e1-ee25-47fe-a808-cb0470669962\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh"
Oct 30 13:10:55.349626 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:55.349589    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42d369e1-ee25-47fe-a808-cb0470669962-istio-token\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh\" (UID: \"42d369e1-ee25-47fe-a808-cb0470669962\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh"
Oct 30 13:10:55.350133 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:55.336288    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d369e1-ee25-47fe-a808-cb0470669962-workload-socket\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh\" (UID: \"42d369e1-ee25-47fe-a808-cb0470669962\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh"
Oct 30 13:10:55.350922 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:55.350899    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d369e1-ee25-47fe-a808-cb0470669962-credential-socket\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh\" (UID: \"42d369e1-ee25-47fe-a808-cb0470669962\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh"
Oct 30 13:10:55.378671 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:55.336958    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42d369e1-ee25-47fe-a808-cb0470669962-istiod-ca-cert\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh\" (UID: \"42d369e1-ee25-47fe-a808-cb0470669962\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh"
Oct 30 13:10:55.378671 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:55.337206    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42d369e1-ee25-47fe-a808-cb0470669962-workload-certs\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh\" (UID: \"42d369e1-ee25-47fe-a808-cb0470669962\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh"
Oct 30 13:10:55.378671 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:55.372386    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh"]
Oct 30 13:10:55.378671 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:55.374462    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42d369e1-ee25-47fe-a808-cb0470669962-istio-data\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh\" (UID: \"42d369e1-ee25-47fe-a808-cb0470669962\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh"
Oct 30 13:10:55.402620 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:55.391272    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42d369e1-ee25-47fe-a808-cb0470669962-istio-token\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh\" (UID: \"42d369e1-ee25-47fe-a808-cb0470669962\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh"
Oct 30 13:10:55.402620 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:55.400239    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8zz9\" (UniqueName: \"kubernetes.io/projected/42d369e1-ee25-47fe-a808-cb0470669962-kube-api-access-p8zz9\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh\" (UID: \"42d369e1-ee25-47fe-a808-cb0470669962\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh"
Oct 30 13:10:55.406618 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:55.405175    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42d369e1-ee25-47fe-a808-cb0470669962-istio-podinfo\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh\" (UID: \"42d369e1-ee25-47fe-a808-cb0470669962\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh"
Oct 30 13:10:55.422649 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:55.422005    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz" podUID="a568f1ce-602d-4069-a41c-9fe1a7635c5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:10:55.423388 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:55.422837    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh" podUID="d41ac9e5-eeed-4e07-a7fa-05d3260630b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 30 13:10:55.439627 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:55.435784    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42d369e1-ee25-47fe-a808-cb0470669962-istio-envoy\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh\" (UID: \"42d369e1-ee25-47fe-a808-cb0470669962\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh"
Oct 30 13:10:55.646692 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:55.646578    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh"
Oct 30 13:10:56.025653 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:56.025535    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74" podUID="4f2b06ac-9bf8-4b57-bd22-85532576f9c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded"
Oct 30 13:10:56.073698 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:56.050590    4364 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-d833bd/distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74" podUID="4f2b06ac-9bf8-4b57-bd22-85532576f9c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/health\": context deadline exceeded"
Oct 30 13:10:56.073698 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:56.039412    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz" podUID="bc5caf69-a649-47e5-8910-9b8f331dfd4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:10:56.073698 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:56.054411    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6" podUID="b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:10:56.073698 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:56.068573    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4" podUID="0cbe6af7-d884-4613-90b8-192e5ffdb027" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:10:56.073698 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:56.071641    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4"
Oct 30 13:10:56.181619 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:56.175844    4364 patch_prober.go:29] interesting pod/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:10:56.181619 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:56.175933    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc" podUID="4710cc0d-1402-489e-826b-5c44a8b6f9f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:10:56.233843 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:56.233789    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr" podUID="446c12df-de12-4c7c-8c71-dfed93cc33f1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded"
Oct 30 13:10:56.234636 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:56.234413    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr"
Oct 30 13:10:56.352089 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:56.352027    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4"
Oct 30 13:10:56.372859 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:56.372839    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr"
Oct 30 13:10:56.814091 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:10:56.814048    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42d369e1_ee25_47fe_a808_cb0470669962.slice/crio-8b49bab2db54d09ac276b584cb1b2521217e73cfefbfc23054772227b6f55661 WatchSource:0}: Error finding container 8b49bab2db54d09ac276b584cb1b2521217e73cfefbfc23054772227b6f55661: Status 404 returned error can't find the container with id 8b49bab2db54d09ac276b584cb1b2521217e73cfefbfc23054772227b6f55661
Oct 30 13:10:56.814691 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:56.814654    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh"]
Oct 30 13:10:56.817045 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:56.817014    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:10:56.817152 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:56.817078    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:10:56.817152 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:56.817110    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:10:57.432822 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:57.432778    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh" event={"ID":"42d369e1-ee25-47fe-a808-cb0470669962","Type":"ContainerStarted","Data":"835dfbab0f094930fb9b622678c7abd6d9ac1a0d33b57275fd0ec8599ad319d5"}
Oct 30 13:10:57.433262 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:57.432872    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh" event={"ID":"42d369e1-ee25-47fe-a808-cb0470669962","Type":"ContainerStarted","Data":"8b49bab2db54d09ac276b584cb1b2521217e73cfefbfc23054772227b6f55661"}
Oct 30 13:10:57.541568 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:57.541509    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh" podStartSLOduration=3.54149003 podStartE2EDuration="3.54149003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:57.508508735 +0000 UTC m=+1156.674873396" watchObservedRunningTime="2026-10-30 13:10:57.54149003 +0000 UTC m=+1156.707854686"
Oct 30 13:10:57.638742 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:57.638394    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b31c2926-6b889ddf99-7dt86"]
Oct 30 13:10:57.647990 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:57.647958    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh"
Oct 30 13:10:57.649062 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:57.649037    4364 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 13:10:57.649174 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:57.649089    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh" podUID="42d369e1-ee25-47fe-a808-cb0470669962" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 13:10:58.652621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:58.648162    4364 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 13:10:58.652621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:58.648265    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh" podUID="42d369e1-ee25-47fe-a808-cb0470669962" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 13:10:58.769883 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:58.769828    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-7e2137bc-59bcbfbcc6-r8q9v"]
Oct 30 13:10:59.651013 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:59.650974    4364 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 13:10:59.651308 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:10:59.651271    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh" podUID="42d369e1-ee25-47fe-a808-cb0470669962" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 13:11:00.647287 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:00.647251    4364 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 13:11:00.647792 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:00.647299    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh" podUID="42d369e1-ee25-47fe-a808-cb0470669962" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 13:11:01.647454 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:01.647418    4364 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 13:11:01.647893 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:01.647468    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh" podUID="42d369e1-ee25-47fe-a808-cb0470669962" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 13:11:01.721425 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:01.721323    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85"]
Oct 30 13:11:01.729314 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:01.727440    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85"
Oct 30 13:11:01.739118 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:01.739092    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85"]
Oct 30 13:11:01.750715 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:01.750690    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-8b77b0-openshift-default-dockercfg-bc5zt\""
Oct 30 13:11:01.773738 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:01.773711    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdd3c1f5-f9fd-400c-90f2-f9693008003a-istiod-ca-cert\") pod \"e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85\" (UID: \"bdd3c1f5-f9fd-400c-90f2-f9693008003a\") " pod="openshift-ingress/e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85"
Oct 30 13:11:01.773870 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:01.773812    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdd3c1f5-f9fd-400c-90f2-f9693008003a-credential-socket\") pod \"e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85\" (UID: \"bdd3c1f5-f9fd-400c-90f2-f9693008003a\") " pod="openshift-ingress/e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85"
Oct 30 13:11:01.773870 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:01.773853    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdd3c1f5-f9fd-400c-90f2-f9693008003a-istio-envoy\") pod \"e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85\" (UID: \"bdd3c1f5-f9fd-400c-90f2-f9693008003a\") " pod="openshift-ingress/e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85"
Oct 30 13:11:01.773870 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:01.773877    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdd3c1f5-f9fd-400c-90f2-f9693008003a-istio-token\") pod \"e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85\" (UID: \"bdd3c1f5-f9fd-400c-90f2-f9693008003a\") " pod="openshift-ingress/e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85"
Oct 30 13:11:01.774066 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:01.773937    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ns6p\" (UniqueName: \"kubernetes.io/projected/bdd3c1f5-f9fd-400c-90f2-f9693008003a-kube-api-access-7ns6p\") pod \"e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85\" (UID: \"bdd3c1f5-f9fd-400c-90f2-f9693008003a\") " pod="openshift-ingress/e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85"
Oct 30 13:11:01.774066 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:01.774020    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdd3c1f5-f9fd-400c-90f2-f9693008003a-istio-data\") pod \"e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85\" (UID: \"bdd3c1f5-f9fd-400c-90f2-f9693008003a\") " pod="openshift-ingress/e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85"
Oct 30 13:11:01.774207 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:01.774098    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdd3c1f5-f9fd-400c-90f2-f9693008003a-workload-socket\") pod \"e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85\" (UID: \"bdd3c1f5-f9fd-400c-90f2-f9693008003a\") " pod="openshift-ingress/e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85"
Oct 30 13:11:01.774207 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:01.774141    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdd3c1f5-f9fd-400c-90f2-f9693008003a-workload-certs\") pod \"e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85\" (UID: \"bdd3c1f5-f9fd-400c-90f2-f9693008003a\") " pod="openshift-ingress/e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85"
Oct 30 13:11:01.774307 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:01.774234    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdd3c1f5-f9fd-400c-90f2-f9693008003a-istio-podinfo\") pod \"e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85\" (UID: \"bdd3c1f5-f9fd-400c-90f2-f9693008003a\") " pod="openshift-ingress/e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85"
Oct 30 13:11:01.914623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:01.888479    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdd3c1f5-f9fd-400c-90f2-f9693008003a-istio-podinfo\") pod \"e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85\" (UID: \"bdd3c1f5-f9fd-400c-90f2-f9693008003a\") " pod="openshift-ingress/e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85"
Oct 30 13:11:01.914623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:01.888733    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdd3c1f5-f9fd-400c-90f2-f9693008003a-istiod-ca-cert\") pod \"e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85\" (UID: \"bdd3c1f5-f9fd-400c-90f2-f9693008003a\") " pod="openshift-ingress/e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85"
Oct 30 13:11:01.914623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:01.888825    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdd3c1f5-f9fd-400c-90f2-f9693008003a-credential-socket\") pod \"e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85\" (UID: \"bdd3c1f5-f9fd-400c-90f2-f9693008003a\") " pod="openshift-ingress/e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85"
Oct 30 13:11:01.914623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:01.888857    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdd3c1f5-f9fd-400c-90f2-f9693008003a-istio-envoy\") pod \"e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85\" (UID: \"bdd3c1f5-f9fd-400c-90f2-f9693008003a\") " pod="openshift-ingress/e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85"
Oct 30 13:11:01.914623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:01.888883    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdd3c1f5-f9fd-400c-90f2-f9693008003a-istio-token\") pod \"e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85\" (UID: \"bdd3c1f5-f9fd-400c-90f2-f9693008003a\") " pod="openshift-ingress/e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85"
Oct 30 13:11:01.914623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:01.888937    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ns6p\" (UniqueName: \"kubernetes.io/projected/bdd3c1f5-f9fd-400c-90f2-f9693008003a-kube-api-access-7ns6p\") pod \"e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85\" (UID: \"bdd3c1f5-f9fd-400c-90f2-f9693008003a\") " pod="openshift-ingress/e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85"
Oct 30 13:11:01.914623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:01.889008    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdd3c1f5-f9fd-400c-90f2-f9693008003a-istio-data\") pod \"e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85\" (UID: \"bdd3c1f5-f9fd-400c-90f2-f9693008003a\") " pod="openshift-ingress/e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85"
Oct 30 13:11:01.914623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:01.889078    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdd3c1f5-f9fd-400c-90f2-f9693008003a-workload-socket\") pod \"e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85\" (UID: \"bdd3c1f5-f9fd-400c-90f2-f9693008003a\") " pod="openshift-ingress/e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85"
Oct 30 13:11:01.914623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:01.889119    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdd3c1f5-f9fd-400c-90f2-f9693008003a-workload-certs\") pod \"e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85\" (UID: \"bdd3c1f5-f9fd-400c-90f2-f9693008003a\") " pod="openshift-ingress/e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85"
Oct 30 13:11:01.914623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:01.889634    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdd3c1f5-f9fd-400c-90f2-f9693008003a-workload-certs\") pod \"e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85\" (UID: \"bdd3c1f5-f9fd-400c-90f2-f9693008003a\") " pod="openshift-ingress/e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85"
Oct 30 13:11:01.994298 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:01.942976    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdd3c1f5-f9fd-400c-90f2-f9693008003a-istiod-ca-cert\") pod \"e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85\" (UID: \"bdd3c1f5-f9fd-400c-90f2-f9693008003a\") " pod="openshift-ingress/e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85"
Oct 30 13:11:01.994298 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:01.943230    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdd3c1f5-f9fd-400c-90f2-f9693008003a-credential-socket\") pod \"e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85\" (UID: \"bdd3c1f5-f9fd-400c-90f2-f9693008003a\") " pod="openshift-ingress/e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85"
Oct 30 13:11:02.010624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:02.004016    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdd3c1f5-f9fd-400c-90f2-f9693008003a-istio-data\") pod \"e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85\" (UID: \"bdd3c1f5-f9fd-400c-90f2-f9693008003a\") " pod="openshift-ingress/e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85"
Oct 30 13:11:02.010624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:02.004633    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdd3c1f5-f9fd-400c-90f2-f9693008003a-workload-socket\") pod \"e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85\" (UID: \"bdd3c1f5-f9fd-400c-90f2-f9693008003a\") " pod="openshift-ingress/e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85"
Oct 30 13:11:02.326676 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:02.326520    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdd3c1f5-f9fd-400c-90f2-f9693008003a-istio-podinfo\") pod \"e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85\" (UID: \"bdd3c1f5-f9fd-400c-90f2-f9693008003a\") " pod="openshift-ingress/e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85"
Oct 30 13:11:02.360624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:02.333181    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdd3c1f5-f9fd-400c-90f2-f9693008003a-istio-token\") pod \"e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85\" (UID: \"bdd3c1f5-f9fd-400c-90f2-f9693008003a\") " pod="openshift-ingress/e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85"
Oct 30 13:11:02.360624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:02.334307    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ns6p\" (UniqueName: \"kubernetes.io/projected/bdd3c1f5-f9fd-400c-90f2-f9693008003a-kube-api-access-7ns6p\") pod \"e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85\" (UID: \"bdd3c1f5-f9fd-400c-90f2-f9693008003a\") " pod="openshift-ingress/e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85"
Oct 30 13:11:02.380657 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:02.380622    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdd3c1f5-f9fd-400c-90f2-f9693008003a-istio-envoy\") pod \"e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85\" (UID: \"bdd3c1f5-f9fd-400c-90f2-f9693008003a\") " pod="openshift-ingress/e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85"
Oct 30 13:11:02.639497 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:02.639449    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-b31c2926-6b889ddf99-7dt86" podUID="6ef82918-510b-4cbc-b4a0-2b9fb48c2c08" containerName="payload-pre-processing" containerID="cri-o://502b8735006063dff54700b1fd5d4aca178d9998d6d0cc2067434989feb20f64" gracePeriod=25
Oct 30 13:11:02.648873 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:02.648842    4364 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 13:11:02.649199 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:02.648910    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh" podUID="42d369e1-ee25-47fe-a808-cb0470669962" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 13:11:02.653174 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:02.653151    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85"
Oct 30 13:11:03.202391 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:03.202333    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85"]
Oct 30 13:11:03.224099 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:03.224057    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:11:03.224204 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:03.224178    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:11:03.224256 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:03.224220    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:11:03.270394 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:03.270373    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b31c2926-6b889ddf99-7dt86"
Oct 30 13:11:03.343803 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:03.342870    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ef82918-510b-4cbc-b4a0-2b9fb48c2c08-kube-api-access-89vgm\" (UniqueName: \"kubernetes.io/projected/6ef82918-510b-4cbc-b4a0-2b9fb48c2c08-kube-api-access-89vgm\") pod \"6ef82918-510b-4cbc-b4a0-2b9fb48c2c08\" (UID: \"6ef82918-510b-4cbc-b4a0-2b9fb48c2c08\") "
Oct 30 13:11:03.343803 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:03.343558    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ef82918-510b-4cbc-b4a0-2b9fb48c2c08-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ef82918-510b-4cbc-b4a0-2b9fb48c2c08-plugins-config-volume\") pod \"6ef82918-510b-4cbc-b4a0-2b9fb48c2c08\" (UID: \"6ef82918-510b-4cbc-b4a0-2b9fb48c2c08\") "
Oct 30 13:11:03.343990 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:03.343965    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ef82918-510b-4cbc-b4a0-2b9fb48c2c08-plugins-config-volume" pod "6ef82918-510b-4cbc-b4a0-2b9fb48c2c08" (UID: "6ef82918-510b-4cbc-b4a0-2b9fb48c2c08"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:11:03.344805 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:03.344779    4364 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ef82918-510b-4cbc-b4a0-2b9fb48c2c08-plugins-config-volume\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:11:03.346179 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:03.346144    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ef82918-510b-4cbc-b4a0-2b9fb48c2c08-kube-api-access-89vgm" pod "6ef82918-510b-4cbc-b4a0-2b9fb48c2c08" (UID: "6ef82918-510b-4cbc-b4a0-2b9fb48c2c08"). InnerVolumeSpecName "kube-api-access-89vgm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:11:03.450073 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:03.447062    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89vgm\" (UniqueName: \"kubernetes.io/projected/6ef82918-510b-4cbc-b4a0-2b9fb48c2c08-kube-api-access-89vgm\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:11:03.647874 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:03.647837    4364 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 13:11:03.648036 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:03.647891    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh" podUID="42d369e1-ee25-47fe-a808-cb0470669962" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 13:11:03.666311 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:03.666279    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85" event={"ID":"bdd3c1f5-f9fd-400c-90f2-f9693008003a","Type":"ContainerStarted","Data":"3ba0c9e791b155c0c55ef49e6410558e335a26c9456e87f0bfb5ec5a720c041d"}
Oct 30 13:11:03.666748 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:03.666340    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85" event={"ID":"bdd3c1f5-f9fd-400c-90f2-f9693008003a","Type":"ContainerStarted","Data":"a2f3eef98f6456a6d5769a7dd5c317aa94440179a5836a64dff6e377fa564dd0"}
Oct 30 13:11:03.668115 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:03.668092    4364 generic.go:356] "Generic (PLEG): container finished" podID="6ef82918-510b-4cbc-b4a0-2b9fb48c2c08" containerID="502b8735006063dff54700b1fd5d4aca178d9998d6d0cc2067434989feb20f64" exitCode=0
Oct 30 13:11:03.668235 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:03.668132    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b31c2926-6b889ddf99-7dt86" event={"ID":"6ef82918-510b-4cbc-b4a0-2b9fb48c2c08","Type":"ContainerDied","Data":"502b8735006063dff54700b1fd5d4aca178d9998d6d0cc2067434989feb20f64"}
Oct 30 13:11:03.668235 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:03.668153    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b31c2926-6b889ddf99-7dt86"
Oct 30 13:11:03.668235 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:03.668171    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b31c2926-6b889ddf99-7dt86" event={"ID":"6ef82918-510b-4cbc-b4a0-2b9fb48c2c08","Type":"ContainerDied","Data":"26e98b4ef0a7956f7b8df5ed0f8a30e82c833f9a7a7ac1638ada192298bd14cc"}
Oct 30 13:11:03.668235 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:03.668192    4364 scope.go:122] "RemoveContainer" containerID="502b8735006063dff54700b1fd5d4aca178d9998d6d0cc2067434989feb20f64"
Oct 30 13:11:03.703752 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:03.696254    4364 scope.go:122] "RemoveContainer" containerID="502b8735006063dff54700b1fd5d4aca178d9998d6d0cc2067434989feb20f64"
Oct 30 13:11:03.703752 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:11:03.696617    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"502b8735006063dff54700b1fd5d4aca178d9998d6d0cc2067434989feb20f64\": container with ID starting with 502b8735006063dff54700b1fd5d4aca178d9998d6d0cc2067434989feb20f64 not found: ID does not exist" containerID="502b8735006063dff54700b1fd5d4aca178d9998d6d0cc2067434989feb20f64"
Oct 30 13:11:03.703752 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:03.696653    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"502b8735006063dff54700b1fd5d4aca178d9998d6d0cc2067434989feb20f64"} err="failed to get container status \"502b8735006063dff54700b1fd5d4aca178d9998d6d0cc2067434989feb20f64\": rpc error: code = NotFound desc = could not find container \"502b8735006063dff54700b1fd5d4aca178d9998d6d0cc2067434989feb20f64\": container with ID starting with 502b8735006063dff54700b1fd5d4aca178d9998d6d0cc2067434989feb20f64 not found: ID does not exist"
Oct 30 13:11:03.719317 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:03.719273    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85" podStartSLOduration=2.719259642 podStartE2EDuration="2.719259642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:03.716480817 +0000 UTC m=+1162.882845466" watchObservedRunningTime="2026-10-30 13:11:03.719259642 +0000 UTC m=+1162.885624289"
Oct 30 13:11:03.762053 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:03.762025    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b31c2926-6b889ddf99-7dt86"]
Oct 30 13:11:03.770311 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:03.770286    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b31c2926-6b889ddf99-7dt86"]
Oct 30 13:11:03.771236 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:03.771202    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-7e2137bc-59bcbfbcc6-r8q9v" podUID="dc82b255-a820-45b5-af91-a9d4f7eb824f" containerName="payload-processing" containerID="cri-o://9e66c526fd24fda071a4826ccbf5987ac3e2d075fdbf46f0aedce763e2718ffb" gracePeriod=25
Oct 30 13:11:04.048907 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:04.048860    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cf4d32e-vvmq5"]
Oct 30 13:11:04.056240 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:04.055873    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cf4d32e-vvmq5"]
Oct 30 13:11:04.204202 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:04.204172    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-7e2137bc-59bcbfbcc6-r8q9v"
Oct 30 13:11:04.262497 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:04.262466    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc82b255-a820-45b5-af91-a9d4f7eb824f-kube-api-access-bznt5\" (UniqueName: \"kubernetes.io/projected/dc82b255-a820-45b5-af91-a9d4f7eb824f-kube-api-access-bznt5\") pod \"dc82b255-a820-45b5-af91-a9d4f7eb824f\" (UID: \"dc82b255-a820-45b5-af91-a9d4f7eb824f\") "
Oct 30 13:11:04.262762 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:04.262745    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc82b255-a820-45b5-af91-a9d4f7eb824f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc82b255-a820-45b5-af91-a9d4f7eb824f-plugins-config-volume\") pod \"dc82b255-a820-45b5-af91-a9d4f7eb824f\" (UID: \"dc82b255-a820-45b5-af91-a9d4f7eb824f\") "
Oct 30 13:11:04.263238 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:04.263175    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc82b255-a820-45b5-af91-a9d4f7eb824f-plugins-config-volume" pod "dc82b255-a820-45b5-af91-a9d4f7eb824f" (UID: "dc82b255-a820-45b5-af91-a9d4f7eb824f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:11:04.263532 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:04.263508    4364 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc82b255-a820-45b5-af91-a9d4f7eb824f-plugins-config-volume\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:11:04.264854 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:04.264828    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc82b255-a820-45b5-af91-a9d4f7eb824f-kube-api-access-bznt5" pod "dc82b255-a820-45b5-af91-a9d4f7eb824f" (UID: "dc82b255-a820-45b5-af91-a9d4f7eb824f"). InnerVolumeSpecName "kube-api-access-bznt5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:11:04.367624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:04.365532    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bznt5\" (UniqueName: \"kubernetes.io/projected/dc82b255-a820-45b5-af91-a9d4f7eb824f-kube-api-access-bznt5\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:11:04.661622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:04.653004    4364 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 13:11:04.661622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:04.653070    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh" podUID="42d369e1-ee25-47fe-a808-cb0470669962" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 13:11:04.661622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:04.653678    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85"
Oct 30 13:11:04.721643 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:04.717525    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85"
Oct 30 13:11:04.733068 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:04.730179    4364 generic.go:356] "Generic (PLEG): container finished" podID="dc82b255-a820-45b5-af91-a9d4f7eb824f" containerID="9e66c526fd24fda071a4826ccbf5987ac3e2d075fdbf46f0aedce763e2718ffb" exitCode=0
Oct 30 13:11:04.733068 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:04.731661    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-7e2137bc-59bcbfbcc6-r8q9v"
Oct 30 13:11:04.733068 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:04.732806    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-7e2137bc-59bcbfbcc6-r8q9v" event={"ID":"dc82b255-a820-45b5-af91-a9d4f7eb824f","Type":"ContainerDied","Data":"9e66c526fd24fda071a4826ccbf5987ac3e2d075fdbf46f0aedce763e2718ffb"}
Oct 30 13:11:04.733068 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:04.732859    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-7e2137bc-59bcbfbcc6-r8q9v" event={"ID":"dc82b255-a820-45b5-af91-a9d4f7eb824f","Type":"ContainerDied","Data":"1a557e2c91ffc05257b1e671df6ca69888ee683b4da52e3fb7d2e0d171ffeb74"}
Oct 30 13:11:04.733068 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:04.732893    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85"
Oct 30 13:11:04.733068 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:04.732924    4364 scope.go:122] "RemoveContainer" containerID="9e66c526fd24fda071a4826ccbf5987ac3e2d075fdbf46f0aedce763e2718ffb"
Oct 30 13:11:04.744629 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:04.736877    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85"
Oct 30 13:11:04.765287 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:04.764156    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7e2137bc-6b578c886f-bhb5f"]
Oct 30 13:11:04.765287 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:04.764567    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7e2137bc-6b578c886f-bhb5f" podUID="715b8a02-efaa-43ea-b8f0-478fb3e11e68" containerName="maas-api" containerID="cri-o://8414ab250aad0980bee5ed5a6b18175164cdac6bc0887c8a48a1d348af660532" gracePeriod=30
Oct 30 13:11:04.834924 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:04.834819    4364 scope.go:122] "RemoveContainer" containerID="9e66c526fd24fda071a4826ccbf5987ac3e2d075fdbf46f0aedce763e2718ffb"
Oct 30 13:11:04.876674 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:11:04.855751    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e66c526fd24fda071a4826ccbf5987ac3e2d075fdbf46f0aedce763e2718ffb\": container with ID starting with 9e66c526fd24fda071a4826ccbf5987ac3e2d075fdbf46f0aedce763e2718ffb not found: ID does not exist" containerID="9e66c526fd24fda071a4826ccbf5987ac3e2d075fdbf46f0aedce763e2718ffb"
Oct 30 13:11:04.876674 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:04.855797    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e66c526fd24fda071a4826ccbf5987ac3e2d075fdbf46f0aedce763e2718ffb"} err="failed to get container status \"9e66c526fd24fda071a4826ccbf5987ac3e2d075fdbf46f0aedce763e2718ffb\": rpc error: code = NotFound desc = could not find container \"9e66c526fd24fda071a4826ccbf5987ac3e2d075fdbf46f0aedce763e2718ffb\": container with ID starting with 9e66c526fd24fda071a4826ccbf5987ac3e2d075fdbf46f0aedce763e2718ffb not found: ID does not exist"
Oct 30 13:11:05.690622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:05.690026    4364 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 13:11:05.690622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:05.690092    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh" podUID="42d369e1-ee25-47fe-a808-cb0470669962" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 13:11:05.737775 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:05.737499    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43d5c676-c1f0-4dfc-9ef3-d027f6c9fd2c" path="/var/lib/kubelet/pods/43d5c676-c1f0-4dfc-9ef3-d027f6c9fd2c/volumes"
Oct 30 13:11:05.741924 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:05.741887    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ef82918-510b-4cbc-b4a0-2b9fb48c2c08" path="/var/lib/kubelet/pods/6ef82918-510b-4cbc-b4a0-2b9fb48c2c08/volumes"
Oct 30 13:11:05.942675 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:05.905405    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7e2137bc-6b578c886f-bhb5f" podUID="715b8a02-efaa-43ea-b8f0-478fb3e11e68" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.102:8443/health\": dial tcp 10.133.0.102:8443: connect: connection refused"
Oct 30 13:11:06.041930 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:06.025648    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2gfr2"]
Oct 30 13:11:06.041930 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:06.026012    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2gfr2" podUID="35d8c3ab-6289-40b1-ac0b-5e155420ee38" containerName="istio-proxy" containerID="cri-o://4af097e9d0caa680e7a29135fee164ac03fbaffa49b66e89d92706576c5be268" gracePeriod=30
Oct 30 13:11:06.129378 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:06.129342    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-zl7wp"]
Oct 30 13:11:06.131387 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:06.131348    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-zl7wp" podUID="3e3c6b89-88e3-460f-8d61-ca9438b25ddb" containerName="istio-proxy" containerID="cri-o://4f73548b61d2e5b2df82d766d1e87d97da56a53da7928db1c7c92c14a466f03b" gracePeriod=30
Oct 30 13:11:06.348822 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:06.348655    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-7e2137bc-59bcbfbcc6-r8q9v"]
Oct 30 13:11:06.426626 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:06.419616    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-7e2137bc-59bcbfbcc6-r8q9v"]
Oct 30 13:11:06.469563 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:06.466571    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f"]
Oct 30 13:11:06.469563 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:06.466836    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f" podUID="e1acea1c-40f3-4414-af50-8281af1a3809" containerName="istio-proxy" containerID="cri-o://124e5d84e38fa49aa3459495b6a769c8f3f524823289aa598a634f59a58386e0" gracePeriod=30
Oct 30 13:11:07.346943 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:07.346562    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc" podUID="8911b790-331d-490c-876b-647e6d2fcd14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:07.547618 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:07.533779    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc82b255-a820-45b5-af91-a9d4f7eb824f" path="/var/lib/kubelet/pods/dc82b255-a820-45b5-af91-a9d4f7eb824f/volumes"
Oct 30 13:11:07.598621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:07.598533    4364 generic.go:356] "Generic (PLEG): container finished" podID="715b8a02-efaa-43ea-b8f0-478fb3e11e68" containerID="8414ab250aad0980bee5ed5a6b18175164cdac6bc0887c8a48a1d348af660532" exitCode=0
Oct 30 13:11:07.599829 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:07.599775    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7e2137bc-6b578c886f-bhb5f" event={"ID":"715b8a02-efaa-43ea-b8f0-478fb3e11e68","Type":"ContainerDied","Data":"8414ab250aad0980bee5ed5a6b18175164cdac6bc0887c8a48a1d348af660532"}
Oct 30 13:11:07.652908 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:07.652527    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7e2137bc-6b578c886f-bhb5f"
Oct 30 13:11:07.713804 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:07.694150    4364 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:11:07.713804 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:07.694223    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh" podUID="42d369e1-ee25-47fe-a808-cb0470669962" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:07.767652 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:07.766159    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/715b8a02-efaa-43ea-b8f0-478fb3e11e68-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/715b8a02-efaa-43ea-b8f0-478fb3e11e68-openshift-service-ca\") pod \"715b8a02-efaa-43ea-b8f0-478fb3e11e68\" (UID: \"715b8a02-efaa-43ea-b8f0-478fb3e11e68\") "
Oct 30 13:11:07.767652 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:07.766214    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/715b8a02-efaa-43ea-b8f0-478fb3e11e68-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/715b8a02-efaa-43ea-b8f0-478fb3e11e68-maas-api-tls\") pod \"715b8a02-efaa-43ea-b8f0-478fb3e11e68\" (UID: \"715b8a02-efaa-43ea-b8f0-478fb3e11e68\") "
Oct 30 13:11:07.767652 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:07.766545    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/715b8a02-efaa-43ea-b8f0-478fb3e11e68-metrics-certs\" (UniqueName: \"kubernetes.io/secret/715b8a02-efaa-43ea-b8f0-478fb3e11e68-metrics-certs\") pod \"715b8a02-efaa-43ea-b8f0-478fb3e11e68\" (UID: \"715b8a02-efaa-43ea-b8f0-478fb3e11e68\") "
Oct 30 13:11:07.767652 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:07.766659    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/715b8a02-efaa-43ea-b8f0-478fb3e11e68-kube-api-access-c4w2x\" (UniqueName: \"kubernetes.io/projected/715b8a02-efaa-43ea-b8f0-478fb3e11e68-kube-api-access-c4w2x\") pod \"715b8a02-efaa-43ea-b8f0-478fb3e11e68\" (UID: \"715b8a02-efaa-43ea-b8f0-478fb3e11e68\") "
Oct 30 13:11:07.802733 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:07.792840    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/715b8a02-efaa-43ea-b8f0-478fb3e11e68-openshift-service-ca" pod "715b8a02-efaa-43ea-b8f0-478fb3e11e68" (UID: "715b8a02-efaa-43ea-b8f0-478fb3e11e68"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:11:07.849864 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:07.829931    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/715b8a02-efaa-43ea-b8f0-478fb3e11e68-maas-api-tls" pod "715b8a02-efaa-43ea-b8f0-478fb3e11e68" (UID: "715b8a02-efaa-43ea-b8f0-478fb3e11e68"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:11:07.849864 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:07.830012    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/715b8a02-efaa-43ea-b8f0-478fb3e11e68-metrics-certs" pod "715b8a02-efaa-43ea-b8f0-478fb3e11e68" (UID: "715b8a02-efaa-43ea-b8f0-478fb3e11e68"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:11:07.854747 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:07.854694    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/715b8a02-efaa-43ea-b8f0-478fb3e11e68-kube-api-access-c4w2x" pod "715b8a02-efaa-43ea-b8f0-478fb3e11e68" (UID: "715b8a02-efaa-43ea-b8f0-478fb3e11e68"). InnerVolumeSpecName "kube-api-access-c4w2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:11:07.977761 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:07.951552    4364 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/715b8a02-efaa-43ea-b8f0-478fb3e11e68-metrics-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:11:07.978628 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:07.951589    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4w2x\" (UniqueName: \"kubernetes.io/projected/715b8a02-efaa-43ea-b8f0-478fb3e11e68-kube-api-access-c4w2x\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:11:07.978845 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:07.978798    4364 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/715b8a02-efaa-43ea-b8f0-478fb3e11e68-openshift-service-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:11:07.978845 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:07.978826    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/715b8a02-efaa-43ea-b8f0-478fb3e11e68-maas-api-tls\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:11:08.555459 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:08.555395    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh"
Oct 30 13:11:08.557047 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:08.557016    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh"
Oct 30 13:11:08.558974 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:08.558945    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh"
Oct 30 13:11:08.616854 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:08.616671    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7e2137bc-6b578c886f-bhb5f" event={"ID":"715b8a02-efaa-43ea-b8f0-478fb3e11e68","Type":"ContainerDied","Data":"9721096e73f617f288d7a17a10568103a428a5444948e88dd83d8dd48479074c"}
Oct 30 13:11:08.616854 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:08.616729    4364 scope.go:122] "RemoveContainer" containerID="8414ab250aad0980bee5ed5a6b18175164cdac6bc0887c8a48a1d348af660532"
Oct 30 13:11:08.617501 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:08.617470    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7e2137bc-6b578c886f-bhb5f"
Oct 30 13:11:08.744340 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:08.743971    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7e2137bc-6b578c886f-bhb5f"]
Oct 30 13:11:08.752842 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:08.751854    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7e2137bc-6b578c886f-bhb5f"]
Oct 30 13:11:08.774883 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:08.774848    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8b77b0-d46dbfdc-lkc4n"]
Oct 30 13:11:08.775615 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:08.775583    4364 cpu_manager.go:414] "containerMap: removing container" podUID="dc82b255-a820-45b5-af91-a9d4f7eb824f" containerName="payload-processing"
Oct 30 13:11:08.775615 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:08.775614    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc82b255-a820-45b5-af91-a9d4f7eb824f" containerName="payload-processing"
Oct 30 13:11:08.775785 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:08.775641    4364 cpu_manager.go:414] "containerMap: removing container" podUID="715b8a02-efaa-43ea-b8f0-478fb3e11e68" containerName="maas-api"
Oct 30 13:11:08.775785 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:08.775647    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="715b8a02-efaa-43ea-b8f0-478fb3e11e68" containerName="maas-api"
Oct 30 13:11:08.775785 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:08.775662    4364 cpu_manager.go:414] "containerMap: removing container" podUID="6ef82918-510b-4cbc-b4a0-2b9fb48c2c08" containerName="payload-pre-processing"
Oct 30 13:11:08.775785 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:08.775667    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ef82918-510b-4cbc-b4a0-2b9fb48c2c08" containerName="payload-pre-processing"
Oct 30 13:11:08.775785 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:08.775777    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ef82918-510b-4cbc-b4a0-2b9fb48c2c08" containerName="payload-pre-processing"
Oct 30 13:11:08.776045 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:08.775793    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc82b255-a820-45b5-af91-a9d4f7eb824f" containerName="payload-processing"
Oct 30 13:11:08.776045 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:08.775801    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="715b8a02-efaa-43ea-b8f0-478fb3e11e68" containerName="maas-api"
Oct 30 13:11:08.780747 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:08.780724    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8b77b0-d46dbfdc-lkc4n"
Oct 30 13:11:08.799471 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:08.799404    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-8b77b0\""
Oct 30 13:11:08.799593 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:08.799519    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-8b77b0-dockercfg-m8xzb\""
Oct 30 13:11:08.799593 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:08.799568    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8b77b0-d46dbfdc-lkc4n"]
Oct 30 13:11:08.824706 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:08.824682    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98935320-7c94-4bd4-a88a-44c9ea91fcc6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-8b77b0-d46dbfdc-lkc4n\" (UID: \"98935320-7c94-4bd4-a88a-44c9ea91fcc6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8b77b0-d46dbfdc-lkc4n"
Oct 30 13:11:08.824860 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:08.824803    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx74p\" (UniqueName: \"kubernetes.io/projected/98935320-7c94-4bd4-a88a-44c9ea91fcc6-kube-api-access-sx74p\") pod \"payload-pre-processing-e2e-shared-a-w2-8b77b0-d46dbfdc-lkc4n\" (UID: \"98935320-7c94-4bd4-a88a-44c9ea91fcc6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8b77b0-d46dbfdc-lkc4n"
Oct 30 13:11:08.832779 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:08.832755    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-8b77b0-75f95c8744-psdzn"]
Oct 30 13:11:08.838284 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:08.838264    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8b77b0-75f95c8744-psdzn"
Oct 30 13:11:08.929625 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:08.926217    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98935320-7c94-4bd4-a88a-44c9ea91fcc6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-8b77b0-d46dbfdc-lkc4n\" (UID: \"98935320-7c94-4bd4-a88a-44c9ea91fcc6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8b77b0-d46dbfdc-lkc4n"
Oct 30 13:11:08.929625 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:08.926287    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91859899-765d-48fd-9c18-a37942683a5a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-8b77b0-75f95c8744-psdzn\" (UID: \"91859899-765d-48fd-9c18-a37942683a5a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8b77b0-75f95c8744-psdzn"
Oct 30 13:11:08.929625 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:08.926433    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx74p\" (UniqueName: \"kubernetes.io/projected/98935320-7c94-4bd4-a88a-44c9ea91fcc6-kube-api-access-sx74p\") pod \"payload-pre-processing-e2e-shared-a-w2-8b77b0-d46dbfdc-lkc4n\" (UID: \"98935320-7c94-4bd4-a88a-44c9ea91fcc6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8b77b0-d46dbfdc-lkc4n"
Oct 30 13:11:08.929625 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:08.926488    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swk67\" (UniqueName: \"kubernetes.io/projected/91859899-765d-48fd-9c18-a37942683a5a-kube-api-access-swk67\") pod \"payload-processing-e2e-shared-a-w2-8b77b0-75f95c8744-psdzn\" (UID: \"91859899-765d-48fd-9c18-a37942683a5a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8b77b0-75f95c8744-psdzn"
Oct 30 13:11:08.929625 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:08.927556    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98935320-7c94-4bd4-a88a-44c9ea91fcc6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-8b77b0-d46dbfdc-lkc4n\" (UID: \"98935320-7c94-4bd4-a88a-44c9ea91fcc6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8b77b0-d46dbfdc-lkc4n"
Oct 30 13:11:09.048635 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:09.047637    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91859899-765d-48fd-9c18-a37942683a5a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-8b77b0-75f95c8744-psdzn\" (UID: \"91859899-765d-48fd-9c18-a37942683a5a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8b77b0-75f95c8744-psdzn"
Oct 30 13:11:09.072671 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:09.070464    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swk67\" (UniqueName: \"kubernetes.io/projected/91859899-765d-48fd-9c18-a37942683a5a-kube-api-access-swk67\") pod \"payload-processing-e2e-shared-a-w2-8b77b0-75f95c8744-psdzn\" (UID: \"91859899-765d-48fd-9c18-a37942683a5a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8b77b0-75f95c8744-psdzn"
Oct 30 13:11:09.072671 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:09.071288    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91859899-765d-48fd-9c18-a37942683a5a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-8b77b0-75f95c8744-psdzn\" (UID: \"91859899-765d-48fd-9c18-a37942683a5a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8b77b0-75f95c8744-psdzn"
Oct 30 13:11:09.214621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:09.205326    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="715b8a02-efaa-43ea-b8f0-478fb3e11e68" path="/var/lib/kubelet/pods/715b8a02-efaa-43ea-b8f0-478fb3e11e68/volumes"
Oct 30 13:11:09.214621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:09.205827    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-8b77b0-75f95c8744-psdzn"]
Oct 30 13:11:09.219626 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:09.217786    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx74p\" (UniqueName: \"kubernetes.io/projected/98935320-7c94-4bd4-a88a-44c9ea91fcc6-kube-api-access-sx74p\") pod \"payload-pre-processing-e2e-shared-a-w2-8b77b0-d46dbfdc-lkc4n\" (UID: \"98935320-7c94-4bd4-a88a-44c9ea91fcc6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8b77b0-d46dbfdc-lkc4n"
Oct 30 13:11:09.228460 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:09.228389    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swk67\" (UniqueName: \"kubernetes.io/projected/91859899-765d-48fd-9c18-a37942683a5a-kube-api-access-swk67\") pod \"payload-processing-e2e-shared-a-w2-8b77b0-75f95c8744-psdzn\" (UID: \"91859899-765d-48fd-9c18-a37942683a5a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8b77b0-75f95c8744-psdzn"
Oct 30 13:11:09.231282 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:09.231078    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8b77b0-75f95c8744-psdzn"
Oct 30 13:11:09.399521 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:09.398919    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8b77b0-d46dbfdc-lkc4n"
Oct 30 13:11:09.649877 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:11:09.649851    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91859899_765d_48fd_9c18_a37942683a5a.slice/crio-bba8c450f9b9dade358bee6a130c06032cd6f9767285f94c3ffca67902b482e6 WatchSource:0}: Error finding container bba8c450f9b9dade358bee6a130c06032cd6f9767285f94c3ffca67902b482e6: Status 404 returned error can't find the container with id bba8c450f9b9dade358bee6a130c06032cd6f9767285f94c3ffca67902b482e6
Oct 30 13:11:09.651861 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:09.651837    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-8b77b0-75f95c8744-psdzn"]
Oct 30 13:11:09.737566 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:09.737075    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8b77b0-d46dbfdc-lkc4n"]
Oct 30 13:11:10.740396 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:10.740349    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8b77b0-75f95c8744-psdzn" event={"ID":"91859899-765d-48fd-9c18-a37942683a5a","Type":"ContainerStarted","Data":"a79a3febd438e573bd96ce4218a46b2a2cb945b852029ef8c52e05cc7fa27929"}
Oct 30 13:11:10.741076 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:10.741036    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8b77b0-75f95c8744-psdzn" event={"ID":"91859899-765d-48fd-9c18-a37942683a5a","Type":"ContainerStarted","Data":"bba8c450f9b9dade358bee6a130c06032cd6f9767285f94c3ffca67902b482e6"}
Oct 30 13:11:10.745616 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:10.742495    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8b77b0-75f95c8744-psdzn"
Oct 30 13:11:10.765313 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:10.751254    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8b77b0-d46dbfdc-lkc4n" event={"ID":"98935320-7c94-4bd4-a88a-44c9ea91fcc6","Type":"ContainerStarted","Data":"a8ec27f0d889c3e3fd326912856d634aa92742ae01012a0ca4b984562ba52036"}
Oct 30 13:11:10.765313 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:10.751305    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8b77b0-d46dbfdc-lkc4n" event={"ID":"98935320-7c94-4bd4-a88a-44c9ea91fcc6","Type":"ContainerStarted","Data":"494ccbaac3e78873fe215c564d66584527f2ba8fff07f665306e6a15fec2484b"}
Oct 30 13:11:10.765313 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:10.752250    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8b77b0-d46dbfdc-lkc4n"
Oct 30 13:11:11.127675 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:11.114172    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8b77b0-75f95c8744-psdzn" podStartSLOduration=3.114144176 podStartE2EDuration="3.114144176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:10.846836252 +0000 UTC m=+1170.013200927" watchObservedRunningTime="2026-10-30 13:11:11.114144176 +0000 UTC m=+1170.280508828"
Oct 30 13:11:11.134617 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:11.134179    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8b77b0-d46dbfdc-lkc4n" podStartSLOduration=3.134150975 podStartE2EDuration="3.134150975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:10.911144236 +0000 UTC m=+1170.077508895" watchObservedRunningTime="2026-10-30 13:11:11.134150975 +0000 UTC m=+1170.300515624"
Oct 30 13:11:11.493357 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:11.493330    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-zl7wp"
Oct 30 13:11:11.512973 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:11:11.512943    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:11:11.555369 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:11.555335    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e3c6b89-88e3-460f-8d61-ca9438b25ddb-kube-api-access-h45dg\" (UniqueName: \"kubernetes.io/projected/3e3c6b89-88e3-460f-8d61-ca9438b25ddb-kube-api-access-h45dg\") pod \"3e3c6b89-88e3-460f-8d61-ca9438b25ddb\" (UID: \"3e3c6b89-88e3-460f-8d61-ca9438b25ddb\") "
Oct 30 13:11:11.555519 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:11.555456    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e3c6b89-88e3-460f-8d61-ca9438b25ddb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e3c6b89-88e3-460f-8d61-ca9438b25ddb-workload-socket\") pod \"3e3c6b89-88e3-460f-8d61-ca9438b25ddb\" (UID: \"3e3c6b89-88e3-460f-8d61-ca9438b25ddb\") "
Oct 30 13:11:11.555519 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:11.555508    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e3c6b89-88e3-460f-8d61-ca9438b25ddb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e3c6b89-88e3-460f-8d61-ca9438b25ddb-workload-certs\") pod \"3e3c6b89-88e3-460f-8d61-ca9438b25ddb\" (UID: \"3e3c6b89-88e3-460f-8d61-ca9438b25ddb\") "
Oct 30 13:11:11.555631 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:11.555584    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3e3c6b89-88e3-460f-8d61-ca9438b25ddb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e3c6b89-88e3-460f-8d61-ca9438b25ddb-istio-podinfo\") pod \"3e3c6b89-88e3-460f-8d61-ca9438b25ddb\" (UID: \"3e3c6b89-88e3-460f-8d61-ca9438b25ddb\") "
Oct 30 13:11:11.555783 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:11.555764    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e3c6b89-88e3-460f-8d61-ca9438b25ddb-istio-token\" (UniqueName: \"kubernetes.io/projected/3e3c6b89-88e3-460f-8d61-ca9438b25ddb-istio-token\") pod \"3e3c6b89-88e3-460f-8d61-ca9438b25ddb\" (UID: \"3e3c6b89-88e3-460f-8d61-ca9438b25ddb\") "
Oct 30 13:11:11.555861 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:11.555809    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e3c6b89-88e3-460f-8d61-ca9438b25ddb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e3c6b89-88e3-460f-8d61-ca9438b25ddb-istio-envoy\") pod \"3e3c6b89-88e3-460f-8d61-ca9438b25ddb\" (UID: \"3e3c6b89-88e3-460f-8d61-ca9438b25ddb\") "
Oct 30 13:11:11.555921 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:11.555847    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e3c6b89-88e3-460f-8d61-ca9438b25ddb-workload-socket" pod "3e3c6b89-88e3-460f-8d61-ca9438b25ddb" (UID: "3e3c6b89-88e3-460f-8d61-ca9438b25ddb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:11:11.555921 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:11.555891    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e3c6b89-88e3-460f-8d61-ca9438b25ddb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e3c6b89-88e3-460f-8d61-ca9438b25ddb-istio-data\") pod \"3e3c6b89-88e3-460f-8d61-ca9438b25ddb\" (UID: \"3e3c6b89-88e3-460f-8d61-ca9438b25ddb\") "
Oct 30 13:11:11.556022 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:11.555961    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e3c6b89-88e3-460f-8d61-ca9438b25ddb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e3c6b89-88e3-460f-8d61-ca9438b25ddb-istiod-ca-cert\") pod \"3e3c6b89-88e3-460f-8d61-ca9438b25ddb\" (UID: \"3e3c6b89-88e3-460f-8d61-ca9438b25ddb\") "
Oct 30 13:11:11.556022 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:11.555981    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e3c6b89-88e3-460f-8d61-ca9438b25ddb-workload-certs" pod "3e3c6b89-88e3-460f-8d61-ca9438b25ddb" (UID: "3e3c6b89-88e3-460f-8d61-ca9438b25ddb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:11:11.556135 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:11.556052    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e3c6b89-88e3-460f-8d61-ca9438b25ddb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e3c6b89-88e3-460f-8d61-ca9438b25ddb-credential-socket\") pod \"3e3c6b89-88e3-460f-8d61-ca9438b25ddb\" (UID: \"3e3c6b89-88e3-460f-8d61-ca9438b25ddb\") "
Oct 30 13:11:11.556238 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:11.556217    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e3c6b89-88e3-460f-8d61-ca9438b25ddb-istio-data" pod "3e3c6b89-88e3-460f-8d61-ca9438b25ddb" (UID: "3e3c6b89-88e3-460f-8d61-ca9438b25ddb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:11:11.556546 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:11.556523    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e3c6b89-88e3-460f-8d61-ca9438b25ddb-istiod-ca-cert" pod "3e3c6b89-88e3-460f-8d61-ca9438b25ddb" (UID: "3e3c6b89-88e3-460f-8d61-ca9438b25ddb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:11:11.556677 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:11.556543    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e3c6b89-88e3-460f-8d61-ca9438b25ddb-credential-socket" pod "3e3c6b89-88e3-460f-8d61-ca9438b25ddb" (UID: "3e3c6b89-88e3-460f-8d61-ca9438b25ddb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:11:11.557065 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:11.557036    4364 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e3c6b89-88e3-460f-8d61-ca9438b25ddb-istio-data\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:11:11.557176 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:11.557069    4364 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e3c6b89-88e3-460f-8d61-ca9438b25ddb-istiod-ca-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:11:11.557176 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:11.557084    4364 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e3c6b89-88e3-460f-8d61-ca9438b25ddb-credential-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:11:11.557176 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:11.557097    4364 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e3c6b89-88e3-460f-8d61-ca9438b25ddb-workload-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:11:11.557176 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:11.557111    4364 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e3c6b89-88e3-460f-8d61-ca9438b25ddb-workload-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:11:11.558334 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:11.558178    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e3c6b89-88e3-460f-8d61-ca9438b25ddb-istio-token" pod "3e3c6b89-88e3-460f-8d61-ca9438b25ddb" (UID: "3e3c6b89-88e3-460f-8d61-ca9438b25ddb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:11:11.558334 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:11.558293    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e3c6b89-88e3-460f-8d61-ca9438b25ddb-istio-podinfo" pod "3e3c6b89-88e3-460f-8d61-ca9438b25ddb" (UID: "3e3c6b89-88e3-460f-8d61-ca9438b25ddb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:11:11.558575 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:11.558552    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e3c6b89-88e3-460f-8d61-ca9438b25ddb-kube-api-access-h45dg" pod "3e3c6b89-88e3-460f-8d61-ca9438b25ddb" (UID: "3e3c6b89-88e3-460f-8d61-ca9438b25ddb"). InnerVolumeSpecName "kube-api-access-h45dg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:11:11.559289 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:11.559265    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e3c6b89-88e3-460f-8d61-ca9438b25ddb-istio-envoy" pod "3e3c6b89-88e3-460f-8d61-ca9438b25ddb" (UID: "3e3c6b89-88e3-460f-8d61-ca9438b25ddb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:11:11.674732 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:11.673900    4364 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e3c6b89-88e3-460f-8d61-ca9438b25ddb-istio-podinfo\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:11:11.674732 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:11.673938    4364 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e3c6b89-88e3-460f-8d61-ca9438b25ddb-istio-token\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:11:11.674732 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:11.673953    4364 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e3c6b89-88e3-460f-8d61-ca9438b25ddb-istio-envoy\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:11:11.674732 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:11.673967    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h45dg\" (UniqueName: \"kubernetes.io/projected/3e3c6b89-88e3-460f-8d61-ca9438b25ddb-kube-api-access-h45dg\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:11:11.761852 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:11.761821    4364 generic.go:356] "Generic (PLEG): container finished" podID="3e3c6b89-88e3-460f-8d61-ca9438b25ddb" containerID="4f73548b61d2e5b2df82d766d1e87d97da56a53da7928db1c7c92c14a466f03b" exitCode=0
Oct 30 13:11:11.762266 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:11.761870    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-zl7wp"
Oct 30 13:11:11.762266 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:11.761908    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-zl7wp" event={"ID":"3e3c6b89-88e3-460f-8d61-ca9438b25ddb","Type":"ContainerDied","Data":"4f73548b61d2e5b2df82d766d1e87d97da56a53da7928db1c7c92c14a466f03b"}
Oct 30 13:11:11.762266 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:11.761964    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-zl7wp" event={"ID":"3e3c6b89-88e3-460f-8d61-ca9438b25ddb","Type":"ContainerDied","Data":"075255871c1fcb97cb9a09b0d719245a8baeff0be8c9ebbc8962c01c2fc4ed2d"}
Oct 30 13:11:11.762266 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:11.761993    4364 scope.go:122] "RemoveContainer" containerID="4f73548b61d2e5b2df82d766d1e87d97da56a53da7928db1c7c92c14a466f03b"
Oct 30 13:11:11.763962 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:11.763940    4364 generic.go:356] "Generic (PLEG): container finished" podID="35d8c3ab-6289-40b1-ac0b-5e155420ee38" containerID="4af097e9d0caa680e7a29135fee164ac03fbaffa49b66e89d92706576c5be268" exitCode=0
Oct 30 13:11:11.764075 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:11.764005    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2gfr2" event={"ID":"35d8c3ab-6289-40b1-ac0b-5e155420ee38","Type":"ContainerDied","Data":"4af097e9d0caa680e7a29135fee164ac03fbaffa49b66e89d92706576c5be268"}
Oct 30 13:11:11.782695 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:11.775976    4364 scope.go:122] "RemoveContainer" containerID="4f73548b61d2e5b2df82d766d1e87d97da56a53da7928db1c7c92c14a466f03b"
Oct 30 13:11:11.782695 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:11:11.776262    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f73548b61d2e5b2df82d766d1e87d97da56a53da7928db1c7c92c14a466f03b\": container with ID starting with 4f73548b61d2e5b2df82d766d1e87d97da56a53da7928db1c7c92c14a466f03b not found: ID does not exist" containerID="4f73548b61d2e5b2df82d766d1e87d97da56a53da7928db1c7c92c14a466f03b"
Oct 30 13:11:11.782695 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:11.776286    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f73548b61d2e5b2df82d766d1e87d97da56a53da7928db1c7c92c14a466f03b"} err="failed to get container status \"4f73548b61d2e5b2df82d766d1e87d97da56a53da7928db1c7c92c14a466f03b\": rpc error: code = NotFound desc = could not find container \"4f73548b61d2e5b2df82d766d1e87d97da56a53da7928db1c7c92c14a466f03b\": container with ID starting with 4f73548b61d2e5b2df82d766d1e87d97da56a53da7928db1c7c92c14a466f03b not found: ID does not exist"
Oct 30 13:11:11.824250 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:11.824227    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-zl7wp"]
Oct 30 13:11:11.840207 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:11.840182    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-zl7wp"]
Oct 30 13:11:11.984939 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:11.984908    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2gfr2"
Oct 30 13:11:12.062477 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.062437    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f"
Oct 30 13:11:12.081074 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.081050    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/35d8c3ab-6289-40b1-ac0b-5e155420ee38-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35d8c3ab-6289-40b1-ac0b-5e155420ee38-istio-podinfo\") pod \"35d8c3ab-6289-40b1-ac0b-5e155420ee38\" (UID: \"35d8c3ab-6289-40b1-ac0b-5e155420ee38\") "
Oct 30 13:11:12.081213 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.081151    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35d8c3ab-6289-40b1-ac0b-5e155420ee38-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35d8c3ab-6289-40b1-ac0b-5e155420ee38-workload-socket\") pod \"35d8c3ab-6289-40b1-ac0b-5e155420ee38\" (UID: \"35d8c3ab-6289-40b1-ac0b-5e155420ee38\") "
Oct 30 13:11:12.081271 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.081244    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35d8c3ab-6289-40b1-ac0b-5e155420ee38-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35d8c3ab-6289-40b1-ac0b-5e155420ee38-credential-socket\") pod \"35d8c3ab-6289-40b1-ac0b-5e155420ee38\" (UID: \"35d8c3ab-6289-40b1-ac0b-5e155420ee38\") "
Oct 30 13:11:12.081309 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.081287    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35d8c3ab-6289-40b1-ac0b-5e155420ee38-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35d8c3ab-6289-40b1-ac0b-5e155420ee38-istio-data\") pod \"35d8c3ab-6289-40b1-ac0b-5e155420ee38\" (UID: \"35d8c3ab-6289-40b1-ac0b-5e155420ee38\") "
Oct 30 13:11:12.081348 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.081339    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35d8c3ab-6289-40b1-ac0b-5e155420ee38-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35d8c3ab-6289-40b1-ac0b-5e155420ee38-workload-certs\") pod \"35d8c3ab-6289-40b1-ac0b-5e155420ee38\" (UID: \"35d8c3ab-6289-40b1-ac0b-5e155420ee38\") "
Oct 30 13:11:12.081725 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.081404    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35d8c3ab-6289-40b1-ac0b-5e155420ee38-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35d8c3ab-6289-40b1-ac0b-5e155420ee38-istiod-ca-cert\") pod \"35d8c3ab-6289-40b1-ac0b-5e155420ee38\" (UID: \"35d8c3ab-6289-40b1-ac0b-5e155420ee38\") "
Oct 30 13:11:12.081725 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.081437    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35d8c3ab-6289-40b1-ac0b-5e155420ee38-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35d8c3ab-6289-40b1-ac0b-5e155420ee38-istio-envoy\") pod \"35d8c3ab-6289-40b1-ac0b-5e155420ee38\" (UID: \"35d8c3ab-6289-40b1-ac0b-5e155420ee38\") "
Oct 30 13:11:12.081725 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.081477    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35d8c3ab-6289-40b1-ac0b-5e155420ee38-kube-api-access-74zcw\" (UniqueName: \"kubernetes.io/projected/35d8c3ab-6289-40b1-ac0b-5e155420ee38-kube-api-access-74zcw\") pod \"35d8c3ab-6289-40b1-ac0b-5e155420ee38\" (UID: \"35d8c3ab-6289-40b1-ac0b-5e155420ee38\") "
Oct 30 13:11:12.081725 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.081489    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35d8c3ab-6289-40b1-ac0b-5e155420ee38-workload-socket" pod "35d8c3ab-6289-40b1-ac0b-5e155420ee38" (UID: "35d8c3ab-6289-40b1-ac0b-5e155420ee38"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:11:12.081725 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.081556    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35d8c3ab-6289-40b1-ac0b-5e155420ee38-credential-socket" pod "35d8c3ab-6289-40b1-ac0b-5e155420ee38" (UID: "35d8c3ab-6289-40b1-ac0b-5e155420ee38"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:11:12.081725 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.081565    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35d8c3ab-6289-40b1-ac0b-5e155420ee38-istio-data" pod "35d8c3ab-6289-40b1-ac0b-5e155420ee38" (UID: "35d8c3ab-6289-40b1-ac0b-5e155420ee38"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:11:12.081725 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.081640    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35d8c3ab-6289-40b1-ac0b-5e155420ee38-istio-token\" (UniqueName: \"kubernetes.io/projected/35d8c3ab-6289-40b1-ac0b-5e155420ee38-istio-token\") pod \"35d8c3ab-6289-40b1-ac0b-5e155420ee38\" (UID: \"35d8c3ab-6289-40b1-ac0b-5e155420ee38\") "
Oct 30 13:11:12.081725 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.081660    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35d8c3ab-6289-40b1-ac0b-5e155420ee38-workload-certs" pod "35d8c3ab-6289-40b1-ac0b-5e155420ee38" (UID: "35d8c3ab-6289-40b1-ac0b-5e155420ee38"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:11:12.082112 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.081798    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35d8c3ab-6289-40b1-ac0b-5e155420ee38-istiod-ca-cert" pod "35d8c3ab-6289-40b1-ac0b-5e155420ee38" (UID: "35d8c3ab-6289-40b1-ac0b-5e155420ee38"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:11:12.082650 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.082629    4364 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35d8c3ab-6289-40b1-ac0b-5e155420ee38-workload-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:11:12.082727 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.082657    4364 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35d8c3ab-6289-40b1-ac0b-5e155420ee38-credential-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:11:12.082727 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.082672    4364 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35d8c3ab-6289-40b1-ac0b-5e155420ee38-istio-data\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:11:12.082727 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.082687    4364 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35d8c3ab-6289-40b1-ac0b-5e155420ee38-workload-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:11:12.082727 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.082700    4364 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35d8c3ab-6289-40b1-ac0b-5e155420ee38-istiod-ca-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:11:12.083827 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.083803    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35d8c3ab-6289-40b1-ac0b-5e155420ee38-istio-podinfo" pod "35d8c3ab-6289-40b1-ac0b-5e155420ee38" (UID: "35d8c3ab-6289-40b1-ac0b-5e155420ee38"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:11:12.083929 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.083827    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35d8c3ab-6289-40b1-ac0b-5e155420ee38-istio-token" pod "35d8c3ab-6289-40b1-ac0b-5e155420ee38" (UID: "35d8c3ab-6289-40b1-ac0b-5e155420ee38"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:11:12.083974 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.083943    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35d8c3ab-6289-40b1-ac0b-5e155420ee38-istio-envoy" pod "35d8c3ab-6289-40b1-ac0b-5e155420ee38" (UID: "35d8c3ab-6289-40b1-ac0b-5e155420ee38"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:11:12.084015 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.083976    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35d8c3ab-6289-40b1-ac0b-5e155420ee38-kube-api-access-74zcw" pod "35d8c3ab-6289-40b1-ac0b-5e155420ee38" (UID: "35d8c3ab-6289-40b1-ac0b-5e155420ee38"). InnerVolumeSpecName "kube-api-access-74zcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:11:12.183707 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.183670    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1acea1c-40f3-4414-af50-8281af1a3809-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1acea1c-40f3-4414-af50-8281af1a3809-workload-certs\") pod \"e1acea1c-40f3-4414-af50-8281af1a3809\" (UID: \"e1acea1c-40f3-4414-af50-8281af1a3809\") "
Oct 30 13:11:12.183885 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.183779    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1acea1c-40f3-4414-af50-8281af1a3809-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1acea1c-40f3-4414-af50-8281af1a3809-istio-envoy\") pod \"e1acea1c-40f3-4414-af50-8281af1a3809\" (UID: \"e1acea1c-40f3-4414-af50-8281af1a3809\") "
Oct 30 13:11:12.183942 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.183922    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1acea1c-40f3-4414-af50-8281af1a3809-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1acea1c-40f3-4414-af50-8281af1a3809-credential-socket\") pod \"e1acea1c-40f3-4414-af50-8281af1a3809\" (UID: \"e1acea1c-40f3-4414-af50-8281af1a3809\") "
Oct 30 13:11:12.183979 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.183960    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1acea1c-40f3-4414-af50-8281af1a3809-workload-certs" pod "e1acea1c-40f3-4414-af50-8281af1a3809" (UID: "e1acea1c-40f3-4414-af50-8281af1a3809"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:11:12.184036 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.184013    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1acea1c-40f3-4414-af50-8281af1a3809-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1acea1c-40f3-4414-af50-8281af1a3809-workload-socket\") pod \"e1acea1c-40f3-4414-af50-8281af1a3809\" (UID: \"e1acea1c-40f3-4414-af50-8281af1a3809\") "
Oct 30 13:11:12.184094 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.184068    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1acea1c-40f3-4414-af50-8281af1a3809-kube-api-access-brn6x\" (UniqueName: \"kubernetes.io/projected/e1acea1c-40f3-4414-af50-8281af1a3809-kube-api-access-brn6x\") pod \"e1acea1c-40f3-4414-af50-8281af1a3809\" (UID: \"e1acea1c-40f3-4414-af50-8281af1a3809\") "
Oct 30 13:11:12.184211 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.184193    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e1acea1c-40f3-4414-af50-8281af1a3809-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1acea1c-40f3-4414-af50-8281af1a3809-istio-podinfo\") pod \"e1acea1c-40f3-4414-af50-8281af1a3809\" (UID: \"e1acea1c-40f3-4414-af50-8281af1a3809\") "
Oct 30 13:11:12.184280 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.184239    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1acea1c-40f3-4414-af50-8281af1a3809-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1acea1c-40f3-4414-af50-8281af1a3809-istiod-ca-cert\") pod \"e1acea1c-40f3-4414-af50-8281af1a3809\" (UID: \"e1acea1c-40f3-4414-af50-8281af1a3809\") "
Oct 30 13:11:12.184345 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.184267    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1acea1c-40f3-4414-af50-8281af1a3809-credential-socket" pod "e1acea1c-40f3-4414-af50-8281af1a3809" (UID: "e1acea1c-40f3-4414-af50-8281af1a3809"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:11:12.184345 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.184297    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1acea1c-40f3-4414-af50-8281af1a3809-workload-socket" pod "e1acea1c-40f3-4414-af50-8281af1a3809" (UID: "e1acea1c-40f3-4414-af50-8281af1a3809"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:11:12.184345 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.184334    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1acea1c-40f3-4414-af50-8281af1a3809-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1acea1c-40f3-4414-af50-8281af1a3809-istio-data\") pod \"e1acea1c-40f3-4414-af50-8281af1a3809\" (UID: \"e1acea1c-40f3-4414-af50-8281af1a3809\") "
Oct 30 13:11:12.184515 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.184374    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1acea1c-40f3-4414-af50-8281af1a3809-istio-token\" (UniqueName: \"kubernetes.io/projected/e1acea1c-40f3-4414-af50-8281af1a3809-istio-token\") pod \"e1acea1c-40f3-4414-af50-8281af1a3809\" (UID: \"e1acea1c-40f3-4414-af50-8281af1a3809\") "
Oct 30 13:11:12.184662 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.184636    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1acea1c-40f3-4414-af50-8281af1a3809-istio-data" pod "e1acea1c-40f3-4414-af50-8281af1a3809" (UID: "e1acea1c-40f3-4414-af50-8281af1a3809"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:11:12.184753 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.184695    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1acea1c-40f3-4414-af50-8281af1a3809-istiod-ca-cert" pod "e1acea1c-40f3-4414-af50-8281af1a3809" (UID: "e1acea1c-40f3-4414-af50-8281af1a3809"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:11:12.185130 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.185103    4364 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1acea1c-40f3-4414-af50-8281af1a3809-istio-data\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:11:12.185130 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.185123    4364 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35d8c3ab-6289-40b1-ac0b-5e155420ee38-istio-token\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:11:12.185293 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.185135    4364 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1acea1c-40f3-4414-af50-8281af1a3809-workload-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:11:12.185293 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.185147    4364 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35d8c3ab-6289-40b1-ac0b-5e155420ee38-istio-podinfo\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:11:12.185293 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.185160    4364 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1acea1c-40f3-4414-af50-8281af1a3809-credential-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:11:12.185293 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.185173    4364 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1acea1c-40f3-4414-af50-8281af1a3809-workload-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:11:12.185293 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.185185    4364 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35d8c3ab-6289-40b1-ac0b-5e155420ee38-istio-envoy\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:11:12.185293 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.185197    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74zcw\" (UniqueName: \"kubernetes.io/projected/35d8c3ab-6289-40b1-ac0b-5e155420ee38-kube-api-access-74zcw\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:11:12.185293 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.185209    4364 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1acea1c-40f3-4414-af50-8281af1a3809-istiod-ca-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:11:12.186565 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.186539    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1acea1c-40f3-4414-af50-8281af1a3809-istio-token" pod "e1acea1c-40f3-4414-af50-8281af1a3809" (UID: "e1acea1c-40f3-4414-af50-8281af1a3809"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:11:12.186772 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.186696    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1acea1c-40f3-4414-af50-8281af1a3809-kube-api-access-brn6x" pod "e1acea1c-40f3-4414-af50-8281af1a3809" (UID: "e1acea1c-40f3-4414-af50-8281af1a3809"). InnerVolumeSpecName "kube-api-access-brn6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:11:12.186871 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.186852    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1acea1c-40f3-4414-af50-8281af1a3809-istio-envoy" pod "e1acea1c-40f3-4414-af50-8281af1a3809" (UID: "e1acea1c-40f3-4414-af50-8281af1a3809"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:11:12.187286 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.187272    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e1acea1c-40f3-4414-af50-8281af1a3809-istio-podinfo" pod "e1acea1c-40f3-4414-af50-8281af1a3809" (UID: "e1acea1c-40f3-4414-af50-8281af1a3809"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:11:12.286270 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.286193    4364 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1acea1c-40f3-4414-af50-8281af1a3809-istio-token\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:11:12.286270 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.286222    4364 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1acea1c-40f3-4414-af50-8281af1a3809-istio-envoy\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:11:12.286270 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.286234    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brn6x\" (UniqueName: \"kubernetes.io/projected/e1acea1c-40f3-4414-af50-8281af1a3809-kube-api-access-brn6x\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:11:12.286270 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.286250    4364 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1acea1c-40f3-4414-af50-8281af1a3809-istio-podinfo\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:11:12.543510 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.543433    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-911fe1d6-6c987d56bd-8rjs2"]
Oct 30 13:11:12.544228 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.544207    4364 cpu_manager.go:414] "containerMap: removing container" podUID="3e3c6b89-88e3-460f-8d61-ca9438b25ddb" containerName="istio-proxy"
Oct 30 13:11:12.544340 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.544229    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e3c6b89-88e3-460f-8d61-ca9438b25ddb" containerName="istio-proxy"
Oct 30 13:11:12.544340 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.544260    4364 cpu_manager.go:414] "containerMap: removing container" podUID="e1acea1c-40f3-4414-af50-8281af1a3809" containerName="istio-proxy"
Oct 30 13:11:12.544340 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.544268    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1acea1c-40f3-4414-af50-8281af1a3809" containerName="istio-proxy"
Oct 30 13:11:12.544340 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.544323    4364 cpu_manager.go:414] "containerMap: removing container" podUID="35d8c3ab-6289-40b1-ac0b-5e155420ee38" containerName="istio-proxy"
Oct 30 13:11:12.544340 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.544331    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35d8c3ab-6289-40b1-ac0b-5e155420ee38" containerName="istio-proxy"
Oct 30 13:11:12.544573 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.544501    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e3c6b89-88e3-460f-8d61-ca9438b25ddb" containerName="istio-proxy"
Oct 30 13:11:12.544573 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.544521    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1acea1c-40f3-4414-af50-8281af1a3809" containerName="istio-proxy"
Oct 30 13:11:12.544573 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.544537    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="35d8c3ab-6289-40b1-ac0b-5e155420ee38" containerName="istio-proxy"
Oct 30 13:11:12.549573 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.549531    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-911fe1d6-6c987d56bd-8rjs2"
Oct 30 13:11:12.571333 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.571297    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-911fe1d6-6c987d56bd-8rjs2"]
Oct 30 13:11:12.572357 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.572324    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-911fe1d6\""
Oct 30 13:11:12.615550 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.615524    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-911fe1d6-688c4bd5cf-6lpwc"]
Oct 30 13:11:12.619152 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.619132    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-911fe1d6-688c4bd5cf-6lpwc"
Oct 30 13:11:12.637115 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.637006    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-911fe1d6-dockercfg-c7s6b\""
Oct 30 13:11:12.643331 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.643309    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-911fe1d6-688c4bd5cf-6lpwc"]
Oct 30 13:11:12.692818 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.692795    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e45fe3cc-5929-4826-83d2-51755bcc7719-plugins-config-volume\") pod \"payload-processing-e2e-ait-911fe1d6-6c987d56bd-8rjs2\" (UID: \"e45fe3cc-5929-4826-83d2-51755bcc7719\") " pod="openshift-ingress/payload-processing-e2e-ait-911fe1d6-6c987d56bd-8rjs2"
Oct 30 13:11:12.692965 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.692943    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b84a65e-b386-4785-885c-5da526c3f2cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-911fe1d6-688c4bd5cf-6lpwc\" (UID: \"1b84a65e-b386-4785-885c-5da526c3f2cc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-911fe1d6-688c4bd5cf-6lpwc"
Oct 30 13:11:12.693019 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.693004    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87995\" (UniqueName: \"kubernetes.io/projected/1b84a65e-b386-4785-885c-5da526c3f2cc-kube-api-access-87995\") pod \"payload-pre-processing-e2e-ait-911fe1d6-688c4bd5cf-6lpwc\" (UID: \"1b84a65e-b386-4785-885c-5da526c3f2cc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-911fe1d6-688c4bd5cf-6lpwc"
Oct 30 13:11:12.693096 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.693086    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdq42\" (UniqueName: \"kubernetes.io/projected/e45fe3cc-5929-4826-83d2-51755bcc7719-kube-api-access-pdq42\") pod \"payload-processing-e2e-ait-911fe1d6-6c987d56bd-8rjs2\" (UID: \"e45fe3cc-5929-4826-83d2-51755bcc7719\") " pod="openshift-ingress/payload-processing-e2e-ait-911fe1d6-6c987d56bd-8rjs2"
Oct 30 13:11:12.806667 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.800296    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b84a65e-b386-4785-885c-5da526c3f2cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-911fe1d6-688c4bd5cf-6lpwc\" (UID: \"1b84a65e-b386-4785-885c-5da526c3f2cc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-911fe1d6-688c4bd5cf-6lpwc"
Oct 30 13:11:12.806667 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.800366    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87995\" (UniqueName: \"kubernetes.io/projected/1b84a65e-b386-4785-885c-5da526c3f2cc-kube-api-access-87995\") pod \"payload-pre-processing-e2e-ait-911fe1d6-688c4bd5cf-6lpwc\" (UID: \"1b84a65e-b386-4785-885c-5da526c3f2cc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-911fe1d6-688c4bd5cf-6lpwc"
Oct 30 13:11:12.806667 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.800504    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdq42\" (UniqueName: \"kubernetes.io/projected/e45fe3cc-5929-4826-83d2-51755bcc7719-kube-api-access-pdq42\") pod \"payload-processing-e2e-ait-911fe1d6-6c987d56bd-8rjs2\" (UID: \"e45fe3cc-5929-4826-83d2-51755bcc7719\") " pod="openshift-ingress/payload-processing-e2e-ait-911fe1d6-6c987d56bd-8rjs2"
Oct 30 13:11:12.806667 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.800711    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e45fe3cc-5929-4826-83d2-51755bcc7719-plugins-config-volume\") pod \"payload-processing-e2e-ait-911fe1d6-6c987d56bd-8rjs2\" (UID: \"e45fe3cc-5929-4826-83d2-51755bcc7719\") " pod="openshift-ingress/payload-processing-e2e-ait-911fe1d6-6c987d56bd-8rjs2"
Oct 30 13:11:12.806667 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.801763    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e45fe3cc-5929-4826-83d2-51755bcc7719-plugins-config-volume\") pod \"payload-processing-e2e-ait-911fe1d6-6c987d56bd-8rjs2\" (UID: \"e45fe3cc-5929-4826-83d2-51755bcc7719\") " pod="openshift-ingress/payload-processing-e2e-ait-911fe1d6-6c987d56bd-8rjs2"
Oct 30 13:11:12.806667 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.802521    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b84a65e-b386-4785-885c-5da526c3f2cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-911fe1d6-688c4bd5cf-6lpwc\" (UID: \"1b84a65e-b386-4785-885c-5da526c3f2cc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-911fe1d6-688c4bd5cf-6lpwc"
Oct 30 13:11:12.888512 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.870326    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2gfr2" event={"ID":"35d8c3ab-6289-40b1-ac0b-5e155420ee38","Type":"ContainerDied","Data":"88f00d746b01509005ba554d2d6da17024d39f31399fc84336be35638d5abc0d"}
Oct 30 13:11:12.888512 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.870409    4364 scope.go:122] "RemoveContainer" containerID="4af097e9d0caa680e7a29135fee164ac03fbaffa49b66e89d92706576c5be268"
Oct 30 13:11:12.888512 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.870593    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2gfr2"
Oct 30 13:11:12.888512 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.883666    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87995\" (UniqueName: \"kubernetes.io/projected/1b84a65e-b386-4785-885c-5da526c3f2cc-kube-api-access-87995\") pod \"payload-pre-processing-e2e-ait-911fe1d6-688c4bd5cf-6lpwc\" (UID: \"1b84a65e-b386-4785-885c-5da526c3f2cc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-911fe1d6-688c4bd5cf-6lpwc"
Oct 30 13:11:12.933747 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.933020    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-911fe1d6-688c4bd5cf-6lpwc"
Oct 30 13:11:12.953987 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:12.952877    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdq42\" (UniqueName: \"kubernetes.io/projected/e45fe3cc-5929-4826-83d2-51755bcc7719-kube-api-access-pdq42\") pod \"payload-processing-e2e-ait-911fe1d6-6c987d56bd-8rjs2\" (UID: \"e45fe3cc-5929-4826-83d2-51755bcc7719\") " pod="openshift-ingress/payload-processing-e2e-ait-911fe1d6-6c987d56bd-8rjs2"
Oct 30 13:11:13.016633 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:13.015434    4364 generic.go:356] "Generic (PLEG): container finished" podID="e1acea1c-40f3-4414-af50-8281af1a3809" containerID="124e5d84e38fa49aa3459495b6a769c8f3f524823289aa598a634f59a58386e0" exitCode=0
Oct 30 13:11:13.016633 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:13.016569    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f"
Oct 30 13:11:13.024702 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:13.024668    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e3c6b89-88e3-460f-8d61-ca9438b25ddb" path="/var/lib/kubelet/pods/3e3c6b89-88e3-460f-8d61-ca9438b25ddb/volumes"
Oct 30 13:11:13.025469 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:13.025441    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f" event={"ID":"e1acea1c-40f3-4414-af50-8281af1a3809","Type":"ContainerDied","Data":"124e5d84e38fa49aa3459495b6a769c8f3f524823289aa598a634f59a58386e0"}
Oct 30 13:11:13.025667 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:13.025497    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f" event={"ID":"e1acea1c-40f3-4414-af50-8281af1a3809","Type":"ContainerDied","Data":"169ace6c94b9365da9a880ae485d92672cd8d5c9b5bf5f67688a6cd7269458d9"}
Oct 30 13:11:13.041666 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:13.034186    4364 scope.go:122] "RemoveContainer" containerID="124e5d84e38fa49aa3459495b6a769c8f3f524823289aa598a634f59a58386e0"
Oct 30 13:11:13.054642 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:13.052900    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2gfr2"]
Oct 30 13:11:13.088541 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:13.088492    4364 scope.go:122] "RemoveContainer" containerID="124e5d84e38fa49aa3459495b6a769c8f3f524823289aa598a634f59a58386e0"
Oct 30 13:11:13.094080 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:11:13.094046    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"124e5d84e38fa49aa3459495b6a769c8f3f524823289aa598a634f59a58386e0\": container with ID starting with 124e5d84e38fa49aa3459495b6a769c8f3f524823289aa598a634f59a58386e0 not found: ID does not exist" containerID="124e5d84e38fa49aa3459495b6a769c8f3f524823289aa598a634f59a58386e0"
Oct 30 13:11:13.094351 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:13.094319    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"124e5d84e38fa49aa3459495b6a769c8f3f524823289aa598a634f59a58386e0"} err="failed to get container status \"124e5d84e38fa49aa3459495b6a769c8f3f524823289aa598a634f59a58386e0\": rpc error: code = NotFound desc = could not find container \"124e5d84e38fa49aa3459495b6a769c8f3f524823289aa598a634f59a58386e0\": container with ID starting with 124e5d84e38fa49aa3459495b6a769c8f3f524823289aa598a634f59a58386e0 not found: ID does not exist"
Oct 30 13:11:13.120630 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:13.120554    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2gfr2"]
Oct 30 13:11:13.140271 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:13.140227    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f"]
Oct 30 13:11:13.148941 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:13.148910    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-b31c2926-openshift-default-8674d596b9-pkv7f"]
Oct 30 13:11:13.170878 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:13.170849    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-911fe1d6-6c987d56bd-8rjs2"
Oct 30 13:11:13.436200 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:11:13.436171    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b84a65e_b386_4785_885c_5da526c3f2cc.slice/crio-523938f1abc0fe139a980c5adab3f3910250cdde56fb6246afcdb882718bf808 WatchSource:0}: Error finding container 523938f1abc0fe139a980c5adab3f3910250cdde56fb6246afcdb882718bf808: Status 404 returned error can't find the container with id 523938f1abc0fe139a980c5adab3f3910250cdde56fb6246afcdb882718bf808
Oct 30 13:11:13.437797 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:13.437773    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-911fe1d6-688c4bd5cf-6lpwc"]
Oct 30 13:11:13.497201 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:11:13.497165    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode45fe3cc_5929_4826_83d2_51755bcc7719.slice/crio-ae97085c59a1c22a229ff87592bf2026c4557fc9187f95c27c0dc83d57e8ddc3 WatchSource:0}: Error finding container ae97085c59a1c22a229ff87592bf2026c4557fc9187f95c27c0dc83d57e8ddc3: Status 404 returned error can't find the container with id ae97085c59a1c22a229ff87592bf2026c4557fc9187f95c27c0dc83d57e8ddc3
Oct 30 13:11:13.497563 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:13.497542    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-911fe1d6-6c987d56bd-8rjs2"]
Oct 30 13:11:14.028299 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:14.028265    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-911fe1d6-6c987d56bd-8rjs2" event={"ID":"e45fe3cc-5929-4826-83d2-51755bcc7719","Type":"ContainerStarted","Data":"0e883c524a0f2ca2cef7a2a2d0f22095284812ae3d9fd3742b6890d17f234c66"}
Oct 30 13:11:14.028856 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:14.028320    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-911fe1d6-6c987d56bd-8rjs2" event={"ID":"e45fe3cc-5929-4826-83d2-51755bcc7719","Type":"ContainerStarted","Data":"ae97085c59a1c22a229ff87592bf2026c4557fc9187f95c27c0dc83d57e8ddc3"}
Oct 30 13:11:14.028856 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:14.028351    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-911fe1d6-6c987d56bd-8rjs2"
Oct 30 13:11:14.031727 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:14.031704    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-911fe1d6-688c4bd5cf-6lpwc" event={"ID":"1b84a65e-b386-4785-885c-5da526c3f2cc","Type":"ContainerStarted","Data":"db1be6e705be61c34246e47e81a51553959dd0bbba2d0d9cf621d8f8eb46e754"}
Oct 30 13:11:14.031843 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:14.031751    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-911fe1d6-688c4bd5cf-6lpwc" event={"ID":"1b84a65e-b386-4785-885c-5da526c3f2cc","Type":"ContainerStarted","Data":"523938f1abc0fe139a980c5adab3f3910250cdde56fb6246afcdb882718bf808"}
Oct 30 13:11:14.031843 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:14.031773    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-911fe1d6-688c4bd5cf-6lpwc"
Oct 30 13:11:14.098620 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:14.098526    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-911fe1d6-6c987d56bd-8rjs2" podStartSLOduration=2.098507959 podStartE2EDuration="2.098507959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:14.081795723 +0000 UTC m=+1173.248160372" watchObservedRunningTime="2026-10-30 13:11:14.098507959 +0000 UTC m=+1173.264872606"
Oct 30 13:11:14.128513 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:14.128464    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-911fe1d6-688c4bd5cf-6lpwc" podStartSLOduration=2.128452037 podStartE2EDuration="2.128452037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:14.126112215 +0000 UTC m=+1173.292476864" watchObservedRunningTime="2026-10-30 13:11:14.128452037 +0000 UTC m=+1173.294816730"
Oct 30 13:11:14.189169 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:14.189142    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-45c4d25f-78488bfdf8-5ksbg"]
Oct 30 13:11:14.192741 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:14.192720    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-45c4d25f-78488bfdf8-5ksbg"
Oct 30 13:11:14.212312 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:14.212287    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-45c4d25f-dockercfg-glzcq\""
Oct 30 13:11:14.212440 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:14.212416    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-45c4d25f\""
Oct 30 13:11:14.213260 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:14.213239    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-45c4d25f-78488bfdf8-5ksbg"]
Oct 30 13:11:14.238638 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:14.238593    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-45c4d25f-767667f47-c6lxc"]
Oct 30 13:11:14.243504 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:14.243483    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-45c4d25f-767667f47-c6lxc"
Oct 30 13:11:14.264546 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:14.264524    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-45c4d25f-767667f47-c6lxc"]
Oct 30 13:11:14.412172 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:14.412138    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d1a108a-ac9f-472e-902a-852faa6a4b11-plugins-config-volume\") pod \"payload-processing-e2e-derive-45c4d25f-78488bfdf8-5ksbg\" (UID: \"9d1a108a-ac9f-472e-902a-852faa6a4b11\") " pod="openshift-ingress/payload-processing-e2e-derive-45c4d25f-78488bfdf8-5ksbg"
Oct 30 13:11:14.412629 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:14.412584    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lgpj\" (UniqueName: \"kubernetes.io/projected/9d1a108a-ac9f-472e-902a-852faa6a4b11-kube-api-access-7lgpj\") pod \"payload-processing-e2e-derive-45c4d25f-78488bfdf8-5ksbg\" (UID: \"9d1a108a-ac9f-472e-902a-852faa6a4b11\") " pod="openshift-ingress/payload-processing-e2e-derive-45c4d25f-78488bfdf8-5ksbg"
Oct 30 13:11:14.412815 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:14.412798    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e38332a6-07be-4d52-a06d-bcc49e8e3cf2-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-45c4d25f-767667f47-c6lxc\" (UID: \"e38332a6-07be-4d52-a06d-bcc49e8e3cf2\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-45c4d25f-767667f47-c6lxc"
Oct 30 13:11:14.412957 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:14.412942    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b877\" (UniqueName: \"kubernetes.io/projected/e38332a6-07be-4d52-a06d-bcc49e8e3cf2-kube-api-access-4b877\") pod \"payload-pre-processing-e2e-derive-45c4d25f-767667f47-c6lxc\" (UID: \"e38332a6-07be-4d52-a06d-bcc49e8e3cf2\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-45c4d25f-767667f47-c6lxc"
Oct 30 13:11:14.567621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:14.521075    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lgpj\" (UniqueName: \"kubernetes.io/projected/9d1a108a-ac9f-472e-902a-852faa6a4b11-kube-api-access-7lgpj\") pod \"payload-processing-e2e-derive-45c4d25f-78488bfdf8-5ksbg\" (UID: \"9d1a108a-ac9f-472e-902a-852faa6a4b11\") " pod="openshift-ingress/payload-processing-e2e-derive-45c4d25f-78488bfdf8-5ksbg"
Oct 30 13:11:14.569311 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:14.569268    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e38332a6-07be-4d52-a06d-bcc49e8e3cf2-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-45c4d25f-767667f47-c6lxc\" (UID: \"e38332a6-07be-4d52-a06d-bcc49e8e3cf2\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-45c4d25f-767667f47-c6lxc"
Oct 30 13:11:14.569563 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:14.569530    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b877\" (UniqueName: \"kubernetes.io/projected/e38332a6-07be-4d52-a06d-bcc49e8e3cf2-kube-api-access-4b877\") pod \"payload-pre-processing-e2e-derive-45c4d25f-767667f47-c6lxc\" (UID: \"e38332a6-07be-4d52-a06d-bcc49e8e3cf2\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-45c4d25f-767667f47-c6lxc"
Oct 30 13:11:14.570070 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:14.570034    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d1a108a-ac9f-472e-902a-852faa6a4b11-plugins-config-volume\") pod \"payload-processing-e2e-derive-45c4d25f-78488bfdf8-5ksbg\" (UID: \"9d1a108a-ac9f-472e-902a-852faa6a4b11\") " pod="openshift-ingress/payload-processing-e2e-derive-45c4d25f-78488bfdf8-5ksbg"
Oct 30 13:11:14.597620 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:14.571492    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d1a108a-ac9f-472e-902a-852faa6a4b11-plugins-config-volume\") pod \"payload-processing-e2e-derive-45c4d25f-78488bfdf8-5ksbg\" (UID: \"9d1a108a-ac9f-472e-902a-852faa6a4b11\") " pod="openshift-ingress/payload-processing-e2e-derive-45c4d25f-78488bfdf8-5ksbg"
Oct 30 13:11:14.597620 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:14.575993    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e38332a6-07be-4d52-a06d-bcc49e8e3cf2-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-45c4d25f-767667f47-c6lxc\" (UID: \"e38332a6-07be-4d52-a06d-bcc49e8e3cf2\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-45c4d25f-767667f47-c6lxc"
Oct 30 13:11:14.626923 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:14.626849    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b877\" (UniqueName: \"kubernetes.io/projected/e38332a6-07be-4d52-a06d-bcc49e8e3cf2-kube-api-access-4b877\") pod \"payload-pre-processing-e2e-derive-45c4d25f-767667f47-c6lxc\" (UID: \"e38332a6-07be-4d52-a06d-bcc49e8e3cf2\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-45c4d25f-767667f47-c6lxc"
Oct 30 13:11:14.670676 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:14.669718    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lgpj\" (UniqueName: \"kubernetes.io/projected/9d1a108a-ac9f-472e-902a-852faa6a4b11-kube-api-access-7lgpj\") pod \"payload-processing-e2e-derive-45c4d25f-78488bfdf8-5ksbg\" (UID: \"9d1a108a-ac9f-472e-902a-852faa6a4b11\") " pod="openshift-ingress/payload-processing-e2e-derive-45c4d25f-78488bfdf8-5ksbg"
Oct 30 13:11:14.806391 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:14.806362    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-45c4d25f-78488bfdf8-5ksbg"
Oct 30 13:11:14.863451 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:14.863423    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-45c4d25f-767667f47-c6lxc"
Oct 30 13:11:14.979461 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:14.979425    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35d8c3ab-6289-40b1-ac0b-5e155420ee38" path="/var/lib/kubelet/pods/35d8c3ab-6289-40b1-ac0b-5e155420ee38/volumes"
Oct 30 13:11:14.980276 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:14.980255    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1acea1c-40f3-4414-af50-8281af1a3809" path="/var/lib/kubelet/pods/e1acea1c-40f3-4414-af50-8281af1a3809/volumes"
Oct 30 13:11:15.115652 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:11:15.115568    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d1a108a_ac9f_472e_902a_852faa6a4b11.slice/crio-c4171acbfb6ed3705a3aba65af26e3af6d518f6edd3d3c979506aa2aae0622c9 WatchSource:0}: Error finding container c4171acbfb6ed3705a3aba65af26e3af6d518f6edd3d3c979506aa2aae0622c9: Status 404 returned error can't find the container with id c4171acbfb6ed3705a3aba65af26e3af6d518f6edd3d3c979506aa2aae0622c9
Oct 30 13:11:15.117497 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:15.117464    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-45c4d25f-78488bfdf8-5ksbg"]
Oct 30 13:11:15.194926 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:11:15.194902    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode38332a6_07be_4d52_a06d_bcc49e8e3cf2.slice/crio-ccd707817530caee97189dcc3c5b6e4d7be30018de46df9eab62d03bd18fd875 WatchSource:0}: Error finding container ccd707817530caee97189dcc3c5b6e4d7be30018de46df9eab62d03bd18fd875: Status 404 returned error can't find the container with id ccd707817530caee97189dcc3c5b6e4d7be30018de46df9eab62d03bd18fd875
Oct 30 13:11:15.197095 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:15.197062    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-45c4d25f-767667f47-c6lxc"]
Oct 30 13:11:16.053582 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:16.053547    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-45c4d25f-767667f47-c6lxc" event={"ID":"e38332a6-07be-4d52-a06d-bcc49e8e3cf2","Type":"ContainerStarted","Data":"a92f6b314ac1b7ee568bea6625fe931ad2eb8ab0acfcd9df16cca3befece3b46"}
Oct 30 13:11:16.053801 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:16.053639    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-45c4d25f-767667f47-c6lxc"
Oct 30 13:11:16.053801 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:16.053660    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-45c4d25f-767667f47-c6lxc" event={"ID":"e38332a6-07be-4d52-a06d-bcc49e8e3cf2","Type":"ContainerStarted","Data":"ccd707817530caee97189dcc3c5b6e4d7be30018de46df9eab62d03bd18fd875"}
Oct 30 13:11:16.055288 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:16.055251    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-45c4d25f-78488bfdf8-5ksbg" event={"ID":"9d1a108a-ac9f-472e-902a-852faa6a4b11","Type":"ContainerStarted","Data":"90871ae811312ccee77ca8863f25776ef43a2b2514fa392ebdd2f84f5ab6e854"}
Oct 30 13:11:16.055392 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:16.055302    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-45c4d25f-78488bfdf8-5ksbg" event={"ID":"9d1a108a-ac9f-472e-902a-852faa6a4b11","Type":"ContainerStarted","Data":"c4171acbfb6ed3705a3aba65af26e3af6d518f6edd3d3c979506aa2aae0622c9"}
Oct 30 13:11:16.055392 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:16.055330    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-45c4d25f-78488bfdf8-5ksbg"
Oct 30 13:11:16.099775 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:16.099726    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-45c4d25f-767667f47-c6lxc" podStartSLOduration=2.099712879 podStartE2EDuration="2.099712879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:16.098062182 +0000 UTC m=+1175.264426830" watchObservedRunningTime="2026-10-30 13:11:16.099712879 +0000 UTC m=+1175.266077526"
Oct 30 13:11:16.143996 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:16.143956    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-45c4d25f-78488bfdf8-5ksbg" podStartSLOduration=2.143944223 podStartE2EDuration="2.143944223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:16.14124749 +0000 UTC m=+1175.307612138" watchObservedRunningTime="2026-10-30 13:11:16.143944223 +0000 UTC m=+1175.310308892"
Oct 30 13:11:16.853533 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:16.853502    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:11:19.692739 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:19.692686    4364 patch_prober.go:29] interesting pod/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:11:19.693357 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:19.693304    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb" podUID="59c001b5-b0f7-4898-ba90-aa8a0b96a6e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:23.019207 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:23.019177    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8b77b0-75f95c8744-psdzn"
Oct 30 13:11:23.020357 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:23.020335    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8b77b0-d46dbfdc-lkc4n"
Oct 30 13:11:23.464629 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:23.464571    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8b77b0-5f7b7ff96f-wggzj"]
Oct 30 13:11:23.468468 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:23.468451    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8b77b0-5f7b7ff96f-wggzj"
Oct 30 13:11:23.486408 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:23.486385    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-8b77b0-serving-cert\""
Oct 30 13:11:23.491373 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:23.491350    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8b77b0-5f7b7ff96f-wggzj"]
Oct 30 13:11:23.530099 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:23.530072    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb144ffb-880b-44cb-ac19-bbd878eb3bff-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-8b77b0-5f7b7ff96f-wggzj\" (UID: \"eb144ffb-880b-44cb-ac19-bbd878eb3bff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8b77b0-5f7b7ff96f-wggzj"
Oct 30 13:11:23.530256 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:23.530198    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb144ffb-880b-44cb-ac19-bbd878eb3bff-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-8b77b0-5f7b7ff96f-wggzj\" (UID: \"eb144ffb-880b-44cb-ac19-bbd878eb3bff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8b77b0-5f7b7ff96f-wggzj"
Oct 30 13:11:23.530325 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:23.530279    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs5gj\" (UniqueName: \"kubernetes.io/projected/eb144ffb-880b-44cb-ac19-bbd878eb3bff-kube-api-access-zs5gj\") pod \"maas-api-e2e-shared-a-w2-8b77b0-5f7b7ff96f-wggzj\" (UID: \"eb144ffb-880b-44cb-ac19-bbd878eb3bff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8b77b0-5f7b7ff96f-wggzj"
Oct 30 13:11:23.530413 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:23.530398    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb144ffb-880b-44cb-ac19-bbd878eb3bff-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-8b77b0-5f7b7ff96f-wggzj\" (UID: \"eb144ffb-880b-44cb-ac19-bbd878eb3bff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8b77b0-5f7b7ff96f-wggzj"
Oct 30 13:11:23.631967 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:23.631940    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb144ffb-880b-44cb-ac19-bbd878eb3bff-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-8b77b0-5f7b7ff96f-wggzj\" (UID: \"eb144ffb-880b-44cb-ac19-bbd878eb3bff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8b77b0-5f7b7ff96f-wggzj"
Oct 30 13:11:23.632111 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:23.632053    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb144ffb-880b-44cb-ac19-bbd878eb3bff-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-8b77b0-5f7b7ff96f-wggzj\" (UID: \"eb144ffb-880b-44cb-ac19-bbd878eb3bff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8b77b0-5f7b7ff96f-wggzj"
Oct 30 13:11:23.632111 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:11:23.632079    4364 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8b77b0-serving-cert: secret "maas-api-e2e-shared-a-w2-8b77b0-serving-cert" not found
Oct 30 13:11:23.632238 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:23.632134    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs5gj\" (UniqueName: \"kubernetes.io/projected/eb144ffb-880b-44cb-ac19-bbd878eb3bff-kube-api-access-zs5gj\") pod \"maas-api-e2e-shared-a-w2-8b77b0-5f7b7ff96f-wggzj\" (UID: \"eb144ffb-880b-44cb-ac19-bbd878eb3bff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8b77b0-5f7b7ff96f-wggzj"
Oct 30 13:11:23.632238 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:11:23.632162    4364 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb144ffb-880b-44cb-ac19-bbd878eb3bff-maas-api-tls podName:eb144ffb-880b-44cb-ac19-bbd878eb3bff nodeName:}" failed. No retries permitted until 2026-10-30 13:11:24.132146503 +0000 UTC m=+1183.298511135 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/eb144ffb-880b-44cb-ac19-bbd878eb3bff-maas-api-tls") pod "maas-api-e2e-shared-a-w2-8b77b0-5f7b7ff96f-wggzj" (UID: "eb144ffb-880b-44cb-ac19-bbd878eb3bff") : secret "maas-api-e2e-shared-a-w2-8b77b0-serving-cert" not found
Oct 30 13:11:23.632349 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:23.632295    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb144ffb-880b-44cb-ac19-bbd878eb3bff-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-8b77b0-5f7b7ff96f-wggzj\" (UID: \"eb144ffb-880b-44cb-ac19-bbd878eb3bff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8b77b0-5f7b7ff96f-wggzj"
Oct 30 13:11:23.632771 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:23.632750    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb144ffb-880b-44cb-ac19-bbd878eb3bff-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-8b77b0-5f7b7ff96f-wggzj\" (UID: \"eb144ffb-880b-44cb-ac19-bbd878eb3bff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8b77b0-5f7b7ff96f-wggzj"
Oct 30 13:11:23.634860 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:23.634840    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb144ffb-880b-44cb-ac19-bbd878eb3bff-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-8b77b0-5f7b7ff96f-wggzj\" (UID: \"eb144ffb-880b-44cb-ac19-bbd878eb3bff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8b77b0-5f7b7ff96f-wggzj"
Oct 30 13:11:23.697668 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:23.688414    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs5gj\" (UniqueName: \"kubernetes.io/projected/eb144ffb-880b-44cb-ac19-bbd878eb3bff-kube-api-access-zs5gj\") pod \"maas-api-e2e-shared-a-w2-8b77b0-5f7b7ff96f-wggzj\" (UID: \"eb144ffb-880b-44cb-ac19-bbd878eb3bff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8b77b0-5f7b7ff96f-wggzj"
Oct 30 13:11:24.141342 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:24.141304    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb144ffb-880b-44cb-ac19-bbd878eb3bff-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-8b77b0-5f7b7ff96f-wggzj\" (UID: \"eb144ffb-880b-44cb-ac19-bbd878eb3bff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8b77b0-5f7b7ff96f-wggzj"
Oct 30 13:11:24.143723 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:24.143700    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb144ffb-880b-44cb-ac19-bbd878eb3bff-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-8b77b0-5f7b7ff96f-wggzj\" (UID: \"eb144ffb-880b-44cb-ac19-bbd878eb3bff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8b77b0-5f7b7ff96f-wggzj"
Oct 30 13:11:24.382624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:24.382581    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8b77b0-5f7b7ff96f-wggzj"
Oct 30 13:11:24.722973 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:11:24.722939    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb144ffb_880b_44cb_ac19_bbd878eb3bff.slice/crio-2d39dd89d030028abf8150d90848e60f3b21a49ed661b05be89d11c2c22d60c7 WatchSource:0}: Error finding container 2d39dd89d030028abf8150d90848e60f3b21a49ed661b05be89d11c2c22d60c7: Status 404 returned error can't find the container with id 2d39dd89d030028abf8150d90848e60f3b21a49ed661b05be89d11c2c22d60c7
Oct 30 13:11:24.724533 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:24.724507    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8b77b0-5f7b7ff96f-wggzj"]
Oct 30 13:11:25.089674 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:25.085978    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-911fe1d6-688c4bd5cf-6lpwc"
Oct 30 13:11:25.089674 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:25.087854    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-911fe1d6-6c987d56bd-8rjs2"
Oct 30 13:11:25.199243 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:25.199207    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8b77b0-5f7b7ff96f-wggzj" event={"ID":"eb144ffb-880b-44cb-ac19-bbd878eb3bff","Type":"ContainerStarted","Data":"2d39dd89d030028abf8150d90848e60f3b21a49ed661b05be89d11c2c22d60c7"}
Oct 30 13:11:25.199682 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:25.199273    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8b77b0-5f7b7ff96f-wggzj"
Oct 30 13:11:25.250292 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:25.250245    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8b77b0-5f7b7ff96f-wggzj" podStartSLOduration=1.864626979 podStartE2EDuration="2.250231198s" totalImagesPullingTime="385.604219ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:25.248227115 +0000 UTC m=+1184.414591784" watchObservedRunningTime="2026-10-30 13:11:25.250231198 +0000 UTC m=+1184.416595847"
Oct 30 13:11:26.215393 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:26.215356    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8b77b0-5f7b7ff96f-wggzj" event={"ID":"eb144ffb-880b-44cb-ac19-bbd878eb3bff","Type":"ContainerStarted","Data":"b31895906ec859a4e4800a143ad66c2e83e3afb043fec3f536201a930184b7a3"}
Oct 30 13:11:27.066573 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:27.066546    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-45c4d25f-767667f47-c6lxc"
Oct 30 13:11:27.066747 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:27.066643    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-45c4d25f-78488bfdf8-5ksbg"
Oct 30 13:11:31.193246 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:31.193215    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-45c4d25f-5b7f7bb64b-lpqj5"]
Oct 30 13:11:31.196754 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:31.196735    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-45c4d25f-5b7f7bb64b-lpqj5"
Oct 30 13:11:31.217861 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:31.217834    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-45c4d25f-serving-cert\""
Oct 30 13:11:31.226223 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:31.226196    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-45c4d25f-5b7f7bb64b-lpqj5"]
Oct 30 13:11:31.243689 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:31.243665    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/002fc4fb-5d07-4c13-ab09-0a8d7d489003-maas-api-tls\") pod \"maas-api-e2e-derive-45c4d25f-5b7f7bb64b-lpqj5\" (UID: \"002fc4fb-5d07-4c13-ab09-0a8d7d489003\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-45c4d25f-5b7f7bb64b-lpqj5"
Oct 30 13:11:31.243850 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:31.243734    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zpmk\" (UniqueName: \"kubernetes.io/projected/002fc4fb-5d07-4c13-ab09-0a8d7d489003-kube-api-access-6zpmk\") pod \"maas-api-e2e-derive-45c4d25f-5b7f7bb64b-lpqj5\" (UID: \"002fc4fb-5d07-4c13-ab09-0a8d7d489003\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-45c4d25f-5b7f7bb64b-lpqj5"
Oct 30 13:11:31.244074 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:31.243992    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/002fc4fb-5d07-4c13-ab09-0a8d7d489003-metrics-certs\") pod \"maas-api-e2e-derive-45c4d25f-5b7f7bb64b-lpqj5\" (UID: \"002fc4fb-5d07-4c13-ab09-0a8d7d489003\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-45c4d25f-5b7f7bb64b-lpqj5"
Oct 30 13:11:31.244173 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:31.244140    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/002fc4fb-5d07-4c13-ab09-0a8d7d489003-openshift-service-ca\") pod \"maas-api-e2e-derive-45c4d25f-5b7f7bb64b-lpqj5\" (UID: \"002fc4fb-5d07-4c13-ab09-0a8d7d489003\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-45c4d25f-5b7f7bb64b-lpqj5"
Oct 30 13:11:31.345526 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:31.345494    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/002fc4fb-5d07-4c13-ab09-0a8d7d489003-metrics-certs\") pod \"maas-api-e2e-derive-45c4d25f-5b7f7bb64b-lpqj5\" (UID: \"002fc4fb-5d07-4c13-ab09-0a8d7d489003\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-45c4d25f-5b7f7bb64b-lpqj5"
Oct 30 13:11:31.345711 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:31.345582    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/002fc4fb-5d07-4c13-ab09-0a8d7d489003-openshift-service-ca\") pod \"maas-api-e2e-derive-45c4d25f-5b7f7bb64b-lpqj5\" (UID: \"002fc4fb-5d07-4c13-ab09-0a8d7d489003\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-45c4d25f-5b7f7bb64b-lpqj5"
Oct 30 13:11:31.345711 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:31.345697    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/002fc4fb-5d07-4c13-ab09-0a8d7d489003-maas-api-tls\") pod \"maas-api-e2e-derive-45c4d25f-5b7f7bb64b-lpqj5\" (UID: \"002fc4fb-5d07-4c13-ab09-0a8d7d489003\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-45c4d25f-5b7f7bb64b-lpqj5"
Oct 30 13:11:31.345802 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:31.345745    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zpmk\" (UniqueName: \"kubernetes.io/projected/002fc4fb-5d07-4c13-ab09-0a8d7d489003-kube-api-access-6zpmk\") pod \"maas-api-e2e-derive-45c4d25f-5b7f7bb64b-lpqj5\" (UID: \"002fc4fb-5d07-4c13-ab09-0a8d7d489003\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-45c4d25f-5b7f7bb64b-lpqj5"
Oct 30 13:11:31.346346 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:31.346326    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/002fc4fb-5d07-4c13-ab09-0a8d7d489003-openshift-service-ca\") pod \"maas-api-e2e-derive-45c4d25f-5b7f7bb64b-lpqj5\" (UID: \"002fc4fb-5d07-4c13-ab09-0a8d7d489003\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-45c4d25f-5b7f7bb64b-lpqj5"
Oct 30 13:11:31.347937 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:31.347912    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/002fc4fb-5d07-4c13-ab09-0a8d7d489003-metrics-certs\") pod \"maas-api-e2e-derive-45c4d25f-5b7f7bb64b-lpqj5\" (UID: \"002fc4fb-5d07-4c13-ab09-0a8d7d489003\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-45c4d25f-5b7f7bb64b-lpqj5"
Oct 30 13:11:31.348045 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:31.348024    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/002fc4fb-5d07-4c13-ab09-0a8d7d489003-maas-api-tls\") pod \"maas-api-e2e-derive-45c4d25f-5b7f7bb64b-lpqj5\" (UID: \"002fc4fb-5d07-4c13-ab09-0a8d7d489003\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-45c4d25f-5b7f7bb64b-lpqj5"
Oct 30 13:11:31.371197 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:31.371175    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zpmk\" (UniqueName: \"kubernetes.io/projected/002fc4fb-5d07-4c13-ab09-0a8d7d489003-kube-api-access-6zpmk\") pod \"maas-api-e2e-derive-45c4d25f-5b7f7bb64b-lpqj5\" (UID: \"002fc4fb-5d07-4c13-ab09-0a8d7d489003\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-45c4d25f-5b7f7bb64b-lpqj5"
Oct 30 13:11:31.513995 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:31.513908    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-45c4d25f-5b7f7bb64b-lpqj5"
Oct 30 13:11:31.860656 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:11:31.860591    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod002fc4fb_5d07_4c13_ab09_0a8d7d489003.slice/crio-5d38c9a7008502af150584fbc3490495a01c7b0a391eedd1e656d9bbc9fe5dbe WatchSource:0}: Error finding container 5d38c9a7008502af150584fbc3490495a01c7b0a391eedd1e656d9bbc9fe5dbe: Status 404 returned error can't find the container with id 5d38c9a7008502af150584fbc3490495a01c7b0a391eedd1e656d9bbc9fe5dbe
Oct 30 13:11:31.862308 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:31.862278    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-45c4d25f-5b7f7bb64b-lpqj5"]
Oct 30 13:11:32.209258 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:32.209233    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g6bnr"]
Oct 30 13:11:32.213321 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:32.213302    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g6bnr"
Oct 30 13:11:32.231470 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:32.231028    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8b77b0-5f7b7ff96f-wggzj"
Oct 30 13:11:32.242641 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:32.240143    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g6bnr"]
Oct 30 13:11:32.272351 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:32.272328    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ea01099-fc92-469c-99cf-875275e18a04-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g6bnr\" (UID: \"8ea01099-fc92-469c-99cf-875275e18a04\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g6bnr"
Oct 30 13:11:32.272506 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:32.272486    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8ea01099-fc92-469c-99cf-875275e18a04-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g6bnr\" (UID: \"8ea01099-fc92-469c-99cf-875275e18a04\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g6bnr"
Oct 30 13:11:32.272588 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:32.272549    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8ea01099-fc92-469c-99cf-875275e18a04-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g6bnr\" (UID: \"8ea01099-fc92-469c-99cf-875275e18a04\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g6bnr"
Oct 30 13:11:32.272798 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:32.272780    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ea01099-fc92-469c-99cf-875275e18a04-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g6bnr\" (UID: \"8ea01099-fc92-469c-99cf-875275e18a04\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g6bnr"
Oct 30 13:11:32.272857 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:32.272813    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8ea01099-fc92-469c-99cf-875275e18a04-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g6bnr\" (UID: \"8ea01099-fc92-469c-99cf-875275e18a04\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g6bnr"
Oct 30 13:11:32.272944 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:32.272929    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7kc4\" (UniqueName: \"kubernetes.io/projected/8ea01099-fc92-469c-99cf-875275e18a04-kube-api-access-k7kc4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g6bnr\" (UID: \"8ea01099-fc92-469c-99cf-875275e18a04\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g6bnr"
Oct 30 13:11:32.273002 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:32.272967    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8ea01099-fc92-469c-99cf-875275e18a04-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g6bnr\" (UID: \"8ea01099-fc92-469c-99cf-875275e18a04\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g6bnr"
Oct 30 13:11:32.282307 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:32.282258    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-45c4d25f-5b7f7bb64b-lpqj5" event={"ID":"002fc4fb-5d07-4c13-ab09-0a8d7d489003","Type":"ContainerStarted","Data":"f6b64e111835331f5cd8375c0626c5f0408805dc78af22c99cf564a1718b2621"}
Oct 30 13:11:32.282463 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:32.282318    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-45c4d25f-5b7f7bb64b-lpqj5" event={"ID":"002fc4fb-5d07-4c13-ab09-0a8d7d489003","Type":"ContainerStarted","Data":"5d38c9a7008502af150584fbc3490495a01c7b0a391eedd1e656d9bbc9fe5dbe"}
Oct 30 13:11:32.282463 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:32.282374    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-45c4d25f-5b7f7bb64b-lpqj5"
Oct 30 13:11:32.367552 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:32.367493    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-45c4d25f-5b7f7bb64b-lpqj5" podStartSLOduration=1.042241795 podStartE2EDuration="1.367474194s" totalImagesPullingTime="325.232399ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:32.361846561 +0000 UTC m=+1191.528211212" watchObservedRunningTime="2026-10-30 13:11:32.367474194 +0000 UTC m=+1191.533838843"
Oct 30 13:11:32.374948 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:32.374916    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ea01099-fc92-469c-99cf-875275e18a04-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g6bnr\" (UID: \"8ea01099-fc92-469c-99cf-875275e18a04\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g6bnr"
Oct 30 13:11:32.375085 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:32.374962    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8ea01099-fc92-469c-99cf-875275e18a04-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g6bnr\" (UID: \"8ea01099-fc92-469c-99cf-875275e18a04\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g6bnr"
Oct 30 13:11:32.375085 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:32.375052    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7kc4\" (UniqueName: \"kubernetes.io/projected/8ea01099-fc92-469c-99cf-875275e18a04-kube-api-access-k7kc4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g6bnr\" (UID: \"8ea01099-fc92-469c-99cf-875275e18a04\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g6bnr"
Oct 30 13:11:32.375217 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:32.375091    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8ea01099-fc92-469c-99cf-875275e18a04-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g6bnr\" (UID: \"8ea01099-fc92-469c-99cf-875275e18a04\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g6bnr"
Oct 30 13:11:32.375278 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:32.375224    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ea01099-fc92-469c-99cf-875275e18a04-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g6bnr\" (UID: \"8ea01099-fc92-469c-99cf-875275e18a04\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g6bnr"
Oct 30 13:11:32.375393 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:32.375373    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8ea01099-fc92-469c-99cf-875275e18a04-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g6bnr\" (UID: \"8ea01099-fc92-469c-99cf-875275e18a04\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g6bnr"
Oct 30 13:11:32.375467 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:32.375440    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8ea01099-fc92-469c-99cf-875275e18a04-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g6bnr\" (UID: \"8ea01099-fc92-469c-99cf-875275e18a04\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g6bnr"
Oct 30 13:11:32.377036 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:32.377009    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8ea01099-fc92-469c-99cf-875275e18a04-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g6bnr\" (UID: \"8ea01099-fc92-469c-99cf-875275e18a04\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g6bnr"
Oct 30 13:11:32.379697 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:32.379676    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8ea01099-fc92-469c-99cf-875275e18a04-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g6bnr\" (UID: \"8ea01099-fc92-469c-99cf-875275e18a04\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g6bnr"
Oct 30 13:11:32.379846 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:32.379824    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8ea01099-fc92-469c-99cf-875275e18a04-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g6bnr\" (UID: \"8ea01099-fc92-469c-99cf-875275e18a04\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g6bnr"
Oct 30 13:11:32.385651 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:32.382461    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ea01099-fc92-469c-99cf-875275e18a04-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g6bnr\" (UID: \"8ea01099-fc92-469c-99cf-875275e18a04\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g6bnr"
Oct 30 13:11:32.385651 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:32.383873    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8ea01099-fc92-469c-99cf-875275e18a04-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g6bnr\" (UID: \"8ea01099-fc92-469c-99cf-875275e18a04\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g6bnr"
Oct 30 13:11:32.400578 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:32.400554    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7kc4\" (UniqueName: \"kubernetes.io/projected/8ea01099-fc92-469c-99cf-875275e18a04-kube-api-access-k7kc4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g6bnr\" (UID: \"8ea01099-fc92-469c-99cf-875275e18a04\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g6bnr"
Oct 30 13:11:32.401418 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:32.401380    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ea01099-fc92-469c-99cf-875275e18a04-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g6bnr\" (UID: \"8ea01099-fc92-469c-99cf-875275e18a04\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g6bnr"
Oct 30 13:11:32.561583 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:32.557889    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g6bnr"
Oct 30 13:11:33.044306 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:11:33.044280    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ea01099_fc92_469c_99cf_875275e18a04.slice/crio-6178e41a64c9015caea6a7836ae453146c1fdad2993bec42707e14fcb425aa62 WatchSource:0}: Error finding container 6178e41a64c9015caea6a7836ae453146c1fdad2993bec42707e14fcb425aa62: Status 404 returned error can't find the container with id 6178e41a64c9015caea6a7836ae453146c1fdad2993bec42707e14fcb425aa62
Oct 30 13:11:33.046961 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:33.046930    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g6bnr"]
Oct 30 13:11:33.047250 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:33.047223    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:11:33.047294 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:33.047280    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:11:33.294619 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:33.294513    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g6bnr" event={"ID":"8ea01099-fc92-469c-99cf-875275e18a04","Type":"ContainerStarted","Data":"89e81246db246657fb88dff1f180d8d242e7f7aac9dd433248e808036c7e0e90"}
Oct 30 13:11:33.294619 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:33.294590    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g6bnr" event={"ID":"8ea01099-fc92-469c-99cf-875275e18a04","Type":"ContainerStarted","Data":"6178e41a64c9015caea6a7836ae453146c1fdad2993bec42707e14fcb425aa62"}
Oct 30 13:11:33.295103 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:33.294630    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g6bnr"
Oct 30 13:11:33.345473 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:33.345424    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g6bnr" podStartSLOduration=1.3454086570000001 podStartE2EDuration="1.345408657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:33.344038849 +0000 UTC m=+1192.510403497" watchObservedRunningTime="2026-10-30 13:11:33.345408657 +0000 UTC m=+1192.511773304"
Oct 30 13:11:34.305856 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:34.305825    4364 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-g6bnr container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:11:34.306307 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:34.305877    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g6bnr" podUID="8ea01099-fc92-469c-99cf-875275e18a04" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:11:35.114670 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:35.111681    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-pjfhj"]
Oct 30 13:11:35.130340 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:35.130305    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-pjfhj"]
Oct 30 13:11:35.130680 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:35.130661    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-pjfhj"
Oct 30 13:11:35.279192 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:35.279161    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aab4aabb-55a4-412c-85c9-bf76db9513b3-workload-certs\") pod \"maas-default-gateway-openshift-default-64d4cc998c-pjfhj\" (UID: \"aab4aabb-55a4-412c-85c9-bf76db9513b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-pjfhj"
Oct 30 13:11:35.279346 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:35.279215    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aab4aabb-55a4-412c-85c9-bf76db9513b3-istio-data\") pod \"maas-default-gateway-openshift-default-64d4cc998c-pjfhj\" (UID: \"aab4aabb-55a4-412c-85c9-bf76db9513b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-pjfhj"
Oct 30 13:11:35.279346 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:35.279317    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aab4aabb-55a4-412c-85c9-bf76db9513b3-istio-envoy\") pod \"maas-default-gateway-openshift-default-64d4cc998c-pjfhj\" (UID: \"aab4aabb-55a4-412c-85c9-bf76db9513b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-pjfhj"
Oct 30 13:11:35.279430 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:35.279358    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aab4aabb-55a4-412c-85c9-bf76db9513b3-credential-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-pjfhj\" (UID: \"aab4aabb-55a4-412c-85c9-bf76db9513b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-pjfhj"
Oct 30 13:11:35.279474 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:35.279458    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxqgw\" (UniqueName: \"kubernetes.io/projected/aab4aabb-55a4-412c-85c9-bf76db9513b3-kube-api-access-pxqgw\") pod \"maas-default-gateway-openshift-default-64d4cc998c-pjfhj\" (UID: \"aab4aabb-55a4-412c-85c9-bf76db9513b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-pjfhj"
Oct 30 13:11:35.279517 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:35.279503    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aab4aabb-55a4-412c-85c9-bf76db9513b3-workload-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-pjfhj\" (UID: \"aab4aabb-55a4-412c-85c9-bf76db9513b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-pjfhj"
Oct 30 13:11:35.279560 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:35.279548    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aab4aabb-55a4-412c-85c9-bf76db9513b3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-64d4cc998c-pjfhj\" (UID: \"aab4aabb-55a4-412c-85c9-bf76db9513b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-pjfhj"
Oct 30 13:11:35.279705 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:35.279688    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aab4aabb-55a4-412c-85c9-bf76db9513b3-istio-token\") pod \"maas-default-gateway-openshift-default-64d4cc998c-pjfhj\" (UID: \"aab4aabb-55a4-412c-85c9-bf76db9513b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-pjfhj"
Oct 30 13:11:35.279755 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:35.279727    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aab4aabb-55a4-412c-85c9-bf76db9513b3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-64d4cc998c-pjfhj\" (UID: \"aab4aabb-55a4-412c-85c9-bf76db9513b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-pjfhj"
Oct 30 13:11:35.380957 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:35.380928    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxqgw\" (UniqueName: \"kubernetes.io/projected/aab4aabb-55a4-412c-85c9-bf76db9513b3-kube-api-access-pxqgw\") pod \"maas-default-gateway-openshift-default-64d4cc998c-pjfhj\" (UID: \"aab4aabb-55a4-412c-85c9-bf76db9513b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-pjfhj"
Oct 30 13:11:35.381371 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:35.381009    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aab4aabb-55a4-412c-85c9-bf76db9513b3-workload-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-pjfhj\" (UID: \"aab4aabb-55a4-412c-85c9-bf76db9513b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-pjfhj"
Oct 30 13:11:35.381371 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:35.381063    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aab4aabb-55a4-412c-85c9-bf76db9513b3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-64d4cc998c-pjfhj\" (UID: \"aab4aabb-55a4-412c-85c9-bf76db9513b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-pjfhj"
Oct 30 13:11:35.381371 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:35.381187    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aab4aabb-55a4-412c-85c9-bf76db9513b3-istio-token\") pod \"maas-default-gateway-openshift-default-64d4cc998c-pjfhj\" (UID: \"aab4aabb-55a4-412c-85c9-bf76db9513b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-pjfhj"
Oct 30 13:11:35.381371 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:35.381231    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aab4aabb-55a4-412c-85c9-bf76db9513b3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-64d4cc998c-pjfhj\" (UID: \"aab4aabb-55a4-412c-85c9-bf76db9513b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-pjfhj"
Oct 30 13:11:35.381371 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:35.381325    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aab4aabb-55a4-412c-85c9-bf76db9513b3-workload-certs\") pod \"maas-default-gateway-openshift-default-64d4cc998c-pjfhj\" (UID: \"aab4aabb-55a4-412c-85c9-bf76db9513b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-pjfhj"
Oct 30 13:11:35.381700 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:35.381376    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aab4aabb-55a4-412c-85c9-bf76db9513b3-istio-data\") pod \"maas-default-gateway-openshift-default-64d4cc998c-pjfhj\" (UID: \"aab4aabb-55a4-412c-85c9-bf76db9513b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-pjfhj"
Oct 30 13:11:35.381700 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:35.381392    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aab4aabb-55a4-412c-85c9-bf76db9513b3-workload-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-pjfhj\" (UID: \"aab4aabb-55a4-412c-85c9-bf76db9513b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-pjfhj"
Oct 30 13:11:35.381700 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:35.381483    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aab4aabb-55a4-412c-85c9-bf76db9513b3-istio-envoy\") pod \"maas-default-gateway-openshift-default-64d4cc998c-pjfhj\" (UID: \"aab4aabb-55a4-412c-85c9-bf76db9513b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-pjfhj"
Oct 30 13:11:35.381700 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:35.381550    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aab4aabb-55a4-412c-85c9-bf76db9513b3-credential-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-pjfhj\" (UID: \"aab4aabb-55a4-412c-85c9-bf76db9513b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-pjfhj"
Oct 30 13:11:35.381912 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:35.381717    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aab4aabb-55a4-412c-85c9-bf76db9513b3-workload-certs\") pod \"maas-default-gateway-openshift-default-64d4cc998c-pjfhj\" (UID: \"aab4aabb-55a4-412c-85c9-bf76db9513b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-pjfhj"
Oct 30 13:11:35.381912 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:35.381731    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aab4aabb-55a4-412c-85c9-bf76db9513b3-istio-data\") pod \"maas-default-gateway-openshift-default-64d4cc998c-pjfhj\" (UID: \"aab4aabb-55a4-412c-85c9-bf76db9513b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-pjfhj"
Oct 30 13:11:35.382028 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:35.381923    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aab4aabb-55a4-412c-85c9-bf76db9513b3-credential-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-pjfhj\" (UID: \"aab4aabb-55a4-412c-85c9-bf76db9513b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-pjfhj"
Oct 30 13:11:35.382028 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:35.381952    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aab4aabb-55a4-412c-85c9-bf76db9513b3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-64d4cc998c-pjfhj\" (UID: \"aab4aabb-55a4-412c-85c9-bf76db9513b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-pjfhj"
Oct 30 13:11:35.383613 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:35.383582    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aab4aabb-55a4-412c-85c9-bf76db9513b3-istio-envoy\") pod \"maas-default-gateway-openshift-default-64d4cc998c-pjfhj\" (UID: \"aab4aabb-55a4-412c-85c9-bf76db9513b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-pjfhj"
Oct 30 13:11:35.383713 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:35.383694    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aab4aabb-55a4-412c-85c9-bf76db9513b3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-64d4cc998c-pjfhj\" (UID: \"aab4aabb-55a4-412c-85c9-bf76db9513b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-pjfhj"
Oct 30 13:11:35.405656 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:35.405635    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aab4aabb-55a4-412c-85c9-bf76db9513b3-istio-token\") pod \"maas-default-gateway-openshift-default-64d4cc998c-pjfhj\" (UID: \"aab4aabb-55a4-412c-85c9-bf76db9513b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-pjfhj"
Oct 30 13:11:35.405904 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:35.405884    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxqgw\" (UniqueName: \"kubernetes.io/projected/aab4aabb-55a4-412c-85c9-bf76db9513b3-kube-api-access-pxqgw\") pod \"maas-default-gateway-openshift-default-64d4cc998c-pjfhj\" (UID: \"aab4aabb-55a4-412c-85c9-bf76db9513b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-pjfhj"
Oct 30 13:11:35.464250 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:35.464224    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-pjfhj"
Oct 30 13:11:35.789845 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:11:35.789803    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaab4aabb_55a4_412c_85c9_bf76db9513b3.slice/crio-88f471f2becd01891447d7a096259002c63ba8f92b37fdcb17c5f675a9f61d85 WatchSource:0}: Error finding container 88f471f2becd01891447d7a096259002c63ba8f92b37fdcb17c5f675a9f61d85: Status 404 returned error can't find the container with id 88f471f2becd01891447d7a096259002c63ba8f92b37fdcb17c5f675a9f61d85
Oct 30 13:11:35.791545 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:35.791516    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-pjfhj"]
Oct 30 13:11:35.792441 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:35.792405    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:11:35.792525 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:35.792482    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:11:35.792640 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:35.792530    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:11:36.322438 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:36.322398    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-pjfhj" event={"ID":"aab4aabb-55a4-412c-85c9-bf76db9513b3","Type":"ContainerStarted","Data":"916993b4c62dd3962a5343045bbbfacbdc027c3eea9f349526aae17df2e30a91"}
Oct 30 13:11:36.322671 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:36.322460    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-pjfhj" event={"ID":"aab4aabb-55a4-412c-85c9-bf76db9513b3","Type":"ContainerStarted","Data":"88f471f2becd01891447d7a096259002c63ba8f92b37fdcb17c5f675a9f61d85"}
Oct 30 13:11:36.410370 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:36.410288    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-pjfhj" podStartSLOduration=1.410268506 podStartE2EDuration="1.410268506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:36.376158323 +0000 UTC m=+1195.542522972" watchObservedRunningTime="2026-10-30 13:11:36.410268506 +0000 UTC m=+1195.576633155"
Oct 30 13:11:36.465169 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:36.465138    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-pjfhj"
Oct 30 13:11:36.467063 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:36.467042    4364 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-pjfhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 30 13:11:36.467125 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:36.467088    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-pjfhj" podUID="aab4aabb-55a4-412c-85c9-bf76db9513b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 30 13:11:37.304453 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:37.304413    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g6bnr"
Oct 30 13:11:37.465491 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:37.465459    4364 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-pjfhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 30 13:11:37.465861 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:37.465514    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-pjfhj" podUID="aab4aabb-55a4-412c-85c9-bf76db9513b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 30 13:11:38.310765 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:38.310733    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-45c4d25f-5b7f7bb64b-lpqj5"
Oct 30 13:11:38.465256 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:38.465223    4364 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-pjfhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 30 13:11:38.465423 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:38.465272    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-pjfhj" podUID="aab4aabb-55a4-412c-85c9-bf76db9513b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 30 13:11:39.465525 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:39.465490    4364 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-pjfhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 30 13:11:39.465921 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:39.465542    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-pjfhj" podUID="aab4aabb-55a4-412c-85c9-bf76db9513b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 30 13:11:40.472624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:40.465631    4364 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-pjfhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 30 13:11:40.472624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:40.465707    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-pjfhj" podUID="aab4aabb-55a4-412c-85c9-bf76db9513b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 30 13:11:41.400621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:41.398964    4364 scope.go:122] "RemoveContainer" containerID="03208f80ce282ec29e13fb9236786533688507fd6b6da3fd7f82e744f0b6947d"
Oct 30 13:11:41.465316 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:41.465289    4364 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-pjfhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 30 13:11:41.465409 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:41.465341    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-pjfhj" podUID="aab4aabb-55a4-412c-85c9-bf76db9513b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 30 13:11:41.529515 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:11:41.529480    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:11:42.465350 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:42.465312    4364 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-pjfhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 30 13:11:42.465524 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:42.465372    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-pjfhj" podUID="aab4aabb-55a4-412c-85c9-bf76db9513b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 30 13:11:43.465252 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:43.465221    4364 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-pjfhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 30 13:11:43.465637 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:43.465278    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-pjfhj" podUID="aab4aabb-55a4-412c-85c9-bf76db9513b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 30 13:11:44.469523 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:44.469494    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-pjfhj"
Oct 30 13:11:44.470037 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:44.470018    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-pjfhj"
Oct 30 13:11:44.470687 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:44.470663    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-pjfhj"
Oct 30 13:11:49.184621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:49.177246    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:11:54.295672 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:54.295337    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh" podUID="d41ac9e5-eeed-4e07-a7fa-05d3260630b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 30 13:11:54.295672 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:54.295647    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74" podUID="4f2b06ac-9bf8-4b57-bd22-85532576f9c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:54.299269 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:54.296287    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc" podUID="8911b790-331d-490c-876b-647e6d2fcd14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:54.299269 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:54.297220    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz" podUID="a568f1ce-602d-4069-a41c-9fe1a7635c5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 30 13:11:54.356459 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:54.354687    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4" podUID="0cbe6af7-d884-4613-90b8-192e5ffdb027" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:54.389253 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:54.389208    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz" podUID="bc5caf69-a649-47e5-8910-9b8f331dfd4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:54.400639 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:54.396157    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr" podUID="446c12df-de12-4c7c-8c71-dfed93cc33f1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:54.582678 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:54.577690    4364 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4" podUID="0cbe6af7-d884-4613-90b8-192e5ffdb027" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:54.909351 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:11:54.908700    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6" podUID="b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 13:12:03.104764 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:03.104733    4364 generic.go:356] "Generic (PLEG): container finished" podID="bdd3c1f5-f9fd-400c-90f2-f9693008003a" containerID="3ba0c9e791b155c0c55ef49e6410558e335a26c9456e87f0bfb5ec5a720c041d" exitCode=0
Oct 30 13:12:03.105179 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:03.104810    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85" event={"ID":"bdd3c1f5-f9fd-400c-90f2-f9693008003a","Type":"ContainerDied","Data":"3ba0c9e791b155c0c55ef49e6410558e335a26c9456e87f0bfb5ec5a720c041d"}
Oct 30 13:12:03.105179 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:03.105149    4364 scope.go:122] "RemoveContainer" containerID="3ba0c9e791b155c0c55ef49e6410558e335a26c9456e87f0bfb5ec5a720c041d"
Oct 30 13:12:03.106558 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:03.106527    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:12:03.106663 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:03.106611    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:12:03.106663 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:03.106646    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:12:04.116297 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:04.116259    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85" event={"ID":"bdd3c1f5-f9fd-400c-90f2-f9693008003a","Type":"ContainerStarted","Data":"c7e6d0a63ede59639ee66ff16092e925ed35cd6e45acf3c202b86c8db382a944"}
Oct 30 13:12:04.654201 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:04.654167    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85"
Oct 30 13:12:04.655791 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:04.655764    4364 patch_prober.go:29] interesting pod/e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 13:12:04.655924 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:04.655827    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85" podUID="bdd3c1f5-f9fd-400c-90f2-f9693008003a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 13:12:04.731982 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:04.731950    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85"
Oct 30 13:12:05.657043 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:05.657001    4364 patch_prober.go:29] interesting pod/e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 13:12:05.657671 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:05.657638    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85" podUID="bdd3c1f5-f9fd-400c-90f2-f9693008003a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 13:12:06.653904 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:06.653871    4364 patch_prober.go:29] interesting pod/e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 13:12:06.654072 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:06.653925    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85" podUID="bdd3c1f5-f9fd-400c-90f2-f9693008003a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 13:12:07.268418 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:07.268381    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-45c4d25f-78488bfdf8-5ksbg"]
Oct 30 13:12:07.309345 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:07.309311    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-45c4d25f-767667f47-c6lxc"]
Oct 30 13:12:07.653905 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:07.653868    4364 patch_prober.go:29] interesting pod/e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 13:12:07.658766 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:07.658722    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85" podUID="bdd3c1f5-f9fd-400c-90f2-f9693008003a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 13:12:08.136369 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:08.136320    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-911fe1d6-6c987d56bd-8rjs2"]
Oct 30 13:12:08.192344 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:08.192309    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-911fe1d6-688c4bd5cf-6lpwc"]
Oct 30 13:12:08.682631 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:08.663649    4364 patch_prober.go:29] interesting pod/e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 13:12:08.692298 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:08.691786    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85" podUID="bdd3c1f5-f9fd-400c-90f2-f9693008003a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 13:12:09.657305 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:09.656736    4364 patch_prober.go:29] interesting pod/e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 13:12:09.657305 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:09.656798    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85" podUID="bdd3c1f5-f9fd-400c-90f2-f9693008003a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 13:12:09.961796 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:09.961696    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6" podUID="b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 13:12:10.657954 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:10.657923    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85"
Oct 30 13:12:10.659162 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:10.659138    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85"
Oct 30 13:12:11.506623 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:12:11.506577    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:12:12.269569 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:12.269532    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-45c4d25f-78488bfdf8-5ksbg" podUID="9d1a108a-ac9f-472e-902a-852faa6a4b11" containerName="payload-processing" containerID="cri-o://90871ae811312ccee77ca8863f25776ef43a2b2514fa392ebdd2f84f5ab6e854" gracePeriod=25
Oct 30 13:12:12.309756 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:12.309723    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-45c4d25f-767667f47-c6lxc" podUID="e38332a6-07be-4d52-a06d-bcc49e8e3cf2" containerName="payload-pre-processing" containerID="cri-o://a92f6b314ac1b7ee568bea6625fe931ad2eb8ab0acfcd9df16cca3befece3b46" gracePeriod=25
Oct 30 13:12:12.559230 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:12.559155    4364 generic.go:356] "Generic (PLEG): container finished" podID="e38332a6-07be-4d52-a06d-bcc49e8e3cf2" containerID="a92f6b314ac1b7ee568bea6625fe931ad2eb8ab0acfcd9df16cca3befece3b46" exitCode=0
Oct 30 13:12:12.559671 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:12.559236    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-45c4d25f-767667f47-c6lxc" event={"ID":"e38332a6-07be-4d52-a06d-bcc49e8e3cf2","Type":"ContainerDied","Data":"a92f6b314ac1b7ee568bea6625fe931ad2eb8ab0acfcd9df16cca3befece3b46"}
Oct 30 13:12:12.561985 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:12.561959    4364 generic.go:356] "Generic (PLEG): container finished" podID="9d1a108a-ac9f-472e-902a-852faa6a4b11" containerID="90871ae811312ccee77ca8863f25776ef43a2b2514fa392ebdd2f84f5ab6e854" exitCode=0
Oct 30 13:12:12.562129 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:12.562009    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-45c4d25f-78488bfdf8-5ksbg" event={"ID":"9d1a108a-ac9f-472e-902a-852faa6a4b11","Type":"ContainerDied","Data":"90871ae811312ccee77ca8863f25776ef43a2b2514fa392ebdd2f84f5ab6e854"}
Oct 30 13:12:12.608352 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:12.608323    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-45c4d25f-78488bfdf8-5ksbg"
Oct 30 13:12:12.641647 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:12.641625    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-45c4d25f-767667f47-c6lxc"
Oct 30 13:12:12.722338 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:12.722308    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d1a108a-ac9f-472e-902a-852faa6a4b11-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d1a108a-ac9f-472e-902a-852faa6a4b11-plugins-config-volume\") pod \"9d1a108a-ac9f-472e-902a-852faa6a4b11\" (UID: \"9d1a108a-ac9f-472e-902a-852faa6a4b11\") "
Oct 30 13:12:12.722493 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:12.722464    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e38332a6-07be-4d52-a06d-bcc49e8e3cf2-kube-api-access-4b877\" (UniqueName: \"kubernetes.io/projected/e38332a6-07be-4d52-a06d-bcc49e8e3cf2-kube-api-access-4b877\") pod \"e38332a6-07be-4d52-a06d-bcc49e8e3cf2\" (UID: \"e38332a6-07be-4d52-a06d-bcc49e8e3cf2\") "
Oct 30 13:12:12.722555 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:12.722540    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e38332a6-07be-4d52-a06d-bcc49e8e3cf2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e38332a6-07be-4d52-a06d-bcc49e8e3cf2-plugins-config-volume\") pod \"e38332a6-07be-4d52-a06d-bcc49e8e3cf2\" (UID: \"e38332a6-07be-4d52-a06d-bcc49e8e3cf2\") "
Oct 30 13:12:12.722776 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:12.722754    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d1a108a-ac9f-472e-902a-852faa6a4b11-kube-api-access-7lgpj\" (UniqueName: \"kubernetes.io/projected/9d1a108a-ac9f-472e-902a-852faa6a4b11-kube-api-access-7lgpj\") pod \"9d1a108a-ac9f-472e-902a-852faa6a4b11\" (UID: \"9d1a108a-ac9f-472e-902a-852faa6a4b11\") "
Oct 30 13:12:12.722878 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:12.722803    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d1a108a-ac9f-472e-902a-852faa6a4b11-plugins-config-volume" pod "9d1a108a-ac9f-472e-902a-852faa6a4b11" (UID: "9d1a108a-ac9f-472e-902a-852faa6a4b11"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:12:12.722961 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:12.722940    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e38332a6-07be-4d52-a06d-bcc49e8e3cf2-plugins-config-volume" pod "e38332a6-07be-4d52-a06d-bcc49e8e3cf2" (UID: "e38332a6-07be-4d52-a06d-bcc49e8e3cf2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:12:12.723565 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:12.723537    4364 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e38332a6-07be-4d52-a06d-bcc49e8e3cf2-plugins-config-volume\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:12:12.723689 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:12.723568    4364 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d1a108a-ac9f-472e-902a-852faa6a4b11-plugins-config-volume\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:12:12.724827 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:12.724806    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e38332a6-07be-4d52-a06d-bcc49e8e3cf2-kube-api-access-4b877" pod "e38332a6-07be-4d52-a06d-bcc49e8e3cf2" (UID: "e38332a6-07be-4d52-a06d-bcc49e8e3cf2"). InnerVolumeSpecName "kube-api-access-4b877". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:12:12.725085 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:12.725065    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d1a108a-ac9f-472e-902a-852faa6a4b11-kube-api-access-7lgpj" pod "9d1a108a-ac9f-472e-902a-852faa6a4b11" (UID: "9d1a108a-ac9f-472e-902a-852faa6a4b11"). InnerVolumeSpecName "kube-api-access-7lgpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:12:12.825533 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:12.825463    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lgpj\" (UniqueName: \"kubernetes.io/projected/9d1a108a-ac9f-472e-902a-852faa6a4b11-kube-api-access-7lgpj\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:12:12.825533 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:12.825495    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4b877\" (UniqueName: \"kubernetes.io/projected/e38332a6-07be-4d52-a06d-bcc49e8e3cf2-kube-api-access-4b877\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:12:13.138462 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:13.138388    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-911fe1d6-6c987d56bd-8rjs2" podUID="e45fe3cc-5929-4826-83d2-51755bcc7719" containerName="payload-processing" containerID="cri-o://0e883c524a0f2ca2cef7a2a2d0f22095284812ae3d9fd3742b6890d17f234c66" gracePeriod=25
Oct 30 13:12:13.239410 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:13.233875    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-911fe1d6-688c4bd5cf-6lpwc" podUID="1b84a65e-b386-4785-885c-5da526c3f2cc" containerName="payload-pre-processing" containerID="cri-o://db1be6e705be61c34246e47e81a51553959dd0bbba2d0d9cf621d8f8eb46e754" gracePeriod=25
Oct 30 13:12:13.552955 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:13.552928    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-911fe1d6-6c987d56bd-8rjs2"
Oct 30 13:12:13.572784 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:13.572747    4364 generic.go:356] "Generic (PLEG): container finished" podID="e45fe3cc-5929-4826-83d2-51755bcc7719" containerID="0e883c524a0f2ca2cef7a2a2d0f22095284812ae3d9fd3742b6890d17f234c66" exitCode=0
Oct 30 13:12:13.573186 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:13.572807    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-911fe1d6-6c987d56bd-8rjs2"
Oct 30 13:12:13.573186 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:13.572835    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-911fe1d6-6c987d56bd-8rjs2" event={"ID":"e45fe3cc-5929-4826-83d2-51755bcc7719","Type":"ContainerDied","Data":"0e883c524a0f2ca2cef7a2a2d0f22095284812ae3d9fd3742b6890d17f234c66"}
Oct 30 13:12:13.573186 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:13.572896    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-911fe1d6-6c987d56bd-8rjs2" event={"ID":"e45fe3cc-5929-4826-83d2-51755bcc7719","Type":"ContainerDied","Data":"ae97085c59a1c22a229ff87592bf2026c4557fc9187f95c27c0dc83d57e8ddc3"}
Oct 30 13:12:13.573186 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:13.572918    4364 scope.go:122] "RemoveContainer" containerID="0e883c524a0f2ca2cef7a2a2d0f22095284812ae3d9fd3742b6890d17f234c66"
Oct 30 13:12:13.574525 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:13.574507    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-45c4d25f-767667f47-c6lxc"
Oct 30 13:12:13.574695 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:13.574546    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-45c4d25f-767667f47-c6lxc" event={"ID":"e38332a6-07be-4d52-a06d-bcc49e8e3cf2","Type":"ContainerDied","Data":"ccd707817530caee97189dcc3c5b6e4d7be30018de46df9eab62d03bd18fd875"}
Oct 30 13:12:13.576124 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:13.576103    4364 generic.go:356] "Generic (PLEG): container finished" podID="1b84a65e-b386-4785-885c-5da526c3f2cc" containerID="db1be6e705be61c34246e47e81a51553959dd0bbba2d0d9cf621d8f8eb46e754" exitCode=0
Oct 30 13:12:13.576216 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:13.576177    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-911fe1d6-688c4bd5cf-6lpwc" event={"ID":"1b84a65e-b386-4785-885c-5da526c3f2cc","Type":"ContainerDied","Data":"db1be6e705be61c34246e47e81a51553959dd0bbba2d0d9cf621d8f8eb46e754"}
Oct 30 13:12:13.577823 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:13.577805    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-45c4d25f-78488bfdf8-5ksbg" event={"ID":"9d1a108a-ac9f-472e-902a-852faa6a4b11","Type":"ContainerDied","Data":"c4171acbfb6ed3705a3aba65af26e3af6d518f6edd3d3c979506aa2aae0622c9"}
Oct 30 13:12:13.577823 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:13.577816    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-45c4d25f-78488bfdf8-5ksbg"
Oct 30 13:12:13.600168 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:13.600144    4364 scope.go:122] "RemoveContainer" containerID="0e883c524a0f2ca2cef7a2a2d0f22095284812ae3d9fd3742b6890d17f234c66"
Oct 30 13:12:13.600366 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:13.600342    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-911fe1d6-688c4bd5cf-6lpwc"
Oct 30 13:12:13.600695 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:12:13.600674    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e883c524a0f2ca2cef7a2a2d0f22095284812ae3d9fd3742b6890d17f234c66\": container with ID starting with 0e883c524a0f2ca2cef7a2a2d0f22095284812ae3d9fd3742b6890d17f234c66 not found: ID does not exist" containerID="0e883c524a0f2ca2cef7a2a2d0f22095284812ae3d9fd3742b6890d17f234c66"
Oct 30 13:12:13.600840 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:13.600704    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e883c524a0f2ca2cef7a2a2d0f22095284812ae3d9fd3742b6890d17f234c66"} err="failed to get container status \"0e883c524a0f2ca2cef7a2a2d0f22095284812ae3d9fd3742b6890d17f234c66\": rpc error: code = NotFound desc = could not find container \"0e883c524a0f2ca2cef7a2a2d0f22095284812ae3d9fd3742b6890d17f234c66\": container with ID starting with 0e883c524a0f2ca2cef7a2a2d0f22095284812ae3d9fd3742b6890d17f234c66 not found: ID does not exist"
Oct 30 13:12:13.600840 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:13.600728    4364 scope.go:122] "RemoveContainer" containerID="a92f6b314ac1b7ee568bea6625fe931ad2eb8ab0acfcd9df16cca3befece3b46"
Oct 30 13:12:13.625150 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:13.625126    4364 scope.go:122] "RemoveContainer" containerID="90871ae811312ccee77ca8863f25776ef43a2b2514fa392ebdd2f84f5ab6e854"
Oct 30 13:12:13.665125 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:13.665057    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b84a65e-b386-4785-885c-5da526c3f2cc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b84a65e-b386-4785-885c-5da526c3f2cc-plugins-config-volume\") pod \"1b84a65e-b386-4785-885c-5da526c3f2cc\" (UID: \"1b84a65e-b386-4785-885c-5da526c3f2cc\") "
Oct 30 13:12:13.665125 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:13.665121    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e45fe3cc-5929-4826-83d2-51755bcc7719-kube-api-access-pdq42\" (UniqueName: \"kubernetes.io/projected/e45fe3cc-5929-4826-83d2-51755bcc7719-kube-api-access-pdq42\") pod \"e45fe3cc-5929-4826-83d2-51755bcc7719\" (UID: \"e45fe3cc-5929-4826-83d2-51755bcc7719\") "
Oct 30 13:12:13.666661 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:13.665271    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b84a65e-b386-4785-885c-5da526c3f2cc-kube-api-access-87995\" (UniqueName: \"kubernetes.io/projected/1b84a65e-b386-4785-885c-5da526c3f2cc-kube-api-access-87995\") pod \"1b84a65e-b386-4785-885c-5da526c3f2cc\" (UID: \"1b84a65e-b386-4785-885c-5da526c3f2cc\") "
Oct 30 13:12:13.666661 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:13.665541    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e45fe3cc-5929-4826-83d2-51755bcc7719-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e45fe3cc-5929-4826-83d2-51755bcc7719-plugins-config-volume\") pod \"e45fe3cc-5929-4826-83d2-51755bcc7719\" (UID: \"e45fe3cc-5929-4826-83d2-51755bcc7719\") "
Oct 30 13:12:13.666661 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:13.666530    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e45fe3cc-5929-4826-83d2-51755bcc7719-plugins-config-volume" pod "e45fe3cc-5929-4826-83d2-51755bcc7719" (UID: "e45fe3cc-5929-4826-83d2-51755bcc7719"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:12:13.666661 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:13.666566    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b84a65e-b386-4785-885c-5da526c3f2cc-plugins-config-volume" pod "1b84a65e-b386-4785-885c-5da526c3f2cc" (UID: "1b84a65e-b386-4785-885c-5da526c3f2cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:12:13.668698 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:13.668677    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b84a65e-b386-4785-885c-5da526c3f2cc-kube-api-access-87995" pod "1b84a65e-b386-4785-885c-5da526c3f2cc" (UID: "1b84a65e-b386-4785-885c-5da526c3f2cc"). InnerVolumeSpecName "kube-api-access-87995". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:12:13.668779 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:13.668706    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e45fe3cc-5929-4826-83d2-51755bcc7719-kube-api-access-pdq42" pod "e45fe3cc-5929-4826-83d2-51755bcc7719" (UID: "e45fe3cc-5929-4826-83d2-51755bcc7719"). InnerVolumeSpecName "kube-api-access-pdq42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:12:13.678970 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:13.678947    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-45c4d25f-78488bfdf8-5ksbg"]
Oct 30 13:12:13.701269 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:13.701245    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-45c4d25f-78488bfdf8-5ksbg"]
Oct 30 13:12:13.768253 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:13.768215    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87995\" (UniqueName: \"kubernetes.io/projected/1b84a65e-b386-4785-885c-5da526c3f2cc-kube-api-access-87995\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:12:13.768253 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:13.768252    4364 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e45fe3cc-5929-4826-83d2-51755bcc7719-plugins-config-volume\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:12:13.768453 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:13.768268    4364 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b84a65e-b386-4785-885c-5da526c3f2cc-plugins-config-volume\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:12:13.768453 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:13.768286    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdq42\" (UniqueName: \"kubernetes.io/projected/e45fe3cc-5929-4826-83d2-51755bcc7719-kube-api-access-pdq42\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:12:13.817683 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:13.817654    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-45c4d25f-767667f47-c6lxc"]
Oct 30 13:12:13.823864 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:13.823831    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-45c4d25f-767667f47-c6lxc"]
Oct 30 13:12:13.930097 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:13.930037    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-911fe1d6-6c987d56bd-8rjs2"]
Oct 30 13:12:13.940138 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:13.940115    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-911fe1d6-6c987d56bd-8rjs2"]
Oct 30 13:12:14.682114 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:14.679072    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-911fe1d6-688c4bd5cf-6lpwc" event={"ID":"1b84a65e-b386-4785-885c-5da526c3f2cc","Type":"ContainerDied","Data":"523938f1abc0fe139a980c5adab3f3910250cdde56fb6246afcdb882718bf808"}
Oct 30 13:12:14.682114 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:14.679144    4364 scope.go:122] "RemoveContainer" containerID="db1be6e705be61c34246e47e81a51553959dd0bbba2d0d9cf621d8f8eb46e754"
Oct 30 13:12:14.682114 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:14.679265    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-911fe1d6-688c4bd5cf-6lpwc"
Oct 30 13:12:14.851836 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:14.851758    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-911fe1d6-688c4bd5cf-6lpwc"]
Oct 30 13:12:14.925620 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:14.918530    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-911fe1d6-688c4bd5cf-6lpwc"]
Oct 30 13:12:15.013709 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:15.012116    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b84a65e-b386-4785-885c-5da526c3f2cc" path="/var/lib/kubelet/pods/1b84a65e-b386-4785-885c-5da526c3f2cc/volumes"
Oct 30 13:12:15.028620 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:15.020914    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d1a108a-ac9f-472e-902a-852faa6a4b11" path="/var/lib/kubelet/pods/9d1a108a-ac9f-472e-902a-852faa6a4b11/volumes"
Oct 30 13:12:15.028620 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:15.021521    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e38332a6-07be-4d52-a06d-bcc49e8e3cf2" path="/var/lib/kubelet/pods/e38332a6-07be-4d52-a06d-bcc49e8e3cf2/volumes"
Oct 30 13:12:15.028620 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:15.022115    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e45fe3cc-5929-4826-83d2-51755bcc7719" path="/var/lib/kubelet/pods/e45fe3cc-5929-4826-83d2-51755bcc7719/volumes"
Oct 30 13:12:15.626633 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:15.623228    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-45c4d25f-5b7f7bb64b-lpqj5"]
Oct 30 13:12:15.626633 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:15.623477    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-45c4d25f-5b7f7bb64b-lpqj5" podUID="002fc4fb-5d07-4c13-ab09-0a8d7d489003" containerName="maas-api" containerID="cri-o://f6b64e111835331f5cd8375c0626c5f0408805dc78af22c99cf564a1718b2621" gracePeriod=30
Oct 30 13:12:15.790100 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:15.790060    4364 generic.go:356] "Generic (PLEG): container finished" podID="002fc4fb-5d07-4c13-ab09-0a8d7d489003" containerID="f6b64e111835331f5cd8375c0626c5f0408805dc78af22c99cf564a1718b2621" exitCode=0
Oct 30 13:12:15.790522 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:15.790358    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-45c4d25f-5b7f7bb64b-lpqj5" event={"ID":"002fc4fb-5d07-4c13-ab09-0a8d7d489003","Type":"ContainerDied","Data":"f6b64e111835331f5cd8375c0626c5f0408805dc78af22c99cf564a1718b2621"}
Oct 30 13:12:16.020468 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:16.020440    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-45c4d25f-5b7f7bb64b-lpqj5"
Oct 30 13:12:16.102847 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:16.102780    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/002fc4fb-5d07-4c13-ab09-0a8d7d489003-metrics-certs\" (UniqueName: \"kubernetes.io/secret/002fc4fb-5d07-4c13-ab09-0a8d7d489003-metrics-certs\") pod \"002fc4fb-5d07-4c13-ab09-0a8d7d489003\" (UID: \"002fc4fb-5d07-4c13-ab09-0a8d7d489003\") "
Oct 30 13:12:16.103081 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:16.102993    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/002fc4fb-5d07-4c13-ab09-0a8d7d489003-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/002fc4fb-5d07-4c13-ab09-0a8d7d489003-maas-api-tls\") pod \"002fc4fb-5d07-4c13-ab09-0a8d7d489003\" (UID: \"002fc4fb-5d07-4c13-ab09-0a8d7d489003\") "
Oct 30 13:12:16.103335 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:16.103309    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/002fc4fb-5d07-4c13-ab09-0a8d7d489003-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/002fc4fb-5d07-4c13-ab09-0a8d7d489003-openshift-service-ca\") pod \"002fc4fb-5d07-4c13-ab09-0a8d7d489003\" (UID: \"002fc4fb-5d07-4c13-ab09-0a8d7d489003\") "
Oct 30 13:12:16.103660 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:16.103493    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/002fc4fb-5d07-4c13-ab09-0a8d7d489003-kube-api-access-6zpmk\" (UniqueName: \"kubernetes.io/projected/002fc4fb-5d07-4c13-ab09-0a8d7d489003-kube-api-access-6zpmk\") pod \"002fc4fb-5d07-4c13-ab09-0a8d7d489003\" (UID: \"002fc4fb-5d07-4c13-ab09-0a8d7d489003\") "
Oct 30 13:12:16.104331 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:16.104293    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/002fc4fb-5d07-4c13-ab09-0a8d7d489003-openshift-service-ca" pod "002fc4fb-5d07-4c13-ab09-0a8d7d489003" (UID: "002fc4fb-5d07-4c13-ab09-0a8d7d489003"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:12:16.105441 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:16.105410    4364 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/002fc4fb-5d07-4c13-ab09-0a8d7d489003-openshift-service-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:12:16.105763 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:16.105740    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/002fc4fb-5d07-4c13-ab09-0a8d7d489003-metrics-certs" pod "002fc4fb-5d07-4c13-ab09-0a8d7d489003" (UID: "002fc4fb-5d07-4c13-ab09-0a8d7d489003"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:12:16.105897 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:16.105828    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/002fc4fb-5d07-4c13-ab09-0a8d7d489003-maas-api-tls" pod "002fc4fb-5d07-4c13-ab09-0a8d7d489003" (UID: "002fc4fb-5d07-4c13-ab09-0a8d7d489003"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:12:16.107198 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:16.107171    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/002fc4fb-5d07-4c13-ab09-0a8d7d489003-kube-api-access-6zpmk" pod "002fc4fb-5d07-4c13-ab09-0a8d7d489003" (UID: "002fc4fb-5d07-4c13-ab09-0a8d7d489003"). InnerVolumeSpecName "kube-api-access-6zpmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:12:16.159191 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:16.159157    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-df6c3661-5866c5b5c5-99krg"]
Oct 30 13:12:16.159920 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:16.159902    4364 cpu_manager.go:414] "containerMap: removing container" podUID="1b84a65e-b386-4785-885c-5da526c3f2cc" containerName="payload-pre-processing"
Oct 30 13:12:16.160024 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:16.159920    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b84a65e-b386-4785-885c-5da526c3f2cc" containerName="payload-pre-processing"
Oct 30 13:12:16.160024 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:16.159949    4364 cpu_manager.go:414] "containerMap: removing container" podUID="002fc4fb-5d07-4c13-ab09-0a8d7d489003" containerName="maas-api"
Oct 30 13:12:16.160024 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:16.159957    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="002fc4fb-5d07-4c13-ab09-0a8d7d489003" containerName="maas-api"
Oct 30 13:12:16.160024 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:16.159996    4364 cpu_manager.go:414] "containerMap: removing container" podUID="e45fe3cc-5929-4826-83d2-51755bcc7719" containerName="payload-processing"
Oct 30 13:12:16.160024 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:16.160003    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e45fe3cc-5929-4826-83d2-51755bcc7719" containerName="payload-processing"
Oct 30 13:12:16.160024 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:16.160014    4364 cpu_manager.go:414] "containerMap: removing container" podUID="9d1a108a-ac9f-472e-902a-852faa6a4b11" containerName="payload-processing"
Oct 30 13:12:16.160024 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:16.160022    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d1a108a-ac9f-472e-902a-852faa6a4b11" containerName="payload-processing"
Oct 30 13:12:16.160347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:16.160058    4364 cpu_manager.go:414] "containerMap: removing container" podUID="e38332a6-07be-4d52-a06d-bcc49e8e3cf2" containerName="payload-pre-processing"
Oct 30 13:12:16.160347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:16.160066    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e38332a6-07be-4d52-a06d-bcc49e8e3cf2" containerName="payload-pre-processing"
Oct 30 13:12:16.160347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:16.160236    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="e38332a6-07be-4d52-a06d-bcc49e8e3cf2" containerName="payload-pre-processing"
Oct 30 13:12:16.160347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:16.160256    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b84a65e-b386-4785-885c-5da526c3f2cc" containerName="payload-pre-processing"
Oct 30 13:12:16.160347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:16.160270    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="002fc4fb-5d07-4c13-ab09-0a8d7d489003" containerName="maas-api"
Oct 30 13:12:16.160347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:16.160285    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="e45fe3cc-5929-4826-83d2-51755bcc7719" containerName="payload-processing"
Oct 30 13:12:16.160347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:16.160295    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d1a108a-ac9f-472e-902a-852faa6a4b11" containerName="payload-processing"
Oct 30 13:12:16.163067 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:16.163050    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-df6c3661-5866c5b5c5-99krg"
Oct 30 13:12:16.181468 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:16.181443    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-df6c3661-dockercfg-hrvmd\""
Oct 30 13:12:16.181468 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:16.181464    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-df6c3661\""
Oct 30 13:12:16.185554 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:16.185534    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-df6c3661-5866c5b5c5-99krg"]
Oct 30 13:12:16.207294 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:16.207269    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/552839a6-adf1-4cb3-adf3-c75500d71782-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-df6c3661-5866c5b5c5-99krg\" (UID: \"552839a6-adf1-4cb3-adf3-c75500d71782\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-df6c3661-5866c5b5c5-99krg"
Oct 30 13:12:16.207566 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:16.207477    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbn4q\" (UniqueName: \"kubernetes.io/projected/552839a6-adf1-4cb3-adf3-c75500d71782-kube-api-access-vbn4q\") pod \"payload-pre-processing-e2e-mt-df6c3661-5866c5b5c5-99krg\" (UID: \"552839a6-adf1-4cb3-adf3-c75500d71782\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-df6c3661-5866c5b5c5-99krg"
Oct 30 13:12:16.207992 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:16.207839    4364 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/002fc4fb-5d07-4c13-ab09-0a8d7d489003-metrics-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:12:16.207992 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:16.207860    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/002fc4fb-5d07-4c13-ab09-0a8d7d489003-maas-api-tls\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:12:16.207992 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:16.207874    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zpmk\" (UniqueName: \"kubernetes.io/projected/002fc4fb-5d07-4c13-ab09-0a8d7d489003-kube-api-access-6zpmk\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:12:16.229062 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:16.228980    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-df6c3661-7f764d9794-zph7s"]
Oct 30 13:12:16.236414 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:16.236369    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-df6c3661-7f764d9794-zph7s"
Oct 30 13:12:16.268713 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:16.257397    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-df6c3661-7f764d9794-zph7s"]
Oct 30 13:12:16.332687 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:16.330939    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e50284eb-016e-4bc8-bf70-b3eb2815493c-plugins-config-volume\") pod \"payload-processing-e2e-mt-df6c3661-7f764d9794-zph7s\" (UID: \"e50284eb-016e-4bc8-bf70-b3eb2815493c\") " pod="openshift-ingress/payload-processing-e2e-mt-df6c3661-7f764d9794-zph7s"
Oct 30 13:12:16.332687 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:16.331002    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/552839a6-adf1-4cb3-adf3-c75500d71782-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-df6c3661-5866c5b5c5-99krg\" (UID: \"552839a6-adf1-4cb3-adf3-c75500d71782\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-df6c3661-5866c5b5c5-99krg"
Oct 30 13:12:16.332687 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:16.331165    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbn4q\" (UniqueName: \"kubernetes.io/projected/552839a6-adf1-4cb3-adf3-c75500d71782-kube-api-access-vbn4q\") pod \"payload-pre-processing-e2e-mt-df6c3661-5866c5b5c5-99krg\" (UID: \"552839a6-adf1-4cb3-adf3-c75500d71782\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-df6c3661-5866c5b5c5-99krg"
Oct 30 13:12:16.332687 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:16.331214    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcg6n\" (UniqueName: \"kubernetes.io/projected/e50284eb-016e-4bc8-bf70-b3eb2815493c-kube-api-access-zcg6n\") pod \"payload-processing-e2e-mt-df6c3661-7f764d9794-zph7s\" (UID: \"e50284eb-016e-4bc8-bf70-b3eb2815493c\") " pod="openshift-ingress/payload-processing-e2e-mt-df6c3661-7f764d9794-zph7s"
Oct 30 13:12:16.332687 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:16.332239    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/552839a6-adf1-4cb3-adf3-c75500d71782-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-df6c3661-5866c5b5c5-99krg\" (UID: \"552839a6-adf1-4cb3-adf3-c75500d71782\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-df6c3661-5866c5b5c5-99krg"
Oct 30 13:12:16.498670 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:16.498451    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcg6n\" (UniqueName: \"kubernetes.io/projected/e50284eb-016e-4bc8-bf70-b3eb2815493c-kube-api-access-zcg6n\") pod \"payload-processing-e2e-mt-df6c3661-7f764d9794-zph7s\" (UID: \"e50284eb-016e-4bc8-bf70-b3eb2815493c\") " pod="openshift-ingress/payload-processing-e2e-mt-df6c3661-7f764d9794-zph7s"
Oct 30 13:12:16.498852 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:16.498744    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e50284eb-016e-4bc8-bf70-b3eb2815493c-plugins-config-volume\") pod \"payload-processing-e2e-mt-df6c3661-7f764d9794-zph7s\" (UID: \"e50284eb-016e-4bc8-bf70-b3eb2815493c\") " pod="openshift-ingress/payload-processing-e2e-mt-df6c3661-7f764d9794-zph7s"
Oct 30 13:12:16.502626 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:16.499794    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e50284eb-016e-4bc8-bf70-b3eb2815493c-plugins-config-volume\") pod \"payload-processing-e2e-mt-df6c3661-7f764d9794-zph7s\" (UID: \"e50284eb-016e-4bc8-bf70-b3eb2815493c\") " pod="openshift-ingress/payload-processing-e2e-mt-df6c3661-7f764d9794-zph7s"
Oct 30 13:12:16.628420 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:16.628390    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbn4q\" (UniqueName: \"kubernetes.io/projected/552839a6-adf1-4cb3-adf3-c75500d71782-kube-api-access-vbn4q\") pod \"payload-pre-processing-e2e-mt-df6c3661-5866c5b5c5-99krg\" (UID: \"552839a6-adf1-4cb3-adf3-c75500d71782\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-df6c3661-5866c5b5c5-99krg"
Oct 30 13:12:16.628665 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:16.628458    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcg6n\" (UniqueName: \"kubernetes.io/projected/e50284eb-016e-4bc8-bf70-b3eb2815493c-kube-api-access-zcg6n\") pod \"payload-processing-e2e-mt-df6c3661-7f764d9794-zph7s\" (UID: \"e50284eb-016e-4bc8-bf70-b3eb2815493c\") " pod="openshift-ingress/payload-processing-e2e-mt-df6c3661-7f764d9794-zph7s"
Oct 30 13:12:16.781209 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:16.781035    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-df6c3661-5866c5b5c5-99krg"
Oct 30 13:12:16.912137 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:16.900566    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-df6c3661-7f764d9794-zph7s"
Oct 30 13:12:17.150621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:17.128384    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-45c4d25f-5b7f7bb64b-lpqj5"
Oct 30 13:12:17.227394 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:17.227354    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-45c4d25f-5b7f7bb64b-lpqj5" event={"ID":"002fc4fb-5d07-4c13-ab09-0a8d7d489003","Type":"ContainerDied","Data":"5d38c9a7008502af150584fbc3490495a01c7b0a391eedd1e656d9bbc9fe5dbe"}
Oct 30 13:12:17.255744 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:17.255706    4364 scope.go:122] "RemoveContainer" containerID="f6b64e111835331f5cd8375c0626c5f0408805dc78af22c99cf564a1718b2621"
Oct 30 13:12:17.297618 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:17.293691    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-45c4d25f-5b7f7bb64b-lpqj5"]
Oct 30 13:12:17.350254 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:17.350169    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-45c4d25f-5b7f7bb64b-lpqj5"]
Oct 30 13:12:17.664165 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:17.663715    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-df6c3661-7f764d9794-zph7s"]
Oct 30 13:12:17.666576 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:12:17.666545    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode50284eb_016e_4bc8_bf70_b3eb2815493c.slice/crio-628debe20ed039a2e79295a7e0aca0e80c2d0eb367c12abd8afe01dfd7f4c41d WatchSource:0}: Error finding container 628debe20ed039a2e79295a7e0aca0e80c2d0eb367c12abd8afe01dfd7f4c41d: Status 404 returned error can't find the container with id 628debe20ed039a2e79295a7e0aca0e80c2d0eb367c12abd8afe01dfd7f4c41d
Oct 30 13:12:17.677973 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:17.677945    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-df6c3661-5866c5b5c5-99krg"]
Oct 30 13:12:17.684376 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:12:17.684352    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod552839a6_adf1_4cb3_adf3_c75500d71782.slice/crio-fddcc9d7bf951892421050e2290f1fb84c65e1f47c80bc32a7c38c370638160a WatchSource:0}: Error finding container fddcc9d7bf951892421050e2290f1fb84c65e1f47c80bc32a7c38c370638160a: Status 404 returned error can't find the container with id fddcc9d7bf951892421050e2290f1fb84c65e1f47c80bc32a7c38c370638160a
Oct 30 13:12:18.174543 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:18.174451    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-df6c3661-7f764d9794-zph7s" event={"ID":"e50284eb-016e-4bc8-bf70-b3eb2815493c","Type":"ContainerStarted","Data":"26c3a8eac05adbbf7801fbcca634ec6809557ac4043655bd20db49bf72a42b2f"}
Oct 30 13:12:18.174543 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:18.174514    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-df6c3661-7f764d9794-zph7s" event={"ID":"e50284eb-016e-4bc8-bf70-b3eb2815493c","Type":"ContainerStarted","Data":"628debe20ed039a2e79295a7e0aca0e80c2d0eb367c12abd8afe01dfd7f4c41d"}
Oct 30 13:12:18.175554 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:18.175528    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-df6c3661-7f764d9794-zph7s"
Oct 30 13:12:18.179656 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:18.179631    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-df6c3661-5866c5b5c5-99krg" event={"ID":"552839a6-adf1-4cb3-adf3-c75500d71782","Type":"ContainerStarted","Data":"16fa7ae7eec2b13d0d6406c1e7bb31b05664dfc6c45ca82fafc3e99fe6a4bc7d"}
Oct 30 13:12:18.179752 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:18.179676    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-df6c3661-5866c5b5c5-99krg" event={"ID":"552839a6-adf1-4cb3-adf3-c75500d71782","Type":"ContainerStarted","Data":"fddcc9d7bf951892421050e2290f1fb84c65e1f47c80bc32a7c38c370638160a"}
Oct 30 13:12:18.179789 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:18.179769    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-df6c3661-5866c5b5c5-99krg"
Oct 30 13:12:18.222450 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:18.222393    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-df6c3661-7f764d9794-zph7s" podStartSLOduration=2.222377627 podStartE2EDuration="2.222377627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:18.218835405 +0000 UTC m=+1237.385200054" watchObservedRunningTime="2026-10-30 13:12:18.222377627 +0000 UTC m=+1237.388742276"
Oct 30 13:12:18.269036 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:18.268998    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-df6c3661-5866c5b5c5-99krg" podStartSLOduration=2.2689864809999998 podStartE2EDuration="2.268986481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:18.26742383 +0000 UTC m=+1237.433788478" watchObservedRunningTime="2026-10-30 13:12:18.268986481 +0000 UTC m=+1237.435351170"
Oct 30 13:12:18.996619 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:18.996455    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="002fc4fb-5d07-4c13-ab09-0a8d7d489003" path="/var/lib/kubelet/pods/002fc4fb-5d07-4c13-ab09-0a8d7d489003/volumes"
Oct 30 13:12:20.467941 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:20.467914    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3778dacd-b89667ff4-jm528"]
Oct 30 13:12:20.471849 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:20.471832    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3778dacd-b89667ff4-jm528"
Oct 30 13:12:20.495080 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:20.495050    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3778dacd-b89667ff4-jm528"]
Oct 30 13:12:20.502145 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:20.501453    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-3778dacd-dockercfg-27f7x\""
Oct 30 13:12:20.502145 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:20.501513    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-3778dacd\""
Oct 30 13:12:20.522209 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:20.522185    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4824\" (UniqueName: \"kubernetes.io/projected/8fac787c-599b-4969-af01-42832173ed13-kube-api-access-j4824\") pod \"payload-processing-e2e-mt-3778dacd-b89667ff4-jm528\" (UID: \"8fac787c-599b-4969-af01-42832173ed13\") " pod="openshift-ingress/payload-processing-e2e-mt-3778dacd-b89667ff4-jm528"
Oct 30 13:12:20.522315 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:20.522226    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fac787c-599b-4969-af01-42832173ed13-plugins-config-volume\") pod \"payload-processing-e2e-mt-3778dacd-b89667ff4-jm528\" (UID: \"8fac787c-599b-4969-af01-42832173ed13\") " pod="openshift-ingress/payload-processing-e2e-mt-3778dacd-b89667ff4-jm528"
Oct 30 13:12:20.657121 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:20.653312    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4824\" (UniqueName: \"kubernetes.io/projected/8fac787c-599b-4969-af01-42832173ed13-kube-api-access-j4824\") pod \"payload-processing-e2e-mt-3778dacd-b89667ff4-jm528\" (UID: \"8fac787c-599b-4969-af01-42832173ed13\") " pod="openshift-ingress/payload-processing-e2e-mt-3778dacd-b89667ff4-jm528"
Oct 30 13:12:20.657121 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:20.653374    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fac787c-599b-4969-af01-42832173ed13-plugins-config-volume\") pod \"payload-processing-e2e-mt-3778dacd-b89667ff4-jm528\" (UID: \"8fac787c-599b-4969-af01-42832173ed13\") " pod="openshift-ingress/payload-processing-e2e-mt-3778dacd-b89667ff4-jm528"
Oct 30 13:12:20.670819 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:20.661445    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fac787c-599b-4969-af01-42832173ed13-plugins-config-volume\") pod \"payload-processing-e2e-mt-3778dacd-b89667ff4-jm528\" (UID: \"8fac787c-599b-4969-af01-42832173ed13\") " pod="openshift-ingress/payload-processing-e2e-mt-3778dacd-b89667ff4-jm528"
Oct 30 13:12:20.768699 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:20.766562    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4824\" (UniqueName: \"kubernetes.io/projected/8fac787c-599b-4969-af01-42832173ed13-kube-api-access-j4824\") pod \"payload-processing-e2e-mt-3778dacd-b89667ff4-jm528\" (UID: \"8fac787c-599b-4969-af01-42832173ed13\") " pod="openshift-ingress/payload-processing-e2e-mt-3778dacd-b89667ff4-jm528"
Oct 30 13:12:20.797621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:20.785888    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3778dacd-b89667ff4-jm528"
Oct 30 13:12:21.342665 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:21.337216    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3778dacd-b89667ff4-jm528"]
Oct 30 13:12:21.349761 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:12:21.347960    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fac787c_599b_4969_af01_42832173ed13.slice/crio-b8e9cc13cd371bb99aec3583b07ca846b143550e04634c16c38615b46dc2fb62 WatchSource:0}: Error finding container b8e9cc13cd371bb99aec3583b07ca846b143550e04634c16c38615b46dc2fb62: Status 404 returned error can't find the container with id b8e9cc13cd371bb99aec3583b07ca846b143550e04634c16c38615b46dc2fb62
Oct 30 13:12:22.327958 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:22.325899    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3778dacd-b89667ff4-jm528" event={"ID":"8fac787c-599b-4969-af01-42832173ed13","Type":"ContainerStarted","Data":"8d3c89fe016a68e7c799e799e27509d1e818a133cd17bfabef4d89329648f87f"}
Oct 30 13:12:22.328578 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:22.328550    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3778dacd-b89667ff4-jm528" event={"ID":"8fac787c-599b-4969-af01-42832173ed13","Type":"ContainerStarted","Data":"b8e9cc13cd371bb99aec3583b07ca846b143550e04634c16c38615b46dc2fb62"}
Oct 30 13:12:22.331310 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:22.331289    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-3778dacd-b89667ff4-jm528"
Oct 30 13:12:22.398759 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:22.398538    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-3778dacd-b89667ff4-jm528" podStartSLOduration=2.398502471 podStartE2EDuration="2.398502471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:22.392964341 +0000 UTC m=+1241.559328990" watchObservedRunningTime="2026-10-30 13:12:22.398502471 +0000 UTC m=+1241.564867118"
Oct 30 13:12:29.189762 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:29.189731    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-df6c3661-5866c5b5c5-99krg"
Oct 30 13:12:30.198529 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:30.198503    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-df6c3661-7f764d9794-zph7s"
Oct 30 13:12:32.455345 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:32.455309    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:12:34.346484 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:34.346454    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-3778dacd-b89667ff4-jm528"
Oct 30 13:12:41.471200 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:41.471165    4364 scope.go:122] "RemoveContainer" containerID="f316989fd066dc19051e435fb98341dcb5d916d229b8ceceff0a51935c4943b8"
Oct 30 13:12:41.482388 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:41.482369    4364 scope.go:122] "RemoveContainer" containerID="c99d0b1f391a1b8fc8c79265021f3fb414c436636f3efe640f4e59e19acfca30"
Oct 30 13:12:41.491714 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:41.491697    4364 scope.go:122] "RemoveContainer" containerID="417d6624328d0a9d834ad2fc8e6a21dcee258a9144a334e3ae2afe295c9b7cad"
Oct 30 13:12:41.505892 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:12:41.505861    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:12:47.062049 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:47.062015    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8234e9d-qxpdh"]
Oct 30 13:12:47.075937 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:47.075911    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8234e9d-qxpdh"]
Oct 30 13:12:48.972231 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:12:48.972195    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b6051c6-a230-4df5-97d5-77401c83bf95" path="/var/lib/kubelet/pods/2b6051c6-a230-4df5-97d5-77401c83bf95/volumes"
Oct 30 13:13:01.744882 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:01.744847    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fafb3f-7674fdb99d-v9prm"]
Oct 30 13:13:01.748299 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:01.748282    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-fafb3f-7674fdb99d-v9prm"
Oct 30 13:13:01.772947 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:01.772925    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-fafb3f-dockercfg-pfrks\""
Oct 30 13:13:01.778415 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:01.778390    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fafb3f-7674fdb99d-v9prm"]
Oct 30 13:13:01.787178 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:01.787156    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-fafb3f\""
Oct 30 13:13:01.812393 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:01.812359    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fafb3f-788c86bfc9-s6m45"]
Oct 30 13:13:01.829461 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:01.829437    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fafb3f-788c86bfc9-s6m45"
Oct 30 13:13:01.861968 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:01.844946    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55402878-108c-4c3d-b6c8-4362f82e82cb-plugins-config-volume\") pod \"payload-processing-e2e-api-a-fafb3f-7674fdb99d-v9prm\" (UID: \"55402878-108c-4c3d-b6c8-4362f82e82cb\") " pod="openshift-ingress/payload-processing-e2e-api-a-fafb3f-7674fdb99d-v9prm"
Oct 30 13:13:01.861968 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:01.844986    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6j6m\" (UniqueName: \"kubernetes.io/projected/55402878-108c-4c3d-b6c8-4362f82e82cb-kube-api-access-c6j6m\") pod \"payload-processing-e2e-api-a-fafb3f-7674fdb99d-v9prm\" (UID: \"55402878-108c-4c3d-b6c8-4362f82e82cb\") " pod="openshift-ingress/payload-processing-e2e-api-a-fafb3f-7674fdb99d-v9prm"
Oct 30 13:13:01.973622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:01.946857    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55402878-108c-4c3d-b6c8-4362f82e82cb-plugins-config-volume\") pod \"payload-processing-e2e-api-a-fafb3f-7674fdb99d-v9prm\" (UID: \"55402878-108c-4c3d-b6c8-4362f82e82cb\") " pod="openshift-ingress/payload-processing-e2e-api-a-fafb3f-7674fdb99d-v9prm"
Oct 30 13:13:01.973622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:01.946915    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6j6m\" (UniqueName: \"kubernetes.io/projected/55402878-108c-4c3d-b6c8-4362f82e82cb-kube-api-access-c6j6m\") pod \"payload-processing-e2e-api-a-fafb3f-7674fdb99d-v9prm\" (UID: \"55402878-108c-4c3d-b6c8-4362f82e82cb\") " pod="openshift-ingress/payload-processing-e2e-api-a-fafb3f-7674fdb99d-v9prm"
Oct 30 13:13:01.973622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:01.962705    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55402878-108c-4c3d-b6c8-4362f82e82cb-plugins-config-volume\") pod \"payload-processing-e2e-api-a-fafb3f-7674fdb99d-v9prm\" (UID: \"55402878-108c-4c3d-b6c8-4362f82e82cb\") " pod="openshift-ingress/payload-processing-e2e-api-a-fafb3f-7674fdb99d-v9prm"
Oct 30 13:13:02.083717 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:02.064416    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f09ff8b4-d468-4c5e-9265-1f13766720f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-fafb3f-788c86bfc9-s6m45\" (UID: \"f09ff8b4-d468-4c5e-9265-1f13766720f2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fafb3f-788c86bfc9-s6m45"
Oct 30 13:13:02.087202 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:02.087165    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t2z7\" (UniqueName: \"kubernetes.io/projected/f09ff8b4-d468-4c5e-9265-1f13766720f2-kube-api-access-4t2z7\") pod \"payload-pre-processing-e2e-api-a-fafb3f-788c86bfc9-s6m45\" (UID: \"f09ff8b4-d468-4c5e-9265-1f13766720f2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fafb3f-788c86bfc9-s6m45"
Oct 30 13:13:02.213341 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:02.197055    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f09ff8b4-d468-4c5e-9265-1f13766720f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-fafb3f-788c86bfc9-s6m45\" (UID: \"f09ff8b4-d468-4c5e-9265-1f13766720f2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fafb3f-788c86bfc9-s6m45"
Oct 30 13:13:02.248905 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:02.201513    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f09ff8b4-d468-4c5e-9265-1f13766720f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-fafb3f-788c86bfc9-s6m45\" (UID: \"f09ff8b4-d468-4c5e-9265-1f13766720f2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fafb3f-788c86bfc9-s6m45"
Oct 30 13:13:02.250040 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:02.249623    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t2z7\" (UniqueName: \"kubernetes.io/projected/f09ff8b4-d468-4c5e-9265-1f13766720f2-kube-api-access-4t2z7\") pod \"payload-pre-processing-e2e-api-a-fafb3f-788c86bfc9-s6m45\" (UID: \"f09ff8b4-d468-4c5e-9265-1f13766720f2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fafb3f-788c86bfc9-s6m45"
Oct 30 13:13:02.360065 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:02.349690    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fafb3f-788c86bfc9-s6m45"]
Oct 30 13:13:02.457627 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:02.416336    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6j6m\" (UniqueName: \"kubernetes.io/projected/55402878-108c-4c3d-b6c8-4362f82e82cb-kube-api-access-c6j6m\") pod \"payload-processing-e2e-api-a-fafb3f-7674fdb99d-v9prm\" (UID: \"55402878-108c-4c3d-b6c8-4362f82e82cb\") " pod="openshift-ingress/payload-processing-e2e-api-a-fafb3f-7674fdb99d-v9prm"
Oct 30 13:13:02.482261 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:02.476011    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t2z7\" (UniqueName: \"kubernetes.io/projected/f09ff8b4-d468-4c5e-9265-1f13766720f2-kube-api-access-4t2z7\") pod \"payload-pre-processing-e2e-api-a-fafb3f-788c86bfc9-s6m45\" (UID: \"f09ff8b4-d468-4c5e-9265-1f13766720f2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fafb3f-788c86bfc9-s6m45"
Oct 30 13:13:02.582619 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:02.581338    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fafb3f-788c86bfc9-s6m45"
Oct 30 13:13:02.668105 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:02.667992    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-fafb3f-7674fdb99d-v9prm"
Oct 30 13:13:03.035409 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:03.035376    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21460530-c5v8z"]
Oct 30 13:13:03.047044 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:03.046983    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21460530-c5v8z"]
Oct 30 13:13:03.119978 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:13:03.119956    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55402878_108c_4c3d_b6c8_4362f82e82cb.slice/crio-2782d5ec4f6bc631860172b9ab6a3e01409f2f8e731966253cf9c3e792147a21 WatchSource:0}: Error finding container 2782d5ec4f6bc631860172b9ab6a3e01409f2f8e731966253cf9c3e792147a21: Status 404 returned error can't find the container with id 2782d5ec4f6bc631860172b9ab6a3e01409f2f8e731966253cf9c3e792147a21
Oct 30 13:13:03.121895 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:03.121819    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fafb3f-7674fdb99d-v9prm"]
Oct 30 13:13:03.146116 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:13:03.146090    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf09ff8b4_d468_4c5e_9265_1f13766720f2.slice/crio-1d30a2958754eb81c21e2af60ac8e4a17dfbfe0cf85e3a9cde13201f39f77a1a WatchSource:0}: Error finding container 1d30a2958754eb81c21e2af60ac8e4a17dfbfe0cf85e3a9cde13201f39f77a1a: Status 404 returned error can't find the container with id 1d30a2958754eb81c21e2af60ac8e4a17dfbfe0cf85e3a9cde13201f39f77a1a
Oct 30 13:13:03.147764 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:03.147743    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fafb3f-788c86bfc9-s6m45"]
Oct 30 13:13:03.227571 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:03.227539    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fafb3f-788c86bfc9-s6m45" event={"ID":"f09ff8b4-d468-4c5e-9265-1f13766720f2","Type":"ContainerStarted","Data":"832487059ca81a31c8508ac93cec717f1c4f2a1b9953a3022f6c98d5f5dbfde8"}
Oct 30 13:13:03.227930 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:03.227622    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fafb3f-788c86bfc9-s6m45" event={"ID":"f09ff8b4-d468-4c5e-9265-1f13766720f2","Type":"ContainerStarted","Data":"1d30a2958754eb81c21e2af60ac8e4a17dfbfe0cf85e3a9cde13201f39f77a1a"}
Oct 30 13:13:03.227930 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:03.227657    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fafb3f-788c86bfc9-s6m45"
Oct 30 13:13:03.234061 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:03.233991    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-fafb3f-7674fdb99d-v9prm" event={"ID":"55402878-108c-4c3d-b6c8-4362f82e82cb","Type":"ContainerStarted","Data":"bf27e7a958b12cdad51267051d4ce1fe68d11bfdf869a28ee97b4c133a56f7e7"}
Oct 30 13:13:03.234061 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:03.234035    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-fafb3f-7674fdb99d-v9prm" event={"ID":"55402878-108c-4c3d-b6c8-4362f82e82cb","Type":"ContainerStarted","Data":"2782d5ec4f6bc631860172b9ab6a3e01409f2f8e731966253cf9c3e792147a21"}
Oct 30 13:13:03.234218 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:03.234149    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-fafb3f-7674fdb99d-v9prm"
Oct 30 13:13:03.273215 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:03.273176    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fafb3f-788c86bfc9-s6m45" podStartSLOduration=2.273163377 podStartE2EDuration="2.273163377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:13:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:13:03.271690028 +0000 UTC m=+1282.438054675" watchObservedRunningTime="2026-10-30 13:13:03.273163377 +0000 UTC m=+1282.439528063"
Oct 30 13:13:03.318502 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:03.318462    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-fafb3f-7674fdb99d-v9prm" podStartSLOduration=2.318450159 podStartE2EDuration="2.318450159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:13:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:13:03.317077901 +0000 UTC m=+1282.483442562" watchObservedRunningTime="2026-10-30 13:13:03.318450159 +0000 UTC m=+1282.484814806"
Oct 30 13:13:04.972425 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:04.972379    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8bfb829-6b1b-4788-ba44-2c2b92a73f66" path="/var/lib/kubelet/pods/b8bfb829-6b1b-4788-ba44-2c2b92a73f66/volumes"
Oct 30 13:13:11.648771 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:13:11.646635    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:13:14.244757 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:14.244727    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-fafb3f-7674fdb99d-v9prm"
Oct 30 13:13:14.245147 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:14.245024    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fafb3f-788c86bfc9-s6m45"
Oct 30 13:13:20.688622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:20.680778    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85"]
Oct 30 13:13:20.735669 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:20.733160    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85"
Oct 30 13:13:20.759620 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:20.753062    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-b29df4-openshift-default-dockercfg-ndf6g\""
Oct 30 13:13:21.034203 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:21.034139    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01c37c15-88bc-4359-9651-fdaae5a98c40-istiod-ca-cert\") pod \"e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85\" (UID: \"01c37c15-88bc-4359-9651-fdaae5a98c40\") " pod="openshift-ingress/e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85"
Oct 30 13:13:21.034515 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:21.034488    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01c37c15-88bc-4359-9651-fdaae5a98c40-istio-data\") pod \"e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85\" (UID: \"01c37c15-88bc-4359-9651-fdaae5a98c40\") " pod="openshift-ingress/e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85"
Oct 30 13:13:21.034737 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:21.034715    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01c37c15-88bc-4359-9651-fdaae5a98c40-credential-socket\") pod \"e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85\" (UID: \"01c37c15-88bc-4359-9651-fdaae5a98c40\") " pod="openshift-ingress/e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85"
Oct 30 13:13:21.034920 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:21.034900    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01c37c15-88bc-4359-9651-fdaae5a98c40-workload-certs\") pod \"e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85\" (UID: \"01c37c15-88bc-4359-9651-fdaae5a98c40\") " pod="openshift-ingress/e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85"
Oct 30 13:13:21.035193 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:21.035170    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01c37c15-88bc-4359-9651-fdaae5a98c40-istio-envoy\") pod \"e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85\" (UID: \"01c37c15-88bc-4359-9651-fdaae5a98c40\") " pod="openshift-ingress/e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85"
Oct 30 13:13:21.035484 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:21.035460    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01c37c15-88bc-4359-9651-fdaae5a98c40-workload-socket\") pod \"e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85\" (UID: \"01c37c15-88bc-4359-9651-fdaae5a98c40\") " pod="openshift-ingress/e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85"
Oct 30 13:13:21.035753 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:21.035730    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp7db\" (UniqueName: \"kubernetes.io/projected/01c37c15-88bc-4359-9651-fdaae5a98c40-kube-api-access-kp7db\") pod \"e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85\" (UID: \"01c37c15-88bc-4359-9651-fdaae5a98c40\") " pod="openshift-ingress/e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85"
Oct 30 13:13:21.035893 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:21.035877    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01c37c15-88bc-4359-9651-fdaae5a98c40-istio-token\") pod \"e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85\" (UID: \"01c37c15-88bc-4359-9651-fdaae5a98c40\") " pod="openshift-ingress/e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85"
Oct 30 13:13:21.036015 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:21.036001    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01c37c15-88bc-4359-9651-fdaae5a98c40-istio-podinfo\") pod \"e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85\" (UID: \"01c37c15-88bc-4359-9651-fdaae5a98c40\") " pod="openshift-ingress/e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85"
Oct 30 13:13:21.092069 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:21.092034    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85"]
Oct 30 13:13:21.146254 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:21.146214    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01c37c15-88bc-4359-9651-fdaae5a98c40-istio-envoy\") pod \"e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85\" (UID: \"01c37c15-88bc-4359-9651-fdaae5a98c40\") " pod="openshift-ingress/e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85"
Oct 30 13:13:21.146442 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:21.146398    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01c37c15-88bc-4359-9651-fdaae5a98c40-workload-socket\") pod \"e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85\" (UID: \"01c37c15-88bc-4359-9651-fdaae5a98c40\") " pod="openshift-ingress/e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85"
Oct 30 13:13:21.146516 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:21.146491    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp7db\" (UniqueName: \"kubernetes.io/projected/01c37c15-88bc-4359-9651-fdaae5a98c40-kube-api-access-kp7db\") pod \"e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85\" (UID: \"01c37c15-88bc-4359-9651-fdaae5a98c40\") " pod="openshift-ingress/e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85"
Oct 30 13:13:21.146573 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:21.146535    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01c37c15-88bc-4359-9651-fdaae5a98c40-istio-token\") pod \"e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85\" (UID: \"01c37c15-88bc-4359-9651-fdaae5a98c40\") " pod="openshift-ingress/e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85"
Oct 30 13:13:21.146678 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:21.146593    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01c37c15-88bc-4359-9651-fdaae5a98c40-istio-podinfo\") pod \"e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85\" (UID: \"01c37c15-88bc-4359-9651-fdaae5a98c40\") " pod="openshift-ingress/e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85"
Oct 30 13:13:21.146745 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:21.146699    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01c37c15-88bc-4359-9651-fdaae5a98c40-istiod-ca-cert\") pod \"e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85\" (UID: \"01c37c15-88bc-4359-9651-fdaae5a98c40\") " pod="openshift-ingress/e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85"
Oct 30 13:13:21.146800 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:21.146744    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01c37c15-88bc-4359-9651-fdaae5a98c40-istio-data\") pod \"e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85\" (UID: \"01c37c15-88bc-4359-9651-fdaae5a98c40\") " pod="openshift-ingress/e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85"
Oct 30 13:13:21.146853 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:21.146836    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01c37c15-88bc-4359-9651-fdaae5a98c40-credential-socket\") pod \"e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85\" (UID: \"01c37c15-88bc-4359-9651-fdaae5a98c40\") " pod="openshift-ingress/e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85"
Oct 30 13:13:21.146938 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:21.146923    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01c37c15-88bc-4359-9651-fdaae5a98c40-workload-certs\") pod \"e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85\" (UID: \"01c37c15-88bc-4359-9651-fdaae5a98c40\") " pod="openshift-ingress/e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85"
Oct 30 13:13:21.147404 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:21.147383    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01c37c15-88bc-4359-9651-fdaae5a98c40-workload-certs\") pod \"e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85\" (UID: \"01c37c15-88bc-4359-9651-fdaae5a98c40\") " pod="openshift-ingress/e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85"
Oct 30 13:13:21.148837 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:21.148438    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01c37c15-88bc-4359-9651-fdaae5a98c40-istiod-ca-cert\") pod \"e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85\" (UID: \"01c37c15-88bc-4359-9651-fdaae5a98c40\") " pod="openshift-ingress/e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85"
Oct 30 13:13:21.149060 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:21.149035    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01c37c15-88bc-4359-9651-fdaae5a98c40-istio-data\") pod \"e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85\" (UID: \"01c37c15-88bc-4359-9651-fdaae5a98c40\") " pod="openshift-ingress/e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85"
Oct 30 13:13:21.157657 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:21.153066    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01c37c15-88bc-4359-9651-fdaae5a98c40-workload-socket\") pod \"e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85\" (UID: \"01c37c15-88bc-4359-9651-fdaae5a98c40\") " pod="openshift-ingress/e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85"
Oct 30 13:13:21.158308 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:21.158268    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01c37c15-88bc-4359-9651-fdaae5a98c40-credential-socket\") pod \"e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85\" (UID: \"01c37c15-88bc-4359-9651-fdaae5a98c40\") " pod="openshift-ingress/e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85"
Oct 30 13:13:21.196261 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:21.195901    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01c37c15-88bc-4359-9651-fdaae5a98c40-istio-envoy\") pod \"e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85\" (UID: \"01c37c15-88bc-4359-9651-fdaae5a98c40\") " pod="openshift-ingress/e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85"
Oct 30 13:13:21.196261 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:21.196241    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp7db\" (UniqueName: \"kubernetes.io/projected/01c37c15-88bc-4359-9651-fdaae5a98c40-kube-api-access-kp7db\") pod \"e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85\" (UID: \"01c37c15-88bc-4359-9651-fdaae5a98c40\") " pod="openshift-ingress/e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85"
Oct 30 13:13:21.197081 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:21.197021    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01c37c15-88bc-4359-9651-fdaae5a98c40-istio-podinfo\") pod \"e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85\" (UID: \"01c37c15-88bc-4359-9651-fdaae5a98c40\") " pod="openshift-ingress/e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85"
Oct 30 13:13:21.199177 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:21.198895    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01c37c15-88bc-4359-9651-fdaae5a98c40-istio-token\") pod \"e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85\" (UID: \"01c37c15-88bc-4359-9651-fdaae5a98c40\") " pod="openshift-ingress/e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85"
Oct 30 13:13:21.376477 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:21.376444    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85"
Oct 30 13:13:21.706019 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:13:21.705978    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01c37c15_88bc_4359_9651_fdaae5a98c40.slice/crio-a4fbf50ddd2967977deef7975c07e7bb5cad3ad2cd7aeec5ea0e2415a030eb21 WatchSource:0}: Error finding container a4fbf50ddd2967977deef7975c07e7bb5cad3ad2cd7aeec5ea0e2415a030eb21: Status 404 returned error can't find the container with id a4fbf50ddd2967977deef7975c07e7bb5cad3ad2cd7aeec5ea0e2415a030eb21
Oct 30 13:13:21.707708 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:21.707684    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85"]
Oct 30 13:13:21.708823 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:21.708791    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:13:21.708919 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:21.708868    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:13:21.708919 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:21.708913    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:13:21.719141 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:21.719117    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85" event={"ID":"01c37c15-88bc-4359-9651-fdaae5a98c40","Type":"ContainerStarted","Data":"a4fbf50ddd2967977deef7975c07e7bb5cad3ad2cd7aeec5ea0e2415a030eb21"}
Oct 30 13:13:22.729440 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:22.729407    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85" event={"ID":"01c37c15-88bc-4359-9651-fdaae5a98c40","Type":"ContainerStarted","Data":"6e3b5d4a99c8fa36fefdd8cc9cb28310c2a270188417fa6fd0de5a998c86c16e"}
Oct 30 13:13:22.809163 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:22.809115    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85" podStartSLOduration=2.809101692 podStartE2EDuration="2.809101692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:13:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:13:22.79338086 +0000 UTC m=+1301.959745511" watchObservedRunningTime="2026-10-30 13:13:22.809101692 +0000 UTC m=+1301.975466393"
Oct 30 13:13:23.376674 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:23.376646    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85"
Oct 30 13:13:23.382330 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:23.382307    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85"
Oct 30 13:13:23.738192 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:23.738102    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85"
Oct 30 13:13:23.738993 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:23.738977    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85"
Oct 30 13:13:27.343654 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:27.343621    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b29df4-857fc5bcc9-lw9kb"]
Oct 30 13:13:27.348438 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:27.348418    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b29df4-857fc5bcc9-lw9kb"
Oct 30 13:13:27.366335 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:27.366314    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-b29df4-dockercfg-cqh6f\""
Oct 30 13:13:27.374649 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:27.372737    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b29df4-857fc5bcc9-lw9kb"]
Oct 30 13:13:27.376426 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:27.376396    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49bf7b08-c971-4722-af65-b145ed2b1f21-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-b29df4-857fc5bcc9-lw9kb\" (UID: \"49bf7b08-c971-4722-af65-b145ed2b1f21\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b29df4-857fc5bcc9-lw9kb"
Oct 30 13:13:27.376768 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:27.376740    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wl4q\" (UniqueName: \"kubernetes.io/projected/49bf7b08-c971-4722-af65-b145ed2b1f21-kube-api-access-4wl4q\") pod \"payload-pre-processing-e2e-shared-b-w4-b29df4-857fc5bcc9-lw9kb\" (UID: \"49bf7b08-c971-4722-af65-b145ed2b1f21\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b29df4-857fc5bcc9-lw9kb"
Oct 30 13:13:27.379221 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:27.379197    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-b29df4\""
Oct 30 13:13:27.486830 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:27.486786    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wl4q\" (UniqueName: \"kubernetes.io/projected/49bf7b08-c971-4722-af65-b145ed2b1f21-kube-api-access-4wl4q\") pod \"payload-pre-processing-e2e-shared-b-w4-b29df4-857fc5bcc9-lw9kb\" (UID: \"49bf7b08-c971-4722-af65-b145ed2b1f21\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b29df4-857fc5bcc9-lw9kb"
Oct 30 13:13:27.488516 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:27.488487    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49bf7b08-c971-4722-af65-b145ed2b1f21-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-b29df4-857fc5bcc9-lw9kb\" (UID: \"49bf7b08-c971-4722-af65-b145ed2b1f21\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b29df4-857fc5bcc9-lw9kb"
Oct 30 13:13:27.490019 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:27.489980    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49bf7b08-c971-4722-af65-b145ed2b1f21-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-b29df4-857fc5bcc9-lw9kb\" (UID: \"49bf7b08-c971-4722-af65-b145ed2b1f21\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b29df4-857fc5bcc9-lw9kb"
Oct 30 13:13:27.686637 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:27.685251    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wl4q\" (UniqueName: \"kubernetes.io/projected/49bf7b08-c971-4722-af65-b145ed2b1f21-kube-api-access-4wl4q\") pod \"payload-pre-processing-e2e-shared-b-w4-b29df4-857fc5bcc9-lw9kb\" (UID: \"49bf7b08-c971-4722-af65-b145ed2b1f21\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b29df4-857fc5bcc9-lw9kb"
Oct 30 13:13:27.964936 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:27.964839    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b29df4-857fc5bcc9-lw9kb"
Oct 30 13:13:28.274202 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:13:28.274166    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49bf7b08_c971_4722_af65_b145ed2b1f21.slice/crio-2a75e15f6b5c18b5ab49d7fe7f5bc8831cd8bdbdfa0431baac0e70ab59f0b36c WatchSource:0}: Error finding container 2a75e15f6b5c18b5ab49d7fe7f5bc8831cd8bdbdfa0431baac0e70ab59f0b36c: Status 404 returned error can't find the container with id 2a75e15f6b5c18b5ab49d7fe7f5bc8831cd8bdbdfa0431baac0e70ab59f0b36c
Oct 30 13:13:28.275662 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:28.275637    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b29df4-857fc5bcc9-lw9kb"]
Oct 30 13:13:28.836583 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:28.836541    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b29df4-857fc5bcc9-lw9kb" event={"ID":"49bf7b08-c971-4722-af65-b145ed2b1f21","Type":"ContainerStarted","Data":"0c9916c6065a29944e8de212bf1390499e8b338060c0e5ed28a7d002b3b3270d"}
Oct 30 13:13:28.837040 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:28.836627    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b29df4-857fc5bcc9-lw9kb"
Oct 30 13:13:28.837040 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:28.836652    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b29df4-857fc5bcc9-lw9kb" event={"ID":"49bf7b08-c971-4722-af65-b145ed2b1f21","Type":"ContainerStarted","Data":"2a75e15f6b5c18b5ab49d7fe7f5bc8831cd8bdbdfa0431baac0e70ab59f0b36c"}
Oct 30 13:13:28.883352 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:28.883307    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b29df4-857fc5bcc9-lw9kb" podStartSLOduration=1.883294435 podStartE2EDuration="1.883294435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:13:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:13:28.881453513 +0000 UTC m=+1308.047818166" watchObservedRunningTime="2026-10-30 13:13:28.883294435 +0000 UTC m=+1308.049659082"
Oct 30 13:13:39.033763 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:39.033727    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29ceb91f-lw2hx"]
Oct 30 13:13:39.046486 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:39.046460    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29ceb91f-lw2hx"]
Oct 30 13:13:39.865107 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:39.865082    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b29df4-857fc5bcc9-lw9kb"
Oct 30 13:13:41.280898 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:41.280859    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a0e173e-b428-45f5-b91a-095475c67cae" path="/var/lib/kubelet/pods/1a0e173e-b428-45f5-b91a-095475c67cae/volumes"
Oct 30 13:13:41.523584 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:13:41.523549    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:13:43.186998 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:43.186956    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:13:46.682122 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:46.682080    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-659d72-c87c7d4f5-cn8jk"]
Oct 30 13:13:46.688427 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:46.688403    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-659d72-c87c7d4f5-cn8jk"
Oct 30 13:13:46.696861 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:46.696784    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b564990b-f890-4e82-a667-c24363ec69d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-659d72-c87c7d4f5-cn8jk\" (UID: \"b564990b-f890-4e82-a667-c24363ec69d2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-659d72-c87c7d4f5-cn8jk"
Oct 30 13:13:46.697035 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:46.697013    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwdtk\" (UniqueName: \"kubernetes.io/projected/b564990b-f890-4e82-a667-c24363ec69d2-kube-api-access-dwdtk\") pod \"payload-pre-processing-e2e-api-a-659d72-c87c7d4f5-cn8jk\" (UID: \"b564990b-f890-4e82-a667-c24363ec69d2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-659d72-c87c7d4f5-cn8jk"
Oct 30 13:13:46.705821 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:46.705797    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-659d72-c87c7d4f5-cn8jk"]
Oct 30 13:13:46.713374 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:46.713318    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-659d72-dockercfg-jfq2k\""
Oct 30 13:13:46.861624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:46.857739    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-659d72\""
Oct 30 13:13:46.909623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:46.901149    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwdtk\" (UniqueName: \"kubernetes.io/projected/b564990b-f890-4e82-a667-c24363ec69d2-kube-api-access-dwdtk\") pod \"payload-pre-processing-e2e-api-a-659d72-c87c7d4f5-cn8jk\" (UID: \"b564990b-f890-4e82-a667-c24363ec69d2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-659d72-c87c7d4f5-cn8jk"
Oct 30 13:13:46.909623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:46.901340    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b564990b-f890-4e82-a667-c24363ec69d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-659d72-c87c7d4f5-cn8jk\" (UID: \"b564990b-f890-4e82-a667-c24363ec69d2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-659d72-c87c7d4f5-cn8jk"
Oct 30 13:13:46.911325 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:46.910717    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b564990b-f890-4e82-a667-c24363ec69d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-659d72-c87c7d4f5-cn8jk\" (UID: \"b564990b-f890-4e82-a667-c24363ec69d2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-659d72-c87c7d4f5-cn8jk"
Oct 30 13:13:47.134629 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:47.130133    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwdtk\" (UniqueName: \"kubernetes.io/projected/b564990b-f890-4e82-a667-c24363ec69d2-kube-api-access-dwdtk\") pod \"payload-pre-processing-e2e-api-a-659d72-c87c7d4f5-cn8jk\" (UID: \"b564990b-f890-4e82-a667-c24363ec69d2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-659d72-c87c7d4f5-cn8jk"
Oct 30 13:13:47.318507 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:47.311872    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-659d72-c87c7d4f5-cn8jk"
Oct 30 13:13:47.657626 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:13:47.657579    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb564990b_f890_4e82_a667_c24363ec69d2.slice/crio-dcf614d8035c004f60776016ce173e9b7aeecb0048ca758d00e42d63b68422f1 WatchSource:0}: Error finding container dcf614d8035c004f60776016ce173e9b7aeecb0048ca758d00e42d63b68422f1: Status 404 returned error can't find the container with id dcf614d8035c004f60776016ce173e9b7aeecb0048ca758d00e42d63b68422f1
Oct 30 13:13:47.659237 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:47.659215    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-659d72-c87c7d4f5-cn8jk"]
Oct 30 13:13:48.458065 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:48.458031    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-659d72-c87c7d4f5-cn8jk" event={"ID":"b564990b-f890-4e82-a667-c24363ec69d2","Type":"ContainerStarted","Data":"d2cb9b717380728e4b1654b82ef84cc7d757f777c62a49d209332c25b01800f3"}
Oct 30 13:13:48.458453 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:48.458088    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-659d72-c87c7d4f5-cn8jk" event={"ID":"b564990b-f890-4e82-a667-c24363ec69d2","Type":"ContainerStarted","Data":"dcf614d8035c004f60776016ce173e9b7aeecb0048ca758d00e42d63b68422f1"}
Oct 30 13:13:48.458453 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:48.458109    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-659d72-c87c7d4f5-cn8jk"
Oct 30 13:13:48.508368 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:48.508317    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-659d72-c87c7d4f5-cn8jk" podStartSLOduration=2.508300169 podStartE2EDuration="2.508300169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:13:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:13:48.502747569 +0000 UTC m=+1327.669112221" watchObservedRunningTime="2026-10-30 13:13:48.508300169 +0000 UTC m=+1327.674664827"
Oct 30 13:13:54.050695 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:54.050658    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ace0686d-ljqxb"]
Oct 30 13:13:54.062882 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:54.062832    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ace0686d-ljqxb"]
Oct 30 13:13:54.972002 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:54.971969    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e4db309-7b4f-4752-84c5-64c248011b6a" path="/var/lib/kubelet/pods/8e4db309-7b4f-4752-84c5-64c248011b6a/volumes"
Oct 30 13:13:59.480507 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:13:59.480480    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-659d72-c87c7d4f5-cn8jk"
Oct 30 13:14:02.203498 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:02.203460    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx"]
Oct 30 13:14:02.208388 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:02.208368    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx"
Oct 30 13:14:02.233562 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:02.233535    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx"]
Oct 30 13:14:02.240368 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:02.240341    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-6b0881-openshift-default-dockercfg-mb9nl\""
Oct 30 13:14:02.339262 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:02.339231    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd6e6964-3e54-48f1-a184-8db6ef093a7b-istio-podinfo\") pod \"e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx\" (UID: \"cd6e6964-3e54-48f1-a184-8db6ef093a7b\") " pod="openshift-ingress/e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx"
Oct 30 13:14:02.339380 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:02.339342    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd6e6964-3e54-48f1-a184-8db6ef093a7b-istiod-ca-cert\") pod \"e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx\" (UID: \"cd6e6964-3e54-48f1-a184-8db6ef093a7b\") " pod="openshift-ingress/e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx"
Oct 30 13:14:02.339438 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:02.339384    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd6e6964-3e54-48f1-a184-8db6ef093a7b-istio-envoy\") pod \"e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx\" (UID: \"cd6e6964-3e54-48f1-a184-8db6ef093a7b\") " pod="openshift-ingress/e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx"
Oct 30 13:14:02.339438 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:02.339408    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd6e6964-3e54-48f1-a184-8db6ef093a7b-credential-socket\") pod \"e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx\" (UID: \"cd6e6964-3e54-48f1-a184-8db6ef093a7b\") " pod="openshift-ingress/e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx"
Oct 30 13:14:02.339520 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:02.339463    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd6e6964-3e54-48f1-a184-8db6ef093a7b-workload-socket\") pod \"e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx\" (UID: \"cd6e6964-3e54-48f1-a184-8db6ef093a7b\") " pod="openshift-ingress/e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx"
Oct 30 13:14:02.339563 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:02.339515    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54qcz\" (UniqueName: \"kubernetes.io/projected/cd6e6964-3e54-48f1-a184-8db6ef093a7b-kube-api-access-54qcz\") pod \"e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx\" (UID: \"cd6e6964-3e54-48f1-a184-8db6ef093a7b\") " pod="openshift-ingress/e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx"
Oct 30 13:14:02.339642 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:02.339576    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd6e6964-3e54-48f1-a184-8db6ef093a7b-istio-token\") pod \"e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx\" (UID: \"cd6e6964-3e54-48f1-a184-8db6ef093a7b\") " pod="openshift-ingress/e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx"
Oct 30 13:14:02.339725 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:02.339705    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd6e6964-3e54-48f1-a184-8db6ef093a7b-workload-certs\") pod \"e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx\" (UID: \"cd6e6964-3e54-48f1-a184-8db6ef093a7b\") " pod="openshift-ingress/e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx"
Oct 30 13:14:02.339838 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:02.339824    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd6e6964-3e54-48f1-a184-8db6ef093a7b-istio-data\") pod \"e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx\" (UID: \"cd6e6964-3e54-48f1-a184-8db6ef093a7b\") " pod="openshift-ingress/e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx"
Oct 30 13:14:02.444626 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:02.443810    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd6e6964-3e54-48f1-a184-8db6ef093a7b-istiod-ca-cert\") pod \"e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx\" (UID: \"cd6e6964-3e54-48f1-a184-8db6ef093a7b\") " pod="openshift-ingress/e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx"
Oct 30 13:14:02.444626 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:02.443878    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd6e6964-3e54-48f1-a184-8db6ef093a7b-istio-envoy\") pod \"e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx\" (UID: \"cd6e6964-3e54-48f1-a184-8db6ef093a7b\") " pod="openshift-ingress/e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx"
Oct 30 13:14:02.444626 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:02.443960    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd6e6964-3e54-48f1-a184-8db6ef093a7b-credential-socket\") pod \"e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx\" (UID: \"cd6e6964-3e54-48f1-a184-8db6ef093a7b\") " pod="openshift-ingress/e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx"
Oct 30 13:14:02.444626 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:02.443991    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd6e6964-3e54-48f1-a184-8db6ef093a7b-workload-socket\") pod \"e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx\" (UID: \"cd6e6964-3e54-48f1-a184-8db6ef093a7b\") " pod="openshift-ingress/e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx"
Oct 30 13:14:02.444626 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:02.444032    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54qcz\" (UniqueName: \"kubernetes.io/projected/cd6e6964-3e54-48f1-a184-8db6ef093a7b-kube-api-access-54qcz\") pod \"e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx\" (UID: \"cd6e6964-3e54-48f1-a184-8db6ef093a7b\") " pod="openshift-ingress/e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx"
Oct 30 13:14:02.444626 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:02.444228    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd6e6964-3e54-48f1-a184-8db6ef093a7b-istio-token\") pod \"e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx\" (UID: \"cd6e6964-3e54-48f1-a184-8db6ef093a7b\") " pod="openshift-ingress/e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx"
Oct 30 13:14:02.444626 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:02.444324    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd6e6964-3e54-48f1-a184-8db6ef093a7b-workload-certs\") pod \"e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx\" (UID: \"cd6e6964-3e54-48f1-a184-8db6ef093a7b\") " pod="openshift-ingress/e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx"
Oct 30 13:14:02.444626 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:02.444488    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd6e6964-3e54-48f1-a184-8db6ef093a7b-istio-data\") pod \"e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx\" (UID: \"cd6e6964-3e54-48f1-a184-8db6ef093a7b\") " pod="openshift-ingress/e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx"
Oct 30 13:14:02.445187 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:02.444679    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd6e6964-3e54-48f1-a184-8db6ef093a7b-istio-podinfo\") pod \"e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx\" (UID: \"cd6e6964-3e54-48f1-a184-8db6ef093a7b\") " pod="openshift-ingress/e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx"
Oct 30 13:14:02.445859 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:02.445836    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd6e6964-3e54-48f1-a184-8db6ef093a7b-workload-socket\") pod \"e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx\" (UID: \"cd6e6964-3e54-48f1-a184-8db6ef093a7b\") " pod="openshift-ingress/e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx"
Oct 30 13:14:02.494054 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:02.454514    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd6e6964-3e54-48f1-a184-8db6ef093a7b-istiod-ca-cert\") pod \"e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx\" (UID: \"cd6e6964-3e54-48f1-a184-8db6ef093a7b\") " pod="openshift-ingress/e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx"
Oct 30 13:14:02.494054 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:02.458880    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd6e6964-3e54-48f1-a184-8db6ef093a7b-workload-certs\") pod \"e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx\" (UID: \"cd6e6964-3e54-48f1-a184-8db6ef093a7b\") " pod="openshift-ingress/e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx"
Oct 30 13:14:02.494054 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:02.459098    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd6e6964-3e54-48f1-a184-8db6ef093a7b-istio-data\") pod \"e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx\" (UID: \"cd6e6964-3e54-48f1-a184-8db6ef093a7b\") " pod="openshift-ingress/e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx"
Oct 30 13:14:02.494054 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:02.477654    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd6e6964-3e54-48f1-a184-8db6ef093a7b-credential-socket\") pod \"e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx\" (UID: \"cd6e6964-3e54-48f1-a184-8db6ef093a7b\") " pod="openshift-ingress/e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx"
Oct 30 13:14:02.515621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:02.498363    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd6e6964-3e54-48f1-a184-8db6ef093a7b-istio-podinfo\") pod \"e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx\" (UID: \"cd6e6964-3e54-48f1-a184-8db6ef093a7b\") " pod="openshift-ingress/e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx"
Oct 30 13:14:02.515878 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:02.515726    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54qcz\" (UniqueName: \"kubernetes.io/projected/cd6e6964-3e54-48f1-a184-8db6ef093a7b-kube-api-access-54qcz\") pod \"e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx\" (UID: \"cd6e6964-3e54-48f1-a184-8db6ef093a7b\") " pod="openshift-ingress/e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx"
Oct 30 13:14:02.546386 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:02.545156    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd6e6964-3e54-48f1-a184-8db6ef093a7b-istio-envoy\") pod \"e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx\" (UID: \"cd6e6964-3e54-48f1-a184-8db6ef093a7b\") " pod="openshift-ingress/e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx"
Oct 30 13:14:02.546386 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:02.545569    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd6e6964-3e54-48f1-a184-8db6ef093a7b-istio-token\") pod \"e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx\" (UID: \"cd6e6964-3e54-48f1-a184-8db6ef093a7b\") " pod="openshift-ingress/e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx"
Oct 30 13:14:02.832116 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:02.827178    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx"
Oct 30 13:14:03.288059 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:14:03.288019    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd6e6964_3e54_48f1_a184_8db6ef093a7b.slice/crio-2e83e0b4ac590f5b7665a2d9a7bfaf600fa0842f2a66aca2cfa53086fa7b6574 WatchSource:0}: Error finding container 2e83e0b4ac590f5b7665a2d9a7bfaf600fa0842f2a66aca2cfa53086fa7b6574: Status 404 returned error can't find the container with id 2e83e0b4ac590f5b7665a2d9a7bfaf600fa0842f2a66aca2cfa53086fa7b6574
Oct 30 13:14:03.291041 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:03.290998    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:14:03.291173 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:03.291100    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:14:03.291173 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:03.291148    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:14:03.292281 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:03.292259    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx"]
Oct 30 13:14:03.662196 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:03.662159    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx" event={"ID":"cd6e6964-3e54-48f1-a184-8db6ef093a7b","Type":"ContainerStarted","Data":"5bcc210571b46ecb21eae031ebf90fca908e5b4d1d757005c3eec4e48ba9b6be"}
Oct 30 13:14:03.662381 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:03.662225    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx" event={"ID":"cd6e6964-3e54-48f1-a184-8db6ef093a7b","Type":"ContainerStarted","Data":"2e83e0b4ac590f5b7665a2d9a7bfaf600fa0842f2a66aca2cfa53086fa7b6574"}
Oct 30 13:14:03.714974 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:03.714927    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx" podStartSLOduration=1.714913829 podStartE2EDuration="1.714913829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:14:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:14:03.712066152 +0000 UTC m=+1342.878430794" watchObservedRunningTime="2026-10-30 13:14:03.714913829 +0000 UTC m=+1342.881278476"
Oct 30 13:14:03.846629 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:03.836676    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx"
Oct 30 13:14:04.833914 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:04.833885    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx"
Oct 30 13:14:05.678179 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:05.678149    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx"
Oct 30 13:14:05.679087 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:05.679068    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx"
Oct 30 13:14:08.006034 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:08.005998    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6b0881-56688d7bdb-jfqwc"]
Oct 30 13:14:08.012253 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:08.011530    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6b0881-56688d7bdb-jfqwc"
Oct 30 13:14:08.025463 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:08.025407    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9w4f\" (UniqueName: \"kubernetes.io/projected/afeac9f9-fa22-47b0-8446-9683377383e2-kube-api-access-r9w4f\") pod \"payload-processing-e2e-api-b-6b0881-56688d7bdb-jfqwc\" (UID: \"afeac9f9-fa22-47b0-8446-9683377383e2\") " pod="openshift-ingress/payload-processing-e2e-api-b-6b0881-56688d7bdb-jfqwc"
Oct 30 13:14:08.025748 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:08.025707    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afeac9f9-fa22-47b0-8446-9683377383e2-plugins-config-volume\") pod \"payload-processing-e2e-api-b-6b0881-56688d7bdb-jfqwc\" (UID: \"afeac9f9-fa22-47b0-8446-9683377383e2\") " pod="openshift-ingress/payload-processing-e2e-api-b-6b0881-56688d7bdb-jfqwc"
Oct 30 13:14:08.034318 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:08.034295    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6b0881-56688d7bdb-jfqwc"]
Oct 30 13:14:08.041867 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:08.041831    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-6b0881\""
Oct 30 13:14:08.042054 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:08.041932    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-6b0881-dockercfg-z5hfs\""
Oct 30 13:14:08.082415 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:08.082380    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6b0881-cf9d79bc-8vds2"]
Oct 30 13:14:08.088405 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:08.088382    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6b0881-cf9d79bc-8vds2"
Oct 30 13:14:08.112408 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:08.112381    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6b0881-cf9d79bc-8vds2"]
Oct 30 13:14:08.130608 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:08.129775    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9w4f\" (UniqueName: \"kubernetes.io/projected/afeac9f9-fa22-47b0-8446-9683377383e2-kube-api-access-r9w4f\") pod \"payload-processing-e2e-api-b-6b0881-56688d7bdb-jfqwc\" (UID: \"afeac9f9-fa22-47b0-8446-9683377383e2\") " pod="openshift-ingress/payload-processing-e2e-api-b-6b0881-56688d7bdb-jfqwc"
Oct 30 13:14:08.130608 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:08.129859    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afeac9f9-fa22-47b0-8446-9683377383e2-plugins-config-volume\") pod \"payload-processing-e2e-api-b-6b0881-56688d7bdb-jfqwc\" (UID: \"afeac9f9-fa22-47b0-8446-9683377383e2\") " pod="openshift-ingress/payload-processing-e2e-api-b-6b0881-56688d7bdb-jfqwc"
Oct 30 13:14:08.130608 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:08.129952    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm6gt\" (UniqueName: \"kubernetes.io/projected/bbbe037e-4f22-4ecd-9d73-d8d32419647f-kube-api-access-fm6gt\") pod \"payload-pre-processing-e2e-api-b-6b0881-cf9d79bc-8vds2\" (UID: \"bbbe037e-4f22-4ecd-9d73-d8d32419647f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6b0881-cf9d79bc-8vds2"
Oct 30 13:14:08.131087 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:08.131025    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afeac9f9-fa22-47b0-8446-9683377383e2-plugins-config-volume\") pod \"payload-processing-e2e-api-b-6b0881-56688d7bdb-jfqwc\" (UID: \"afeac9f9-fa22-47b0-8446-9683377383e2\") " pod="openshift-ingress/payload-processing-e2e-api-b-6b0881-56688d7bdb-jfqwc"
Oct 30 13:14:08.131514 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:08.131487    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbbe037e-4f22-4ecd-9d73-d8d32419647f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-6b0881-cf9d79bc-8vds2\" (UID: \"bbbe037e-4f22-4ecd-9d73-d8d32419647f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6b0881-cf9d79bc-8vds2"
Oct 30 13:14:08.239623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:08.236413    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm6gt\" (UniqueName: \"kubernetes.io/projected/bbbe037e-4f22-4ecd-9d73-d8d32419647f-kube-api-access-fm6gt\") pod \"payload-pre-processing-e2e-api-b-6b0881-cf9d79bc-8vds2\" (UID: \"bbbe037e-4f22-4ecd-9d73-d8d32419647f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6b0881-cf9d79bc-8vds2"
Oct 30 13:14:08.239623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:08.236717    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbbe037e-4f22-4ecd-9d73-d8d32419647f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-6b0881-cf9d79bc-8vds2\" (UID: \"bbbe037e-4f22-4ecd-9d73-d8d32419647f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6b0881-cf9d79bc-8vds2"
Oct 30 13:14:08.239623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:08.237817    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbbe037e-4f22-4ecd-9d73-d8d32419647f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-6b0881-cf9d79bc-8vds2\" (UID: \"bbbe037e-4f22-4ecd-9d73-d8d32419647f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6b0881-cf9d79bc-8vds2"
Oct 30 13:14:08.330681 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:08.307312    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9w4f\" (UniqueName: \"kubernetes.io/projected/afeac9f9-fa22-47b0-8446-9683377383e2-kube-api-access-r9w4f\") pod \"payload-processing-e2e-api-b-6b0881-56688d7bdb-jfqwc\" (UID: \"afeac9f9-fa22-47b0-8446-9683377383e2\") " pod="openshift-ingress/payload-processing-e2e-api-b-6b0881-56688d7bdb-jfqwc"
Oct 30 13:14:08.511632 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:08.485533    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm6gt\" (UniqueName: \"kubernetes.io/projected/bbbe037e-4f22-4ecd-9d73-d8d32419647f-kube-api-access-fm6gt\") pod \"payload-pre-processing-e2e-api-b-6b0881-cf9d79bc-8vds2\" (UID: \"bbbe037e-4f22-4ecd-9d73-d8d32419647f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6b0881-cf9d79bc-8vds2"
Oct 30 13:14:08.633933 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:08.633893    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6b0881-56688d7bdb-jfqwc"
Oct 30 13:14:08.768227 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:08.768188    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6b0881-cf9d79bc-8vds2"
Oct 30 13:14:08.967935 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:14:08.967903    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafeac9f9_fa22_47b0_8446_9683377383e2.slice/crio-ba14d550484e8b97665c073ab3665a2b7a2e66852afa0fd38c33a99823183bcf WatchSource:0}: Error finding container ba14d550484e8b97665c073ab3665a2b7a2e66852afa0fd38c33a99823183bcf: Status 404 returned error can't find the container with id ba14d550484e8b97665c073ab3665a2b7a2e66852afa0fd38c33a99823183bcf
Oct 30 13:14:08.973805 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:08.973781    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6b0881-56688d7bdb-jfqwc"]
Oct 30 13:14:09.097487 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:14:09.097459    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbbe037e_4f22_4ecd_9d73_d8d32419647f.slice/crio-bc26edc017eaf98e2fba62715e250cbbd1d5bd18d1c885d4c8c56acc767d9649 WatchSource:0}: Error finding container bc26edc017eaf98e2fba62715e250cbbd1d5bd18d1c885d4c8c56acc767d9649: Status 404 returned error can't find the container with id bc26edc017eaf98e2fba62715e250cbbd1d5bd18d1c885d4c8c56acc767d9649
Oct 30 13:14:09.099294 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:09.099275    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6b0881-cf9d79bc-8vds2"]
Oct 30 13:14:09.761924 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:09.761885    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6b0881-cf9d79bc-8vds2" event={"ID":"bbbe037e-4f22-4ecd-9d73-d8d32419647f","Type":"ContainerStarted","Data":"15c4f44d30b26687802ef40c690aa7ba191cb8f8a7da687340e610adbc24434c"}
Oct 30 13:14:09.762090 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:09.761952    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6b0881-cf9d79bc-8vds2" event={"ID":"bbbe037e-4f22-4ecd-9d73-d8d32419647f","Type":"ContainerStarted","Data":"bc26edc017eaf98e2fba62715e250cbbd1d5bd18d1c885d4c8c56acc767d9649"}
Oct 30 13:14:09.762090 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:09.761985    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6b0881-cf9d79bc-8vds2"
Oct 30 13:14:09.763577 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:09.763553    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6b0881-56688d7bdb-jfqwc" event={"ID":"afeac9f9-fa22-47b0-8446-9683377383e2","Type":"ContainerStarted","Data":"0198b8b567b08ac6aa51fbf71bf019ef7a3302f5d980165813e01bf721cfb72a"}
Oct 30 13:14:09.763718 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:09.763591    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6b0881-56688d7bdb-jfqwc" event={"ID":"afeac9f9-fa22-47b0-8446-9683377383e2","Type":"ContainerStarted","Data":"ba14d550484e8b97665c073ab3665a2b7a2e66852afa0fd38c33a99823183bcf"}
Oct 30 13:14:09.763718 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:09.763646    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-6b0881-56688d7bdb-jfqwc"
Oct 30 13:14:09.814646 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:09.814581    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6b0881-cf9d79bc-8vds2" podStartSLOduration=1.8145660989999999 podStartE2EDuration="1.814566099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:14:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:14:09.810753003 +0000 UTC m=+1348.977117655" watchObservedRunningTime="2026-10-30 13:14:09.814566099 +0000 UTC m=+1348.980930747"
Oct 30 13:14:09.869558 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:09.869499    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-6b0881-56688d7bdb-jfqwc" podStartSLOduration=2.869484123 podStartE2EDuration="2.869484123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:14:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:14:09.869033882 +0000 UTC m=+1349.035398531" watchObservedRunningTime="2026-10-30 13:14:09.869484123 +0000 UTC m=+1349.035848773"
Oct 30 13:14:11.506064 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:14:11.506031    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:14:18.032191 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:18.032156    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aff624f9-44trc"]
Oct 30 13:14:18.037119 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:18.037095    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aff624f9-44trc"]
Oct 30 13:14:18.973366 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:18.973332    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f075f88-c8cd-4215-be7b-be7f2da241be" path="/var/lib/kubelet/pods/8f075f88-c8cd-4215-be7b-be7f2da241be/volumes"
Oct 30 13:14:20.775527 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:20.775501    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6b0881-cf9d79bc-8vds2"
Oct 30 13:14:20.775961 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:20.775559    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-6b0881-56688d7bdb-jfqwc"
Oct 30 13:14:20.937738 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:20.937706    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:14:28.769842 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:28.769812    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9fcb69-59fd4c8c56-w96vh"]
Oct 30 13:14:28.776061 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:28.776041    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9fcb69-59fd4c8c56-w96vh"
Oct 30 13:14:28.811485 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:28.811439    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9fcb69-59fd4c8c56-w96vh"]
Oct 30 13:14:28.814923 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:28.814899    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-9fcb69\""
Oct 30 13:14:28.815163 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:28.815148    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-9fcb69-dockercfg-tm5ns\""
Oct 30 13:14:28.957621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:28.924264    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6zx7\" (UniqueName: \"kubernetes.io/projected/79f77504-1067-40e5-9ea6-1f7b752d0f8d-kube-api-access-f6zx7\") pod \"payload-processing-e2e-api-a-9fcb69-59fd4c8c56-w96vh\" (UID: \"79f77504-1067-40e5-9ea6-1f7b752d0f8d\") " pod="openshift-ingress/payload-processing-e2e-api-a-9fcb69-59fd4c8c56-w96vh"
Oct 30 13:14:29.079847 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:29.079755    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79f77504-1067-40e5-9ea6-1f7b752d0f8d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-9fcb69-59fd4c8c56-w96vh\" (UID: \"79f77504-1067-40e5-9ea6-1f7b752d0f8d\") " pod="openshift-ingress/payload-processing-e2e-api-a-9fcb69-59fd4c8c56-w96vh"
Oct 30 13:14:29.237305 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:29.237257    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79f77504-1067-40e5-9ea6-1f7b752d0f8d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-9fcb69-59fd4c8c56-w96vh\" (UID: \"79f77504-1067-40e5-9ea6-1f7b752d0f8d\") " pod="openshift-ingress/payload-processing-e2e-api-a-9fcb69-59fd4c8c56-w96vh"
Oct 30 13:14:29.237484 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:29.237370    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6zx7\" (UniqueName: \"kubernetes.io/projected/79f77504-1067-40e5-9ea6-1f7b752d0f8d-kube-api-access-f6zx7\") pod \"payload-processing-e2e-api-a-9fcb69-59fd4c8c56-w96vh\" (UID: \"79f77504-1067-40e5-9ea6-1f7b752d0f8d\") " pod="openshift-ingress/payload-processing-e2e-api-a-9fcb69-59fd4c8c56-w96vh"
Oct 30 13:14:29.238502 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:29.238473    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79f77504-1067-40e5-9ea6-1f7b752d0f8d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-9fcb69-59fd4c8c56-w96vh\" (UID: \"79f77504-1067-40e5-9ea6-1f7b752d0f8d\") " pod="openshift-ingress/payload-processing-e2e-api-a-9fcb69-59fd4c8c56-w96vh"
Oct 30 13:14:29.352012 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:29.351931    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6zx7\" (UniqueName: \"kubernetes.io/projected/79f77504-1067-40e5-9ea6-1f7b752d0f8d-kube-api-access-f6zx7\") pod \"payload-processing-e2e-api-a-9fcb69-59fd4c8c56-w96vh\" (UID: \"79f77504-1067-40e5-9ea6-1f7b752d0f8d\") " pod="openshift-ingress/payload-processing-e2e-api-a-9fcb69-59fd4c8c56-w96vh"
Oct 30 13:14:29.444865 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:29.444834    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9fcb69-59fd4c8c56-w96vh"
Oct 30 13:14:29.783007 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:14:29.782975    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79f77504_1067_40e5_9ea6_1f7b752d0f8d.slice/crio-03fc8df8ac6da358ba0fe7da32afe79ef5ba67a15b7e63a789afd1e9337a29ee WatchSource:0}: Error finding container 03fc8df8ac6da358ba0fe7da32afe79ef5ba67a15b7e63a789afd1e9337a29ee: Status 404 returned error can't find the container with id 03fc8df8ac6da358ba0fe7da32afe79ef5ba67a15b7e63a789afd1e9337a29ee
Oct 30 13:14:29.784839 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:29.784782    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9fcb69-59fd4c8c56-w96vh"]
Oct 30 13:14:30.290850 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:30.289428    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9fcb69-59fd4c8c56-w96vh" event={"ID":"79f77504-1067-40e5-9ea6-1f7b752d0f8d","Type":"ContainerStarted","Data":"7308e7164d8e5aba0eb93f581c5c1036dee1df1c8a3fe89418923ada8e068674"}
Oct 30 13:14:30.290850 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:30.289499    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9fcb69-59fd4c8c56-w96vh" event={"ID":"79f77504-1067-40e5-9ea6-1f7b752d0f8d","Type":"ContainerStarted","Data":"03fc8df8ac6da358ba0fe7da32afe79ef5ba67a15b7e63a789afd1e9337a29ee"}
Oct 30 13:14:30.290850 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:30.290716    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-9fcb69-59fd4c8c56-w96vh"
Oct 30 13:14:30.350210 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:30.350160    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-9fcb69-59fd4c8c56-w96vh" podStartSLOduration=2.350147313 podStartE2EDuration="2.350147313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:14:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:14:30.335576256 +0000 UTC m=+1369.501940920" watchObservedRunningTime="2026-10-30 13:14:30.350147313 +0000 UTC m=+1369.516511960"
Oct 30 13:14:31.307812 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:31.307774    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9c6033-6546465fff-z4rrd"]
Oct 30 13:14:31.319351 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:31.319320    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9c6033-6546465fff-z4rrd"
Oct 30 13:14:31.339653 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:31.339624    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9c6033-6546465fff-z4rrd"]
Oct 30 13:14:31.343935 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:31.343904    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-9c6033-dockercfg-v6n67\""
Oct 30 13:14:31.345772 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:31.345730    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-9c6033\""
Oct 30 13:14:31.373254 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:31.372824    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-9c6033-855647cb84-pkjp9"]
Oct 30 13:14:31.385202 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:31.384110    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-9c6033-855647cb84-pkjp9"
Oct 30 13:14:31.531053 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:31.530945    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-9c6033-855647cb84-pkjp9"]
Oct 30 13:14:31.601677 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:31.569577    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/362b87d6-a207-44a8-a247-92bc1e05e554-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-9c6033-6546465fff-z4rrd\" (UID: \"362b87d6-a207-44a8-a247-92bc1e05e554\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9c6033-6546465fff-z4rrd"
Oct 30 13:14:31.602063 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:31.602022    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t84gq\" (UniqueName: \"kubernetes.io/projected/362b87d6-a207-44a8-a247-92bc1e05e554-kube-api-access-t84gq\") pod \"payload-pre-processing-e2e-shared-a-w6-9c6033-6546465fff-z4rrd\" (UID: \"362b87d6-a207-44a8-a247-92bc1e05e554\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9c6033-6546465fff-z4rrd"
Oct 30 13:14:31.602291 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:31.602269    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5hwn\" (UniqueName: \"kubernetes.io/projected/3a5999a2-3c46-4554-8b5b-0e1ccf464ec8-kube-api-access-g5hwn\") pod \"payload-processing-e2e-shared-a-w6-9c6033-855647cb84-pkjp9\" (UID: \"3a5999a2-3c46-4554-8b5b-0e1ccf464ec8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-9c6033-855647cb84-pkjp9"
Oct 30 13:14:31.609835 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:31.609797    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a5999a2-3c46-4554-8b5b-0e1ccf464ec8-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-9c6033-855647cb84-pkjp9\" (UID: \"3a5999a2-3c46-4554-8b5b-0e1ccf464ec8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-9c6033-855647cb84-pkjp9"
Oct 30 13:14:31.724624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:31.715098    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/362b87d6-a207-44a8-a247-92bc1e05e554-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-9c6033-6546465fff-z4rrd\" (UID: \"362b87d6-a207-44a8-a247-92bc1e05e554\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9c6033-6546465fff-z4rrd"
Oct 30 13:14:31.724624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:31.715222    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t84gq\" (UniqueName: \"kubernetes.io/projected/362b87d6-a207-44a8-a247-92bc1e05e554-kube-api-access-t84gq\") pod \"payload-pre-processing-e2e-shared-a-w6-9c6033-6546465fff-z4rrd\" (UID: \"362b87d6-a207-44a8-a247-92bc1e05e554\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9c6033-6546465fff-z4rrd"
Oct 30 13:14:31.724624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:31.715260    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5hwn\" (UniqueName: \"kubernetes.io/projected/3a5999a2-3c46-4554-8b5b-0e1ccf464ec8-kube-api-access-g5hwn\") pod \"payload-processing-e2e-shared-a-w6-9c6033-855647cb84-pkjp9\" (UID: \"3a5999a2-3c46-4554-8b5b-0e1ccf464ec8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-9c6033-855647cb84-pkjp9"
Oct 30 13:14:31.724624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:31.715377    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a5999a2-3c46-4554-8b5b-0e1ccf464ec8-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-9c6033-855647cb84-pkjp9\" (UID: \"3a5999a2-3c46-4554-8b5b-0e1ccf464ec8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-9c6033-855647cb84-pkjp9"
Oct 30 13:14:31.724624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:31.716620    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a5999a2-3c46-4554-8b5b-0e1ccf464ec8-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-9c6033-855647cb84-pkjp9\" (UID: \"3a5999a2-3c46-4554-8b5b-0e1ccf464ec8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-9c6033-855647cb84-pkjp9"
Oct 30 13:14:31.724624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:31.717358    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/362b87d6-a207-44a8-a247-92bc1e05e554-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-9c6033-6546465fff-z4rrd\" (UID: \"362b87d6-a207-44a8-a247-92bc1e05e554\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9c6033-6546465fff-z4rrd"
Oct 30 13:14:31.960626 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:31.930376    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5hwn\" (UniqueName: \"kubernetes.io/projected/3a5999a2-3c46-4554-8b5b-0e1ccf464ec8-kube-api-access-g5hwn\") pod \"payload-processing-e2e-shared-a-w6-9c6033-855647cb84-pkjp9\" (UID: \"3a5999a2-3c46-4554-8b5b-0e1ccf464ec8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-9c6033-855647cb84-pkjp9"
Oct 30 13:14:31.960626 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:31.944411    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t84gq\" (UniqueName: \"kubernetes.io/projected/362b87d6-a207-44a8-a247-92bc1e05e554-kube-api-access-t84gq\") pod \"payload-pre-processing-e2e-shared-a-w6-9c6033-6546465fff-z4rrd\" (UID: \"362b87d6-a207-44a8-a247-92bc1e05e554\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9c6033-6546465fff-z4rrd"
Oct 30 13:14:31.985888 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:31.985804    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9c6033-6546465fff-z4rrd"
Oct 30 13:14:32.160579 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:32.160545    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-9c6033-855647cb84-pkjp9"
Oct 30 13:14:32.316840 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:32.316808    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9c6033-6546465fff-z4rrd"]
Oct 30 13:14:32.473910 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:14:32.473880    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a5999a2_3c46_4554_8b5b_0e1ccf464ec8.slice/crio-709e0bb87f162f72bb0f96e13d426cda3811ddd64f58d1c072c445efeb57d924 WatchSource:0}: Error finding container 709e0bb87f162f72bb0f96e13d426cda3811ddd64f58d1c072c445efeb57d924: Status 404 returned error can't find the container with id 709e0bb87f162f72bb0f96e13d426cda3811ddd64f58d1c072c445efeb57d924
Oct 30 13:14:32.475215 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:32.475191    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-9c6033-855647cb84-pkjp9"]
Oct 30 13:14:33.329181 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:33.329146    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-9c6033-855647cb84-pkjp9" event={"ID":"3a5999a2-3c46-4554-8b5b-0e1ccf464ec8","Type":"ContainerStarted","Data":"2bf3c9192711fe8aec4966e4f745f5294735a4991f5bd8529fdbf4cd36ab7c95"}
Oct 30 13:14:33.329688 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:33.329205    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-9c6033-855647cb84-pkjp9" event={"ID":"3a5999a2-3c46-4554-8b5b-0e1ccf464ec8","Type":"ContainerStarted","Data":"709e0bb87f162f72bb0f96e13d426cda3811ddd64f58d1c072c445efeb57d924"}
Oct 30 13:14:33.329688 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:33.329235    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-9c6033-855647cb84-pkjp9"
Oct 30 13:14:33.331439 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:33.331403    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9c6033-6546465fff-z4rrd" event={"ID":"362b87d6-a207-44a8-a247-92bc1e05e554","Type":"ContainerStarted","Data":"56faa51cac10db60007730500063d06a536488b96e31d9fd0ee541fc9a53519e"}
Oct 30 13:14:33.331532 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:33.331453    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9c6033-6546465fff-z4rrd" event={"ID":"362b87d6-a207-44a8-a247-92bc1e05e554","Type":"ContainerStarted","Data":"b7fd3694ef72c53853735edcae204044a20448ac78b52e6f2b168d59479a3c94"}
Oct 30 13:14:33.331532 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:33.331495    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9c6033-6546465fff-z4rrd"
Oct 30 13:14:33.385338 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:33.385278    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-9c6033-855647cb84-pkjp9" podStartSLOduration=2.385260335 podStartE2EDuration="2.385260335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:14:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:14:33.381250578 +0000 UTC m=+1372.547615226" watchObservedRunningTime="2026-10-30 13:14:33.385260335 +0000 UTC m=+1372.551624986"
Oct 30 13:14:33.435906 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:33.435845    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9c6033-6546465fff-z4rrd" podStartSLOduration=2.435827692 podStartE2EDuration="2.435827692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:14:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:14:33.429628682 +0000 UTC m=+1372.595993333" watchObservedRunningTime="2026-10-30 13:14:33.435827692 +0000 UTC m=+1372.602192341"
Oct 30 13:14:35.527397 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:35.527369    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df6c3661-5tfnb"]
Oct 30 13:14:35.531004 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:35.530987    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df6c3661-5tfnb"
Oct 30 13:14:35.552313 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:35.552279    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df6c3661-5tfnb"]
Oct 30 13:14:35.560409 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:35.560377    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:14:35.560409 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:35.560392    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\""
Oct 30 13:14:35.613735 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:35.613707    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e82ae840-d302-42ed-aa7c-daea8b5e1c23-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-df6c3661-5tfnb\" (UID: \"e82ae840-d302-42ed-aa7c-daea8b5e1c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df6c3661-5tfnb"
Oct 30 13:14:35.613843 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:35.613766    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e82ae840-d302-42ed-aa7c-daea8b5e1c23-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-df6c3661-5tfnb\" (UID: \"e82ae840-d302-42ed-aa7c-daea8b5e1c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df6c3661-5tfnb"
Oct 30 13:14:35.613902 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:35.613887    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlh5m\" (UniqueName: \"kubernetes.io/projected/e82ae840-d302-42ed-aa7c-daea8b5e1c23-kube-api-access-hlh5m\") pod \"maas-api-revoke-keys-e2e-mt-df6c3661-5tfnb\" (UID: \"e82ae840-d302-42ed-aa7c-daea8b5e1c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df6c3661-5tfnb"
Oct 30 13:14:35.715703 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:35.715651    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e82ae840-d302-42ed-aa7c-daea8b5e1c23-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-df6c3661-5tfnb\" (UID: \"e82ae840-d302-42ed-aa7c-daea8b5e1c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df6c3661-5tfnb"
Oct 30 13:14:35.715814 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:35.715711    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e82ae840-d302-42ed-aa7c-daea8b5e1c23-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-df6c3661-5tfnb\" (UID: \"e82ae840-d302-42ed-aa7c-daea8b5e1c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df6c3661-5tfnb"
Oct 30 13:14:35.715814 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:35.715790    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlh5m\" (UniqueName: \"kubernetes.io/projected/e82ae840-d302-42ed-aa7c-daea8b5e1c23-kube-api-access-hlh5m\") pod \"maas-api-revoke-keys-e2e-mt-df6c3661-5tfnb\" (UID: \"e82ae840-d302-42ed-aa7c-daea8b5e1c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df6c3661-5tfnb"
Oct 30 13:14:35.716414 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:35.716393    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e82ae840-d302-42ed-aa7c-daea8b5e1c23-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-df6c3661-5tfnb\" (UID: \"e82ae840-d302-42ed-aa7c-daea8b5e1c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df6c3661-5tfnb"
Oct 30 13:14:35.718063 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:35.718028    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e82ae840-d302-42ed-aa7c-daea8b5e1c23-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-df6c3661-5tfnb\" (UID: \"e82ae840-d302-42ed-aa7c-daea8b5e1c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df6c3661-5tfnb"
Oct 30 13:14:35.755739 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:35.755716    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlh5m\" (UniqueName: \"kubernetes.io/projected/e82ae840-d302-42ed-aa7c-daea8b5e1c23-kube-api-access-hlh5m\") pod \"maas-api-revoke-keys-e2e-mt-df6c3661-5tfnb\" (UID: \"e82ae840-d302-42ed-aa7c-daea8b5e1c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df6c3661-5tfnb"
Oct 30 13:14:35.844633 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:35.844611    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df6c3661-5tfnb"
Oct 30 13:14:36.174204 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:14:36.174176    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode82ae840_d302_42ed_aa7c_daea8b5e1c23.slice/crio-13a2a8e3d7662ac78543b8ba8f9f102b71ef3a56a20f55611ac042db13092248 WatchSource:0}: Error finding container 13a2a8e3d7662ac78543b8ba8f9f102b71ef3a56a20f55611ac042db13092248: Status 404 returned error can't find the container with id 13a2a8e3d7662ac78543b8ba8f9f102b71ef3a56a20f55611ac042db13092248
Oct 30 13:14:36.175700 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:36.175677    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df6c3661-5tfnb"]
Oct 30 13:14:36.361074 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:36.361044    4364 generic.go:356] "Generic (PLEG): container finished" podID="01c37c15-88bc-4359-9651-fdaae5a98c40" containerID="6e3b5d4a99c8fa36fefdd8cc9cb28310c2a270188417fa6fd0de5a998c86c16e" exitCode=0
Oct 30 13:14:36.361074 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:36.361061    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85" event={"ID":"01c37c15-88bc-4359-9651-fdaae5a98c40","Type":"ContainerDied","Data":"6e3b5d4a99c8fa36fefdd8cc9cb28310c2a270188417fa6fd0de5a998c86c16e"}
Oct 30 13:14:36.361467 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:36.361452    4364 scope.go:122] "RemoveContainer" containerID="6e3b5d4a99c8fa36fefdd8cc9cb28310c2a270188417fa6fd0de5a998c86c16e"
Oct 30 13:14:36.362789 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:36.362763    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df6c3661-5tfnb" event={"ID":"e82ae840-d302-42ed-aa7c-daea8b5e1c23","Type":"ContainerStarted","Data":"13a2a8e3d7662ac78543b8ba8f9f102b71ef3a56a20f55611ac042db13092248"}
Oct 30 13:14:36.363174 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:36.363146    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:14:36.363236 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:36.363222    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:14:36.363272 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:36.363263    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:14:37.433635 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:37.428746    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85" event={"ID":"01c37c15-88bc-4359-9651-fdaae5a98c40","Type":"ContainerStarted","Data":"abd8e875ac42ebe0d62371418107fac03e62f2eb648bd937a344e9931f745a24"}
Oct 30 13:14:37.472918 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:37.471804    4364 generic.go:356] "Generic (PLEG): container finished" podID="e82ae840-d302-42ed-aa7c-daea8b5e1c23" containerID="898750e5cc78fdbee11c9a70b19b7b9ab00516e1d8178dcf98fbb6b6fd36ad71" exitCode=0
Oct 30 13:14:37.472918 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:37.472057    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df6c3661-5tfnb" event={"ID":"e82ae840-d302-42ed-aa7c-daea8b5e1c23","Type":"ContainerDied","Data":"898750e5cc78fdbee11c9a70b19b7b9ab00516e1d8178dcf98fbb6b6fd36ad71"}
Oct 30 13:14:38.377090 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:38.377058    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85"
Oct 30 13:14:38.378370 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:38.378344    4364 patch_prober.go:29] interesting pod/e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 30 13:14:38.378504 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:38.378403    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85" podUID="01c37c15-88bc-4359-9651-fdaae5a98c40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 30 13:14:38.731242 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:38.731214    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df6c3661-5tfnb"
Oct 30 13:14:38.738632 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:38.738583    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85"
Oct 30 13:14:38.826175 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:38.826142    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e82ae840-d302-42ed-aa7c-daea8b5e1c23-kube-api-access-hlh5m\" (UniqueName: \"kubernetes.io/projected/e82ae840-d302-42ed-aa7c-daea8b5e1c23-kube-api-access-hlh5m\") pod \"e82ae840-d302-42ed-aa7c-daea8b5e1c23\" (UID: \"e82ae840-d302-42ed-aa7c-daea8b5e1c23\") "
Oct 30 13:14:38.826388 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:38.826229    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e82ae840-d302-42ed-aa7c-daea8b5e1c23-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e82ae840-d302-42ed-aa7c-daea8b5e1c23-maas-api-xks-ca\") pod \"e82ae840-d302-42ed-aa7c-daea8b5e1c23\" (UID: \"e82ae840-d302-42ed-aa7c-daea8b5e1c23\") "
Oct 30 13:14:38.826553 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:38.826537    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e82ae840-d302-42ed-aa7c-daea8b5e1c23-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e82ae840-d302-42ed-aa7c-daea8b5e1c23-maas-api-openshift-ca\") pod \"e82ae840-d302-42ed-aa7c-daea8b5e1c23\" (UID: \"e82ae840-d302-42ed-aa7c-daea8b5e1c23\") "
Oct 30 13:14:38.827539 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:38.827506    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e82ae840-d302-42ed-aa7c-daea8b5e1c23-maas-api-openshift-ca" pod "e82ae840-d302-42ed-aa7c-daea8b5e1c23" (UID: "e82ae840-d302-42ed-aa7c-daea8b5e1c23"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:14:38.833083 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:38.833054    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e82ae840-d302-42ed-aa7c-daea8b5e1c23-maas-api-xks-ca" pod "e82ae840-d302-42ed-aa7c-daea8b5e1c23" (UID: "e82ae840-d302-42ed-aa7c-daea8b5e1c23"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:14:38.833368 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:38.833346    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e82ae840-d302-42ed-aa7c-daea8b5e1c23-kube-api-access-hlh5m" pod "e82ae840-d302-42ed-aa7c-daea8b5e1c23" (UID: "e82ae840-d302-42ed-aa7c-daea8b5e1c23"). InnerVolumeSpecName "kube-api-access-hlh5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:14:38.929016 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:38.928935    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e82ae840-d302-42ed-aa7c-daea8b5e1c23-maas-api-openshift-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:14:38.929016 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:38.928964    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlh5m\" (UniqueName: \"kubernetes.io/projected/e82ae840-d302-42ed-aa7c-daea8b5e1c23-kube-api-access-hlh5m\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:14:38.929016 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:38.928977    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e82ae840-d302-42ed-aa7c-daea8b5e1c23-maas-api-xks-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:14:39.377948 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:39.377436    4364 patch_prober.go:29] interesting pod/e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 30 13:14:39.377948 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:39.377479    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85" podUID="01c37c15-88bc-4359-9651-fdaae5a98c40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 30 13:14:39.513197 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:39.513159    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8af0fe-5cf9cd699-gzxlc"]
Oct 30 13:14:39.514520 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:39.514496    4364 cpu_manager.go:414] "containerMap: removing container" podUID="e82ae840-d302-42ed-aa7c-daea8b5e1c23" containerName="revoke-keys"
Oct 30 13:14:39.514593 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:39.514524    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e82ae840-d302-42ed-aa7c-daea8b5e1c23" containerName="revoke-keys"
Oct 30 13:14:39.514842 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:39.514826    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="e82ae840-d302-42ed-aa7c-daea8b5e1c23" containerName="revoke-keys"
Oct 30 13:14:39.518516 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:39.518488    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8af0fe-5cf9cd699-gzxlc"
Oct 30 13:14:39.547623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:39.537651    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-8af0fe\""
Oct 30 13:14:39.547623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:39.538018    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-8af0fe-dockercfg-8fh49\""
Oct 30 13:14:39.665353 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:39.665274    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8af0fe-5cf9cd699-gzxlc"]
Oct 30 13:14:39.732704 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:39.710275    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fb63d98-c5f7-49fd-9ba0-b58f7329bd63-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-8af0fe-5cf9cd699-gzxlc\" (UID: \"3fb63d98-c5f7-49fd-9ba0-b58f7329bd63\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8af0fe-5cf9cd699-gzxlc"
Oct 30 13:14:39.764655 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:39.760989    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvcn5\" (UniqueName: \"kubernetes.io/projected/3fb63d98-c5f7-49fd-9ba0-b58f7329bd63-kube-api-access-wvcn5\") pod \"payload-pre-processing-e2e-api-a-8af0fe-5cf9cd699-gzxlc\" (UID: \"3fb63d98-c5f7-49fd-9ba0-b58f7329bd63\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8af0fe-5cf9cd699-gzxlc"
Oct 30 13:14:39.764655 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:39.734929    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df6c3661-5tfnb" event={"ID":"e82ae840-d302-42ed-aa7c-daea8b5e1c23","Type":"ContainerDied","Data":"13a2a8e3d7662ac78543b8ba8f9f102b71ef3a56a20f55611ac042db13092248"}
Oct 30 13:14:39.764655 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:39.761289    4364 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13a2a8e3d7662ac78543b8ba8f9f102b71ef3a56a20f55611ac042db13092248"
Oct 30 13:14:39.764655 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:39.734910    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df6c3661-5tfnb"
Oct 30 13:14:39.884745 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:39.864557    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fb63d98-c5f7-49fd-9ba0-b58f7329bd63-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-8af0fe-5cf9cd699-gzxlc\" (UID: \"3fb63d98-c5f7-49fd-9ba0-b58f7329bd63\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8af0fe-5cf9cd699-gzxlc"
Oct 30 13:14:39.884745 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:39.864766    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvcn5\" (UniqueName: \"kubernetes.io/projected/3fb63d98-c5f7-49fd-9ba0-b58f7329bd63-kube-api-access-wvcn5\") pod \"payload-pre-processing-e2e-api-a-8af0fe-5cf9cd699-gzxlc\" (UID: \"3fb63d98-c5f7-49fd-9ba0-b58f7329bd63\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8af0fe-5cf9cd699-gzxlc"
Oct 30 13:14:39.884745 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:39.882548    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fb63d98-c5f7-49fd-9ba0-b58f7329bd63-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-8af0fe-5cf9cd699-gzxlc\" (UID: \"3fb63d98-c5f7-49fd-9ba0-b58f7329bd63\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8af0fe-5cf9cd699-gzxlc"
Oct 30 13:14:40.052670 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:40.045469    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvcn5\" (UniqueName: \"kubernetes.io/projected/3fb63d98-c5f7-49fd-9ba0-b58f7329bd63-kube-api-access-wvcn5\") pod \"payload-pre-processing-e2e-api-a-8af0fe-5cf9cd699-gzxlc\" (UID: \"3fb63d98-c5f7-49fd-9ba0-b58f7329bd63\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8af0fe-5cf9cd699-gzxlc"
Oct 30 13:14:40.267124 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:40.267080    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8af0fe-5cf9cd699-gzxlc"
Oct 30 13:14:40.378320 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:40.378268    4364 patch_prober.go:29] interesting pod/e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 30 13:14:40.378459 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:40.378352    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85" podUID="01c37c15-88bc-4359-9651-fdaae5a98c40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 30 13:14:40.624556 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:14:40.624518    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fb63d98_c5f7_49fd_9ba0_b58f7329bd63.slice/crio-114ba6e8da54a06506b441a923381b0019cafa19f0991b452c549ab99aabfb49 WatchSource:0}: Error finding container 114ba6e8da54a06506b441a923381b0019cafa19f0991b452c549ab99aabfb49: Status 404 returned error can't find the container with id 114ba6e8da54a06506b441a923381b0019cafa19f0991b452c549ab99aabfb49
Oct 30 13:14:40.626436 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:40.626409    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8af0fe-5cf9cd699-gzxlc"]
Oct 30 13:14:40.745904 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:40.745873    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8af0fe-5cf9cd699-gzxlc" event={"ID":"3fb63d98-c5f7-49fd-9ba0-b58f7329bd63","Type":"ContainerStarted","Data":"114ba6e8da54a06506b441a923381b0019cafa19f0991b452c549ab99aabfb49"}
Oct 30 13:14:41.377421 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:41.377392    4364 patch_prober.go:29] interesting pod/e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 30 13:14:41.377629 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:41.377444    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85" podUID="01c37c15-88bc-4359-9651-fdaae5a98c40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 30 13:14:41.530226 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:14:41.530194    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:14:41.570783 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:41.570755    4364 scope.go:122] "RemoveContainer" containerID="3b6043c8778d6b00479bee773a80ba445a4d1f68b4a5460eff7e9b0f03f3d40c"
Oct 30 13:14:41.603943 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:41.603772    4364 scope.go:122] "RemoveContainer" containerID="f635944dcfd581313882ad0be4f66259c0ebfed92bbf7bd52e44161d469f6803"
Oct 30 13:14:41.615014 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:41.614993    4364 scope.go:122] "RemoveContainer" containerID="6a242d72328973a402309222e06bf84a310279229fc7810d7090acb47545fafb"
Oct 30 13:14:41.755926 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:41.755894    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8af0fe-5cf9cd699-gzxlc" event={"ID":"3fb63d98-c5f7-49fd-9ba0-b58f7329bd63","Type":"ContainerStarted","Data":"3d3ef208745d7571d97084e026c4249d2c86001cc59fbb52329887a707d59ab0"}
Oct 30 13:14:41.756268 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:41.755970    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8af0fe-5cf9cd699-gzxlc"
Oct 30 13:14:41.808607 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:41.808395    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8af0fe-5cf9cd699-gzxlc" podStartSLOduration=2.808378476 podStartE2EDuration="2.808378476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:14:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:14:41.804913192 +0000 UTC m=+1380.971277839" watchObservedRunningTime="2026-10-30 13:14:41.808378476 +0000 UTC m=+1380.974743124"
Oct 30 13:14:42.319362 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:42.319334    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-9fcb69-59fd4c8c56-w96vh"
Oct 30 13:14:42.383624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:42.377247    4364 patch_prober.go:29] interesting pod/e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 30 13:14:42.383624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:42.377315    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85" podUID="01c37c15-88bc-4359-9651-fdaae5a98c40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 30 13:14:43.395417 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:43.394396    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85"
Oct 30 13:14:43.412043 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:43.412015    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85"
Oct 30 13:14:44.341342 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:44.341316    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-9c6033-855647cb84-pkjp9"
Oct 30 13:14:44.341515 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:44.341377    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9c6033-6546465fff-z4rrd"
Oct 30 13:14:45.970561 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:45.970523    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-df6c3661-7f764d9794-zph7s"]
Oct 30 13:14:46.023359 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:46.023329    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-df6c3661-5866c5b5c5-99krg"]
Oct 30 13:14:46.917403 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:46.916889    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3778dacd-fqdm6"]
Oct 30 13:14:47.018622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:47.006255    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3778dacd-fqdm6"
Oct 30 13:14:47.031196 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:47.031067    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:14:47.031351 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:47.031310    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\""
Oct 30 13:14:47.044648 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:47.044343    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3778dacd-fqdm6"]
Oct 30 13:14:47.111114 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:47.111086    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd032232-133c-4c37-b99b-88b14b88cebb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-3778dacd-fqdm6\" (UID: \"fd032232-133c-4c37-b99b-88b14b88cebb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3778dacd-fqdm6"
Oct 30 13:14:47.111244 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:47.111159    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd032232-133c-4c37-b99b-88b14b88cebb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-3778dacd-fqdm6\" (UID: \"fd032232-133c-4c37-b99b-88b14b88cebb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3778dacd-fqdm6"
Oct 30 13:14:47.111298 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:47.111259    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtt5b\" (UniqueName: \"kubernetes.io/projected/fd032232-133c-4c37-b99b-88b14b88cebb-kube-api-access-xtt5b\") pod \"maas-api-revoke-keys-e2e-mt-3778dacd-fqdm6\" (UID: \"fd032232-133c-4c37-b99b-88b14b88cebb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3778dacd-fqdm6"
Oct 30 13:14:47.212493 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:47.212434    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd032232-133c-4c37-b99b-88b14b88cebb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-3778dacd-fqdm6\" (UID: \"fd032232-133c-4c37-b99b-88b14b88cebb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3778dacd-fqdm6"
Oct 30 13:14:47.212616 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:47.212492    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtt5b\" (UniqueName: \"kubernetes.io/projected/fd032232-133c-4c37-b99b-88b14b88cebb-kube-api-access-xtt5b\") pod \"maas-api-revoke-keys-e2e-mt-3778dacd-fqdm6\" (UID: \"fd032232-133c-4c37-b99b-88b14b88cebb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3778dacd-fqdm6"
Oct 30 13:14:47.212734 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:47.212722    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd032232-133c-4c37-b99b-88b14b88cebb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-3778dacd-fqdm6\" (UID: \"fd032232-133c-4c37-b99b-88b14b88cebb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3778dacd-fqdm6"
Oct 30 13:14:47.213265 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:47.213245    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd032232-133c-4c37-b99b-88b14b88cebb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-3778dacd-fqdm6\" (UID: \"fd032232-133c-4c37-b99b-88b14b88cebb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3778dacd-fqdm6"
Oct 30 13:14:47.214854 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:47.214834    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd032232-133c-4c37-b99b-88b14b88cebb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-3778dacd-fqdm6\" (UID: \"fd032232-133c-4c37-b99b-88b14b88cebb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3778dacd-fqdm6"
Oct 30 13:14:47.249360 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:47.249334    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtt5b\" (UniqueName: \"kubernetes.io/projected/fd032232-133c-4c37-b99b-88b14b88cebb-kube-api-access-xtt5b\") pod \"maas-api-revoke-keys-e2e-mt-3778dacd-fqdm6\" (UID: \"fd032232-133c-4c37-b99b-88b14b88cebb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3778dacd-fqdm6"
Oct 30 13:14:47.344721 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:47.344701    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3778dacd-fqdm6"
Oct 30 13:14:47.653947 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:14:47.653913    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd032232_133c_4c37_b99b_88b14b88cebb.slice/crio-22d8b7960b4a4e1fa9410bf3c1d0b96df183151b2e36e307bd07545bc24e20ee WatchSource:0}: Error finding container 22d8b7960b4a4e1fa9410bf3c1d0b96df183151b2e36e307bd07545bc24e20ee: Status 404 returned error can't find the container with id 22d8b7960b4a4e1fa9410bf3c1d0b96df183151b2e36e307bd07545bc24e20ee
Oct 30 13:14:47.654987 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:47.654966    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3778dacd-fqdm6"]
Oct 30 13:14:47.935668 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:47.935545    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3778dacd-fqdm6" event={"ID":"fd032232-133c-4c37-b99b-88b14b88cebb","Type":"ContainerStarted","Data":"22d8b7960b4a4e1fa9410bf3c1d0b96df183151b2e36e307bd07545bc24e20ee"}
Oct 30 13:14:48.945941 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:48.945905    4364 generic.go:356] "Generic (PLEG): container finished" podID="fd032232-133c-4c37-b99b-88b14b88cebb" containerID="d18fc03b84b7028bf478c34e31c7360691f7f8f28f6266bd0a9c30c69777bb0e" exitCode=0
Oct 30 13:14:48.946379 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:48.945985    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3778dacd-fqdm6" event={"ID":"fd032232-133c-4c37-b99b-88b14b88cebb","Type":"ContainerDied","Data":"d18fc03b84b7028bf478c34e31c7360691f7f8f28f6266bd0a9c30c69777bb0e"}
Oct 30 13:14:50.192392 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:50.192368    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3778dacd-fqdm6"
Oct 30 13:14:50.262182 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:50.262106    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd032232-133c-4c37-b99b-88b14b88cebb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd032232-133c-4c37-b99b-88b14b88cebb-maas-api-openshift-ca\") pod \"fd032232-133c-4c37-b99b-88b14b88cebb\" (UID: \"fd032232-133c-4c37-b99b-88b14b88cebb\") "
Oct 30 13:14:50.262182 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:50.262159    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd032232-133c-4c37-b99b-88b14b88cebb-kube-api-access-xtt5b\" (UniqueName: \"kubernetes.io/projected/fd032232-133c-4c37-b99b-88b14b88cebb-kube-api-access-xtt5b\") pod \"fd032232-133c-4c37-b99b-88b14b88cebb\" (UID: \"fd032232-133c-4c37-b99b-88b14b88cebb\") "
Oct 30 13:14:50.262402 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:50.262327    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd032232-133c-4c37-b99b-88b14b88cebb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd032232-133c-4c37-b99b-88b14b88cebb-maas-api-xks-ca\") pod \"fd032232-133c-4c37-b99b-88b14b88cebb\" (UID: \"fd032232-133c-4c37-b99b-88b14b88cebb\") "
Oct 30 13:14:50.262511 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:50.262489    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd032232-133c-4c37-b99b-88b14b88cebb-maas-api-openshift-ca" pod "fd032232-133c-4c37-b99b-88b14b88cebb" (UID: "fd032232-133c-4c37-b99b-88b14b88cebb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:14:50.263248 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:50.263212    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd032232-133c-4c37-b99b-88b14b88cebb-maas-api-openshift-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:14:50.264545 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:50.264514    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd032232-133c-4c37-b99b-88b14b88cebb-kube-api-access-xtt5b" pod "fd032232-133c-4c37-b99b-88b14b88cebb" (UID: "fd032232-133c-4c37-b99b-88b14b88cebb"). InnerVolumeSpecName "kube-api-access-xtt5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:14:50.264851 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:50.264830    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd032232-133c-4c37-b99b-88b14b88cebb-maas-api-xks-ca" pod "fd032232-133c-4c37-b99b-88b14b88cebb" (UID: "fd032232-133c-4c37-b99b-88b14b88cebb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:14:50.365060 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:50.365035    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtt5b\" (UniqueName: \"kubernetes.io/projected/fd032232-133c-4c37-b99b-88b14b88cebb-kube-api-access-xtt5b\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:14:50.365060 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:50.365061    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd032232-133c-4c37-b99b-88b14b88cebb-maas-api-xks-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:14:50.979342 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:50.979294    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-df6c3661-7f764d9794-zph7s" podUID="e50284eb-016e-4bc8-bf70-b3eb2815493c" containerName="payload-processing" containerID="cri-o://26c3a8eac05adbbf7801fbcca634ec6809557ac4043655bd20db49bf72a42b2f" gracePeriod=25
Oct 30 13:14:50.987038 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:50.987012    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3778dacd-fqdm6"
Oct 30 13:14:50.987854 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:50.987827    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3778dacd-fqdm6" event={"ID":"fd032232-133c-4c37-b99b-88b14b88cebb","Type":"ContainerDied","Data":"22d8b7960b4a4e1fa9410bf3c1d0b96df183151b2e36e307bd07545bc24e20ee"}
Oct 30 13:14:50.987935 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:50.987877    4364 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22d8b7960b4a4e1fa9410bf3c1d0b96df183151b2e36e307bd07545bc24e20ee"
Oct 30 13:14:51.024229 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:51.024186    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-df6c3661-5866c5b5c5-99krg" podUID="552839a6-adf1-4cb3-adf3-c75500d71782" containerName="payload-pre-processing" containerID="cri-o://16fa7ae7eec2b13d0d6406c1e7bb31b05664dfc6c45ca82fafc3e99fe6a4bc7d" gracePeriod=25
Oct 30 13:14:52.768285 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:52.768255    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8af0fe-5cf9cd699-gzxlc"
Oct 30 13:14:54.349463 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:54.349388    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425"]
Oct 30 13:14:54.349916 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:54.349575    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425" podUID="9059a7c7-81e0-4f77-b2dd-2e44f5c10d06" containerName="istio-proxy" containerID="cri-o://aa8476f1323579494f81445c97c4f7b9bf1bb9c2704b99d7d691701a3ced19db" gracePeriod=30
Oct 30 13:14:54.361153 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:54.361126    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww"]
Oct 30 13:14:54.361343 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:54.361320    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww" podUID="3f1549da-ef30-4001-9eec-35b1040de3b6" containerName="istio-proxy" containerID="cri-o://8245749a780d8d79e3a62f3120c8bfb5456abe2999f435a12322852a787eae13" gracePeriod=30
Oct 30 13:14:54.367009 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:54.366981    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q"]
Oct 30 13:14:54.367241 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:54.367214    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q" podUID="db45cea6-0b96-46b2-94ff-48caa8747410" containerName="istio-proxy" containerID="cri-o://82d874cf78bd4de431a430ad9cc436dde8678c678fabea99abe352c627a0e7b2" gracePeriod=30
Oct 30 13:14:57.249525 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:57.249468    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3778dacd-b89667ff4-jm528"]
Oct 30 13:14:59.188815 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.188773    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-df6c3661-5866c5b5c5-99krg" podUID="552839a6-adf1-4cb3-adf3-c75500d71782" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:14:59.682052 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.682023    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425"
Oct 30 13:14:59.760329 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.760299    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww"
Oct 30 13:14:59.771670 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.768088    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n"]
Oct 30 13:14:59.771670 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.769590    4364 cpu_manager.go:414] "containerMap: removing container" podUID="fd032232-133c-4c37-b99b-88b14b88cebb" containerName="revoke-keys"
Oct 30 13:14:59.771670 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.769628    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd032232-133c-4c37-b99b-88b14b88cebb" containerName="revoke-keys"
Oct 30 13:14:59.771670 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.769662    4364 cpu_manager.go:414] "containerMap: removing container" podUID="3f1549da-ef30-4001-9eec-35b1040de3b6" containerName="istio-proxy"
Oct 30 13:14:59.771670 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.769671    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f1549da-ef30-4001-9eec-35b1040de3b6" containerName="istio-proxy"
Oct 30 13:14:59.771670 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.769729    4364 cpu_manager.go:414] "containerMap: removing container" podUID="9059a7c7-81e0-4f77-b2dd-2e44f5c10d06" containerName="istio-proxy"
Oct 30 13:14:59.771670 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.769737    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9059a7c7-81e0-4f77-b2dd-2e44f5c10d06" containerName="istio-proxy"
Oct 30 13:14:59.771670 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.769921    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd032232-133c-4c37-b99b-88b14b88cebb" containerName="revoke-keys"
Oct 30 13:14:59.771670 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.769939    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f1549da-ef30-4001-9eec-35b1040de3b6" containerName="istio-proxy"
Oct 30 13:14:59.771670 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.769956    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="9059a7c7-81e0-4f77-b2dd-2e44f5c10d06" containerName="istio-proxy"
Oct 30 13:14:59.773138 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.773106    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9059a7c7-81e0-4f77-b2dd-2e44f5c10d06-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9059a7c7-81e0-4f77-b2dd-2e44f5c10d06-workload-socket\") pod \"9059a7c7-81e0-4f77-b2dd-2e44f5c10d06\" (UID: \"9059a7c7-81e0-4f77-b2dd-2e44f5c10d06\") "
Oct 30 13:14:59.773263 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.773179    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9059a7c7-81e0-4f77-b2dd-2e44f5c10d06-kube-api-access-gjj9z\" (UniqueName: \"kubernetes.io/projected/9059a7c7-81e0-4f77-b2dd-2e44f5c10d06-kube-api-access-gjj9z\") pod \"9059a7c7-81e0-4f77-b2dd-2e44f5c10d06\" (UID: \"9059a7c7-81e0-4f77-b2dd-2e44f5c10d06\") "
Oct 30 13:14:59.773323 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.773272    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9059a7c7-81e0-4f77-b2dd-2e44f5c10d06-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9059a7c7-81e0-4f77-b2dd-2e44f5c10d06-istiod-ca-cert\") pod \"9059a7c7-81e0-4f77-b2dd-2e44f5c10d06\" (UID: \"9059a7c7-81e0-4f77-b2dd-2e44f5c10d06\") "
Oct 30 13:14:59.773480 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.773375    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9059a7c7-81e0-4f77-b2dd-2e44f5c10d06-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9059a7c7-81e0-4f77-b2dd-2e44f5c10d06-istio-envoy\") pod \"9059a7c7-81e0-4f77-b2dd-2e44f5c10d06\" (UID: \"9059a7c7-81e0-4f77-b2dd-2e44f5c10d06\") "
Oct 30 13:14:59.773480 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.773385    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9059a7c7-81e0-4f77-b2dd-2e44f5c10d06-workload-socket" pod "9059a7c7-81e0-4f77-b2dd-2e44f5c10d06" (UID: "9059a7c7-81e0-4f77-b2dd-2e44f5c10d06"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:14:59.773480 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.773408    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9059a7c7-81e0-4f77-b2dd-2e44f5c10d06-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9059a7c7-81e0-4f77-b2dd-2e44f5c10d06-credential-socket\") pod \"9059a7c7-81e0-4f77-b2dd-2e44f5c10d06\" (UID: \"9059a7c7-81e0-4f77-b2dd-2e44f5c10d06\") "
Oct 30 13:14:59.773480 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.773439    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9059a7c7-81e0-4f77-b2dd-2e44f5c10d06-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9059a7c7-81e0-4f77-b2dd-2e44f5c10d06-istio-data\") pod \"9059a7c7-81e0-4f77-b2dd-2e44f5c10d06\" (UID: \"9059a7c7-81e0-4f77-b2dd-2e44f5c10d06\") "
Oct 30 13:14:59.773696 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.773555    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9059a7c7-81e0-4f77-b2dd-2e44f5c10d06-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9059a7c7-81e0-4f77-b2dd-2e44f5c10d06-workload-certs\") pod \"9059a7c7-81e0-4f77-b2dd-2e44f5c10d06\" (UID: \"9059a7c7-81e0-4f77-b2dd-2e44f5c10d06\") "
Oct 30 13:14:59.773696 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.773675    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9059a7c7-81e0-4f77-b2dd-2e44f5c10d06-istio-token\" (UniqueName: \"kubernetes.io/projected/9059a7c7-81e0-4f77-b2dd-2e44f5c10d06-istio-token\") pod \"9059a7c7-81e0-4f77-b2dd-2e44f5c10d06\" (UID: \"9059a7c7-81e0-4f77-b2dd-2e44f5c10d06\") "
Oct 30 13:14:59.773794 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.773713    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9059a7c7-81e0-4f77-b2dd-2e44f5c10d06-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9059a7c7-81e0-4f77-b2dd-2e44f5c10d06-istio-podinfo\") pod \"9059a7c7-81e0-4f77-b2dd-2e44f5c10d06\" (UID: \"9059a7c7-81e0-4f77-b2dd-2e44f5c10d06\") "
Oct 30 13:14:59.773794 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.773723    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9059a7c7-81e0-4f77-b2dd-2e44f5c10d06-istiod-ca-cert" pod "9059a7c7-81e0-4f77-b2dd-2e44f5c10d06" (UID: "9059a7c7-81e0-4f77-b2dd-2e44f5c10d06"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:14:59.775940 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.774935    4364 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9059a7c7-81e0-4f77-b2dd-2e44f5c10d06-workload-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:14:59.775940 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.774963    4364 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9059a7c7-81e0-4f77-b2dd-2e44f5c10d06-istiod-ca-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:14:59.775940 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.774936    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9059a7c7-81e0-4f77-b2dd-2e44f5c10d06-istio-data" pod "9059a7c7-81e0-4f77-b2dd-2e44f5c10d06" (UID: "9059a7c7-81e0-4f77-b2dd-2e44f5c10d06"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:14:59.775940 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.774961    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9059a7c7-81e0-4f77-b2dd-2e44f5c10d06-credential-socket" pod "9059a7c7-81e0-4f77-b2dd-2e44f5c10d06" (UID: "9059a7c7-81e0-4f77-b2dd-2e44f5c10d06"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:14:59.775940 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.775462    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9059a7c7-81e0-4f77-b2dd-2e44f5c10d06-workload-certs" pod "9059a7c7-81e0-4f77-b2dd-2e44f5c10d06" (UID: "9059a7c7-81e0-4f77-b2dd-2e44f5c10d06"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:14:59.787947 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.782355    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n"
Oct 30 13:14:59.787947 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.783834    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9059a7c7-81e0-4f77-b2dd-2e44f5c10d06-istio-token" pod "9059a7c7-81e0-4f77-b2dd-2e44f5c10d06" (UID: "9059a7c7-81e0-4f77-b2dd-2e44f5c10d06"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:14:59.787947 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.787383    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9059a7c7-81e0-4f77-b2dd-2e44f5c10d06-istio-podinfo" pod "9059a7c7-81e0-4f77-b2dd-2e44f5c10d06" (UID: "9059a7c7-81e0-4f77-b2dd-2e44f5c10d06"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:14:59.787947 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.787535    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q"
Oct 30 13:14:59.790038 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.790011    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9059a7c7-81e0-4f77-b2dd-2e44f5c10d06-kube-api-access-gjj9z" pod "9059a7c7-81e0-4f77-b2dd-2e44f5c10d06" (UID: "9059a7c7-81e0-4f77-b2dd-2e44f5c10d06"). InnerVolumeSpecName "kube-api-access-gjj9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:14:59.791966 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.791935    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9059a7c7-81e0-4f77-b2dd-2e44f5c10d06-istio-envoy" pod "9059a7c7-81e0-4f77-b2dd-2e44f5c10d06" (UID: "9059a7c7-81e0-4f77-b2dd-2e44f5c10d06"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:14:59.792871 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.792849    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n"]
Oct 30 13:14:59.798209 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.798181    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-df6c3661-openshift-default-dockercfg-hfkrp\""
Oct 30 13:14:59.828673 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.828643    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm"]
Oct 30 13:14:59.830027 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.830000    4364 cpu_manager.go:414] "containerMap: removing container" podUID="db45cea6-0b96-46b2-94ff-48caa8747410" containerName="istio-proxy"
Oct 30 13:14:59.830481 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.830461    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db45cea6-0b96-46b2-94ff-48caa8747410" containerName="istio-proxy"
Oct 30 13:14:59.830936 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.830915    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="db45cea6-0b96-46b2-94ff-48caa8747410" containerName="istio-proxy"
Oct 30 13:14:59.834392 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.834371    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm"
Oct 30 13:14:59.877015 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.876982    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f1549da-ef30-4001-9eec-35b1040de3b6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f1549da-ef30-4001-9eec-35b1040de3b6-istiod-ca-cert\") pod \"3f1549da-ef30-4001-9eec-35b1040de3b6\" (UID: \"3f1549da-ef30-4001-9eec-35b1040de3b6\") "
Oct 30 13:14:59.877181 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.877062    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db45cea6-0b96-46b2-94ff-48caa8747410-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db45cea6-0b96-46b2-94ff-48caa8747410-credential-socket\") pod \"db45cea6-0b96-46b2-94ff-48caa8747410\" (UID: \"db45cea6-0b96-46b2-94ff-48caa8747410\") "
Oct 30 13:14:59.877547 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.877520    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f1549da-ef30-4001-9eec-35b1040de3b6-istiod-ca-cert" pod "3f1549da-ef30-4001-9eec-35b1040de3b6" (UID: "3f1549da-ef30-4001-9eec-35b1040de3b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:14:59.877918 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.877893    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db45cea6-0b96-46b2-94ff-48caa8747410-credential-socket" pod "db45cea6-0b96-46b2-94ff-48caa8747410" (UID: "db45cea6-0b96-46b2-94ff-48caa8747410"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:14:59.877985 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.877936    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db45cea6-0b96-46b2-94ff-48caa8747410-kube-api-access-lfshc\" (UniqueName: \"kubernetes.io/projected/db45cea6-0b96-46b2-94ff-48caa8747410-kube-api-access-lfshc\") pod \"db45cea6-0b96-46b2-94ff-48caa8747410\" (UID: \"db45cea6-0b96-46b2-94ff-48caa8747410\") "
Oct 30 13:14:59.878054 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.878038    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f1549da-ef30-4001-9eec-35b1040de3b6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f1549da-ef30-4001-9eec-35b1040de3b6-istio-data\") pod \"3f1549da-ef30-4001-9eec-35b1040de3b6\" (UID: \"3f1549da-ef30-4001-9eec-35b1040de3b6\") "
Oct 30 13:14:59.878108 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.878084    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f1549da-ef30-4001-9eec-35b1040de3b6-istio-token\" (UniqueName: \"kubernetes.io/projected/3f1549da-ef30-4001-9eec-35b1040de3b6-istio-token\") pod \"3f1549da-ef30-4001-9eec-35b1040de3b6\" (UID: \"3f1549da-ef30-4001-9eec-35b1040de3b6\") "
Oct 30 13:14:59.878164 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.878125    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db45cea6-0b96-46b2-94ff-48caa8747410-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db45cea6-0b96-46b2-94ff-48caa8747410-workload-certs\") pod \"db45cea6-0b96-46b2-94ff-48caa8747410\" (UID: \"db45cea6-0b96-46b2-94ff-48caa8747410\") "
Oct 30 13:14:59.878219 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.878188    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f1549da-ef30-4001-9eec-35b1040de3b6-kube-api-access-t5kdk\" (UniqueName: \"kubernetes.io/projected/3f1549da-ef30-4001-9eec-35b1040de3b6-kube-api-access-t5kdk\") pod \"3f1549da-ef30-4001-9eec-35b1040de3b6\" (UID: \"3f1549da-ef30-4001-9eec-35b1040de3b6\") "
Oct 30 13:14:59.878277 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.878225    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db45cea6-0b96-46b2-94ff-48caa8747410-istio-token\" (UniqueName: \"kubernetes.io/projected/db45cea6-0b96-46b2-94ff-48caa8747410-istio-token\") pod \"db45cea6-0b96-46b2-94ff-48caa8747410\" (UID: \"db45cea6-0b96-46b2-94ff-48caa8747410\") "
Oct 30 13:14:59.878523 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.878502    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db45cea6-0b96-46b2-94ff-48caa8747410-workload-certs" pod "db45cea6-0b96-46b2-94ff-48caa8747410" (UID: "db45cea6-0b96-46b2-94ff-48caa8747410"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:14:59.878695 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.878565    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f1549da-ef30-4001-9eec-35b1040de3b6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f1549da-ef30-4001-9eec-35b1040de3b6-istio-podinfo\") pod \"3f1549da-ef30-4001-9eec-35b1040de3b6\" (UID: \"3f1549da-ef30-4001-9eec-35b1040de3b6\") "
Oct 30 13:14:59.878695 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.878662    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db45cea6-0b96-46b2-94ff-48caa8747410-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db45cea6-0b96-46b2-94ff-48caa8747410-istiod-ca-cert\") pod \"db45cea6-0b96-46b2-94ff-48caa8747410\" (UID: \"db45cea6-0b96-46b2-94ff-48caa8747410\") "
Oct 30 13:14:59.878802 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.878733    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f1549da-ef30-4001-9eec-35b1040de3b6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f1549da-ef30-4001-9eec-35b1040de3b6-istio-envoy\") pod \"3f1549da-ef30-4001-9eec-35b1040de3b6\" (UID: \"3f1549da-ef30-4001-9eec-35b1040de3b6\") "
Oct 30 13:14:59.878857 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.878819    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db45cea6-0b96-46b2-94ff-48caa8747410-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db45cea6-0b96-46b2-94ff-48caa8747410-istio-podinfo\") pod \"db45cea6-0b96-46b2-94ff-48caa8747410\" (UID: \"db45cea6-0b96-46b2-94ff-48caa8747410\") "
Oct 30 13:14:59.878902 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.878856    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db45cea6-0b96-46b2-94ff-48caa8747410-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db45cea6-0b96-46b2-94ff-48caa8747410-workload-socket\") pod \"db45cea6-0b96-46b2-94ff-48caa8747410\" (UID: \"db45cea6-0b96-46b2-94ff-48caa8747410\") "
Oct 30 13:14:59.878954 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.878926    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f1549da-ef30-4001-9eec-35b1040de3b6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f1549da-ef30-4001-9eec-35b1040de3b6-workload-socket\") pod \"3f1549da-ef30-4001-9eec-35b1040de3b6\" (UID: \"3f1549da-ef30-4001-9eec-35b1040de3b6\") "
Oct 30 13:14:59.879004 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.878954    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db45cea6-0b96-46b2-94ff-48caa8747410-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db45cea6-0b96-46b2-94ff-48caa8747410-istio-data\") pod \"db45cea6-0b96-46b2-94ff-48caa8747410\" (UID: \"db45cea6-0b96-46b2-94ff-48caa8747410\") "
Oct 30 13:14:59.879053 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.878996    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f1549da-ef30-4001-9eec-35b1040de3b6-istio-data" pod "3f1549da-ef30-4001-9eec-35b1040de3b6" (UID: "3f1549da-ef30-4001-9eec-35b1040de3b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:14:59.879053 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.879010    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db45cea6-0b96-46b2-94ff-48caa8747410-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db45cea6-0b96-46b2-94ff-48caa8747410-istio-envoy\") pod \"db45cea6-0b96-46b2-94ff-48caa8747410\" (UID: \"db45cea6-0b96-46b2-94ff-48caa8747410\") "
Oct 30 13:14:59.879158 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.879139    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f1549da-ef30-4001-9eec-35b1040de3b6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f1549da-ef30-4001-9eec-35b1040de3b6-workload-certs\") pod \"3f1549da-ef30-4001-9eec-35b1040de3b6\" (UID: \"3f1549da-ef30-4001-9eec-35b1040de3b6\") "
Oct 30 13:14:59.879926 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.879679    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f1549da-ef30-4001-9eec-35b1040de3b6-workload-socket" pod "3f1549da-ef30-4001-9eec-35b1040de3b6" (UID: "3f1549da-ef30-4001-9eec-35b1040de3b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:14:59.879926 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.879879    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db45cea6-0b96-46b2-94ff-48caa8747410-workload-socket" pod "db45cea6-0b96-46b2-94ff-48caa8747410" (UID: "db45cea6-0b96-46b2-94ff-48caa8747410"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:14:59.880089 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.880044    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db45cea6-0b96-46b2-94ff-48caa8747410-istio-data" pod "db45cea6-0b96-46b2-94ff-48caa8747410" (UID: "db45cea6-0b96-46b2-94ff-48caa8747410"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:14:59.880376 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.880353    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f1549da-ef30-4001-9eec-35b1040de3b6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f1549da-ef30-4001-9eec-35b1040de3b6-credential-socket\") pod \"3f1549da-ef30-4001-9eec-35b1040de3b6\" (UID: \"3f1549da-ef30-4001-9eec-35b1040de3b6\") "
Oct 30 13:14:59.880702 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.880675    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93fa52c8-91ad-40ef-a3b0-46c740175647-istio-podinfo\") pod \"e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n\" (UID: \"93fa52c8-91ad-40ef-a3b0-46c740175647\") " pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n"
Oct 30 13:14:59.880891 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.880871    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93fa52c8-91ad-40ef-a3b0-46c740175647-istio-token\") pod \"e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n\" (UID: \"93fa52c8-91ad-40ef-a3b0-46c740175647\") " pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n"
Oct 30 13:14:59.880966 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.880905    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93fa52c8-91ad-40ef-a3b0-46c740175647-istiod-ca-cert\") pod \"e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n\" (UID: \"93fa52c8-91ad-40ef-a3b0-46c740175647\") " pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n"
Oct 30 13:14:59.880966 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.880945    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93fa52c8-91ad-40ef-a3b0-46c740175647-workload-socket\") pod \"e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n\" (UID: \"93fa52c8-91ad-40ef-a3b0-46c740175647\") " pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n"
Oct 30 13:14:59.881061 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.880987    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93fa52c8-91ad-40ef-a3b0-46c740175647-istio-envoy\") pod \"e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n\" (UID: \"93fa52c8-91ad-40ef-a3b0-46c740175647\") " pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n"
Oct 30 13:14:59.881116 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.881093    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qskw\" (UniqueName: \"kubernetes.io/projected/93fa52c8-91ad-40ef-a3b0-46c740175647-kube-api-access-8qskw\") pod \"e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n\" (UID: \"93fa52c8-91ad-40ef-a3b0-46c740175647\") " pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n"
Oct 30 13:14:59.881209 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.881193    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93fa52c8-91ad-40ef-a3b0-46c740175647-credential-socket\") pod \"e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n\" (UID: \"93fa52c8-91ad-40ef-a3b0-46c740175647\") " pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n"
Oct 30 13:14:59.881289 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.881273    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93fa52c8-91ad-40ef-a3b0-46c740175647-istio-data\") pod \"e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n\" (UID: \"93fa52c8-91ad-40ef-a3b0-46c740175647\") " pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n"
Oct 30 13:14:59.881350 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.881334    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93fa52c8-91ad-40ef-a3b0-46c740175647-workload-certs\") pod \"e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n\" (UID: \"93fa52c8-91ad-40ef-a3b0-46c740175647\") " pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n"
Oct 30 13:14:59.881682 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.881662    4364 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db45cea6-0b96-46b2-94ff-48caa8747410-workload-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:14:59.881773 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.881689    4364 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f1549da-ef30-4001-9eec-35b1040de3b6-workload-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:14:59.881773 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.881702    4364 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db45cea6-0b96-46b2-94ff-48caa8747410-istio-data\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:14:59.881773 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.881714    4364 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9059a7c7-81e0-4f77-b2dd-2e44f5c10d06-workload-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:14:59.881773 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.881726    4364 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9059a7c7-81e0-4f77-b2dd-2e44f5c10d06-istio-token\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:14:59.881773 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.881740    4364 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9059a7c7-81e0-4f77-b2dd-2e44f5c10d06-istio-podinfo\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:14:59.881773 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.881752    4364 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f1549da-ef30-4001-9eec-35b1040de3b6-istiod-ca-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:14:59.881773 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.881764    4364 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db45cea6-0b96-46b2-94ff-48caa8747410-credential-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:14:59.882096 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.881777    4364 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f1549da-ef30-4001-9eec-35b1040de3b6-istio-data\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:14:59.882096 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.881791    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjj9z\" (UniqueName: \"kubernetes.io/projected/9059a7c7-81e0-4f77-b2dd-2e44f5c10d06-kube-api-access-gjj9z\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:14:59.882096 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.881805    4364 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db45cea6-0b96-46b2-94ff-48caa8747410-workload-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:14:59.882096 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.881816    4364 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9059a7c7-81e0-4f77-b2dd-2e44f5c10d06-istio-envoy\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:14:59.882096 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.881827    4364 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9059a7c7-81e0-4f77-b2dd-2e44f5c10d06-credential-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:14:59.882096 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.881840    4364 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9059a7c7-81e0-4f77-b2dd-2e44f5c10d06-istio-data\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:14:59.882096 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.882060    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f1549da-ef30-4001-9eec-35b1040de3b6-workload-certs" pod "3f1549da-ef30-4001-9eec-35b1040de3b6" (UID: "3f1549da-ef30-4001-9eec-35b1040de3b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:14:59.882460 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.882126    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f1549da-ef30-4001-9eec-35b1040de3b6-credential-socket" pod "3f1549da-ef30-4001-9eec-35b1040de3b6" (UID: "3f1549da-ef30-4001-9eec-35b1040de3b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:14:59.882573 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.882554    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm"]
Oct 30 13:14:59.882834 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.882802    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db45cea6-0b96-46b2-94ff-48caa8747410-istiod-ca-cert" pod "db45cea6-0b96-46b2-94ff-48caa8747410" (UID: "db45cea6-0b96-46b2-94ff-48caa8747410"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:14:59.885393 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.885365    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db45cea6-0b96-46b2-94ff-48caa8747410-istio-token" pod "db45cea6-0b96-46b2-94ff-48caa8747410" (UID: "db45cea6-0b96-46b2-94ff-48caa8747410"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:14:59.885550 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.885527    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f1549da-ef30-4001-9eec-35b1040de3b6-istio-token" pod "3f1549da-ef30-4001-9eec-35b1040de3b6" (UID: "3f1549da-ef30-4001-9eec-35b1040de3b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:14:59.885672 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.885640    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f1549da-ef30-4001-9eec-35b1040de3b6-kube-api-access-t5kdk" pod "3f1549da-ef30-4001-9eec-35b1040de3b6" (UID: "3f1549da-ef30-4001-9eec-35b1040de3b6"). InnerVolumeSpecName "kube-api-access-t5kdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:14:59.885975 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.885955    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db45cea6-0b96-46b2-94ff-48caa8747410-kube-api-access-lfshc" pod "db45cea6-0b96-46b2-94ff-48caa8747410" (UID: "db45cea6-0b96-46b2-94ff-48caa8747410"). InnerVolumeSpecName "kube-api-access-lfshc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:14:59.886348 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.886325    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f1549da-ef30-4001-9eec-35b1040de3b6-istio-podinfo" pod "3f1549da-ef30-4001-9eec-35b1040de3b6" (UID: "3f1549da-ef30-4001-9eec-35b1040de3b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:14:59.886436 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.886404    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db45cea6-0b96-46b2-94ff-48caa8747410-istio-envoy" pod "db45cea6-0b96-46b2-94ff-48caa8747410" (UID: "db45cea6-0b96-46b2-94ff-48caa8747410"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:14:59.886683 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.886649    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f1549da-ef30-4001-9eec-35b1040de3b6-istio-envoy" pod "3f1549da-ef30-4001-9eec-35b1040de3b6" (UID: "3f1549da-ef30-4001-9eec-35b1040de3b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:14:59.887421 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.887396    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db45cea6-0b96-46b2-94ff-48caa8747410-istio-podinfo" pod "db45cea6-0b96-46b2-94ff-48caa8747410" (UID: "db45cea6-0b96-46b2-94ff-48caa8747410"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:14:59.898440 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.898421    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-2daeae-openshift-default-dockercfg-nmz8n\""
Oct 30 13:14:59.983987 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.983859    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ef21594-2f1f-4b43-9c5f-529c9f9763f7-istio-podinfo\") pod \"e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm\" (UID: \"9ef21594-2f1f-4b43-9c5f-529c9f9763f7\") " pod="openshift-ingress/e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm"
Oct 30 13:14:59.994582 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.985798    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93fa52c8-91ad-40ef-a3b0-46c740175647-istio-podinfo\") pod \"e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n\" (UID: \"93fa52c8-91ad-40ef-a3b0-46c740175647\") " pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n"
Oct 30 13:14:59.994582 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.985862    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ef21594-2f1f-4b43-9c5f-529c9f9763f7-workload-certs\") pod \"e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm\" (UID: \"9ef21594-2f1f-4b43-9c5f-529c9f9763f7\") " pod="openshift-ingress/e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm"
Oct 30 13:14:59.994582 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.985955    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93fa52c8-91ad-40ef-a3b0-46c740175647-istio-token\") pod \"e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n\" (UID: \"93fa52c8-91ad-40ef-a3b0-46c740175647\") " pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n"
Oct 30 13:14:59.994582 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.985979    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93fa52c8-91ad-40ef-a3b0-46c740175647-istiod-ca-cert\") pod \"e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n\" (UID: \"93fa52c8-91ad-40ef-a3b0-46c740175647\") " pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n"
Oct 30 13:14:59.994582 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.986004    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ef21594-2f1f-4b43-9c5f-529c9f9763f7-credential-socket\") pod \"e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm\" (UID: \"9ef21594-2f1f-4b43-9c5f-529c9f9763f7\") " pod="openshift-ingress/e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm"
Oct 30 13:14:59.994582 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.986037    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93fa52c8-91ad-40ef-a3b0-46c740175647-workload-socket\") pod \"e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n\" (UID: \"93fa52c8-91ad-40ef-a3b0-46c740175647\") " pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n"
Oct 30 13:14:59.994582 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.986071    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93fa52c8-91ad-40ef-a3b0-46c740175647-istio-envoy\") pod \"e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n\" (UID: \"93fa52c8-91ad-40ef-a3b0-46c740175647\") " pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n"
Oct 30 13:14:59.994582 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.986124    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ef21594-2f1f-4b43-9c5f-529c9f9763f7-istio-data\") pod \"e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm\" (UID: \"9ef21594-2f1f-4b43-9c5f-529c9f9763f7\") " pod="openshift-ingress/e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm"
Oct 30 13:14:59.994582 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.986176    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qskw\" (UniqueName: \"kubernetes.io/projected/93fa52c8-91ad-40ef-a3b0-46c740175647-kube-api-access-8qskw\") pod \"e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n\" (UID: \"93fa52c8-91ad-40ef-a3b0-46c740175647\") " pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n"
Oct 30 13:14:59.994582 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.986256    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93fa52c8-91ad-40ef-a3b0-46c740175647-credential-socket\") pod \"e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n\" (UID: \"93fa52c8-91ad-40ef-a3b0-46c740175647\") " pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n"
Oct 30 13:14:59.994582 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.986302    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ef21594-2f1f-4b43-9c5f-529c9f9763f7-istiod-ca-cert\") pod \"e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm\" (UID: \"9ef21594-2f1f-4b43-9c5f-529c9f9763f7\") " pod="openshift-ingress/e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm"
Oct 30 13:14:59.994582 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.986342    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ef21594-2f1f-4b43-9c5f-529c9f9763f7-workload-socket\") pod \"e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm\" (UID: \"9ef21594-2f1f-4b43-9c5f-529c9f9763f7\") " pod="openshift-ingress/e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm"
Oct 30 13:14:59.994582 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.986374    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfpbm\" (UniqueName: \"kubernetes.io/projected/9ef21594-2f1f-4b43-9c5f-529c9f9763f7-kube-api-access-gfpbm\") pod \"e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm\" (UID: \"9ef21594-2f1f-4b43-9c5f-529c9f9763f7\") " pod="openshift-ingress/e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm"
Oct 30 13:14:59.994582 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.986404    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93fa52c8-91ad-40ef-a3b0-46c740175647-istio-data\") pod \"e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n\" (UID: \"93fa52c8-91ad-40ef-a3b0-46c740175647\") " pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n"
Oct 30 13:14:59.994582 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.986448    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93fa52c8-91ad-40ef-a3b0-46c740175647-workload-certs\") pod \"e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n\" (UID: \"93fa52c8-91ad-40ef-a3b0-46c740175647\") " pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n"
Oct 30 13:14:59.994582 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.986482    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ef21594-2f1f-4b43-9c5f-529c9f9763f7-istio-envoy\") pod \"e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm\" (UID: \"9ef21594-2f1f-4b43-9c5f-529c9f9763f7\") " pod="openshift-ingress/e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm"
Oct 30 13:14:59.994582 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.986515    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ef21594-2f1f-4b43-9c5f-529c9f9763f7-istio-token\") pod \"e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm\" (UID: \"9ef21594-2f1f-4b43-9c5f-529c9f9763f7\") " pod="openshift-ingress/e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm"
Oct 30 13:14:59.994582 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.986624    4364 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db45cea6-0b96-46b2-94ff-48caa8747410-istiod-ca-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:14:59.994582 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.986641    4364 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f1549da-ef30-4001-9eec-35b1040de3b6-istio-envoy\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:14:59.994582 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.986655    4364 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db45cea6-0b96-46b2-94ff-48caa8747410-istio-podinfo\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:14:59.994582 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.986670    4364 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db45cea6-0b96-46b2-94ff-48caa8747410-istio-envoy\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:14:59.994582 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.986683    4364 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f1549da-ef30-4001-9eec-35b1040de3b6-workload-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:14:59.994582 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.986696    4364 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f1549da-ef30-4001-9eec-35b1040de3b6-credential-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:14:59.994582 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.986711    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfshc\" (UniqueName: \"kubernetes.io/projected/db45cea6-0b96-46b2-94ff-48caa8747410-kube-api-access-lfshc\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:14:59.994582 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.986724    4364 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f1549da-ef30-4001-9eec-35b1040de3b6-istio-token\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:14:59.994582 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.986740    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5kdk\" (UniqueName: \"kubernetes.io/projected/3f1549da-ef30-4001-9eec-35b1040de3b6-kube-api-access-t5kdk\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:14:59.994582 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.986754    4364 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db45cea6-0b96-46b2-94ff-48caa8747410-istio-token\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:14:59.994582 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.986767    4364 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f1549da-ef30-4001-9eec-35b1040de3b6-istio-podinfo\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:14:59.994582 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.988167    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93fa52c8-91ad-40ef-a3b0-46c740175647-istiod-ca-cert\") pod \"e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n\" (UID: \"93fa52c8-91ad-40ef-a3b0-46c740175647\") " pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n"
Oct 30 13:14:59.994582 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.988455    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93fa52c8-91ad-40ef-a3b0-46c740175647-credential-socket\") pod \"e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n\" (UID: \"93fa52c8-91ad-40ef-a3b0-46c740175647\") " pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n"
Oct 30 13:14:59.994582 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.989034    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93fa52c8-91ad-40ef-a3b0-46c740175647-workload-socket\") pod \"e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n\" (UID: \"93fa52c8-91ad-40ef-a3b0-46c740175647\") " pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n"
Oct 30 13:14:59.994582 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.989274    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93fa52c8-91ad-40ef-a3b0-46c740175647-workload-certs\") pod \"e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n\" (UID: \"93fa52c8-91ad-40ef-a3b0-46c740175647\") " pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n"
Oct 30 13:14:59.994582 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:14:59.989575    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93fa52c8-91ad-40ef-a3b0-46c740175647-istio-data\") pod \"e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n\" (UID: \"93fa52c8-91ad-40ef-a3b0-46c740175647\") " pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n"
Oct 30 13:15:00.035552 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.028142    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93fa52c8-91ad-40ef-a3b0-46c740175647-istio-podinfo\") pod \"e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n\" (UID: \"93fa52c8-91ad-40ef-a3b0-46c740175647\") " pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n"
Oct 30 13:15:00.067461 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.051387    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93fa52c8-91ad-40ef-a3b0-46c740175647-istio-envoy\") pod \"e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n\" (UID: \"93fa52c8-91ad-40ef-a3b0-46c740175647\") " pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n"
Oct 30 13:15:00.076696 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.076527    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93fa52c8-91ad-40ef-a3b0-46c740175647-istio-token\") pod \"e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n\" (UID: \"93fa52c8-91ad-40ef-a3b0-46c740175647\") " pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n"
Oct 30 13:15:00.121524 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.098985    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ef21594-2f1f-4b43-9c5f-529c9f9763f7-istio-envoy\") pod \"e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm\" (UID: \"9ef21594-2f1f-4b43-9c5f-529c9f9763f7\") " pod="openshift-ingress/e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm"
Oct 30 13:15:00.121524 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.099043    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ef21594-2f1f-4b43-9c5f-529c9f9763f7-istio-token\") pod \"e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm\" (UID: \"9ef21594-2f1f-4b43-9c5f-529c9f9763f7\") " pod="openshift-ingress/e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm"
Oct 30 13:15:00.121524 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.099100    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ef21594-2f1f-4b43-9c5f-529c9f9763f7-istio-podinfo\") pod \"e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm\" (UID: \"9ef21594-2f1f-4b43-9c5f-529c9f9763f7\") " pod="openshift-ingress/e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm"
Oct 30 13:15:00.121524 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.099271    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ef21594-2f1f-4b43-9c5f-529c9f9763f7-workload-certs\") pod \"e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm\" (UID: \"9ef21594-2f1f-4b43-9c5f-529c9f9763f7\") " pod="openshift-ingress/e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm"
Oct 30 13:15:00.121524 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.099395    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ef21594-2f1f-4b43-9c5f-529c9f9763f7-credential-socket\") pod \"e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm\" (UID: \"9ef21594-2f1f-4b43-9c5f-529c9f9763f7\") " pod="openshift-ingress/e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm"
Oct 30 13:15:00.121524 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.099482    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ef21594-2f1f-4b43-9c5f-529c9f9763f7-istio-data\") pod \"e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm\" (UID: \"9ef21594-2f1f-4b43-9c5f-529c9f9763f7\") " pod="openshift-ingress/e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm"
Oct 30 13:15:00.121524 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.099627    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ef21594-2f1f-4b43-9c5f-529c9f9763f7-istiod-ca-cert\") pod \"e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm\" (UID: \"9ef21594-2f1f-4b43-9c5f-529c9f9763f7\") " pod="openshift-ingress/e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm"
Oct 30 13:15:00.121524 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.099668    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ef21594-2f1f-4b43-9c5f-529c9f9763f7-workload-socket\") pod \"e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm\" (UID: \"9ef21594-2f1f-4b43-9c5f-529c9f9763f7\") " pod="openshift-ingress/e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm"
Oct 30 13:15:00.121524 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.099699    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfpbm\" (UniqueName: \"kubernetes.io/projected/9ef21594-2f1f-4b43-9c5f-529c9f9763f7-kube-api-access-gfpbm\") pod \"e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm\" (UID: \"9ef21594-2f1f-4b43-9c5f-529c9f9763f7\") " pod="openshift-ingress/e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm"
Oct 30 13:15:00.121524 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.100768    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ef21594-2f1f-4b43-9c5f-529c9f9763f7-credential-socket\") pod \"e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm\" (UID: \"9ef21594-2f1f-4b43-9c5f-529c9f9763f7\") " pod="openshift-ingress/e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm"
Oct 30 13:15:00.121524 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.102146    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ef21594-2f1f-4b43-9c5f-529c9f9763f7-istiod-ca-cert\") pod \"e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm\" (UID: \"9ef21594-2f1f-4b43-9c5f-529c9f9763f7\") " pod="openshift-ingress/e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm"
Oct 30 13:15:00.121524 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.102433    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ef21594-2f1f-4b43-9c5f-529c9f9763f7-istio-data\") pod \"e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm\" (UID: \"9ef21594-2f1f-4b43-9c5f-529c9f9763f7\") " pod="openshift-ingress/e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm"
Oct 30 13:15:00.121524 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.102749    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ef21594-2f1f-4b43-9c5f-529c9f9763f7-workload-certs\") pod \"e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm\" (UID: \"9ef21594-2f1f-4b43-9c5f-529c9f9763f7\") " pod="openshift-ingress/e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm"
Oct 30 13:15:00.121524 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.102960    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ef21594-2f1f-4b43-9c5f-529c9f9763f7-workload-socket\") pod \"e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm\" (UID: \"9ef21594-2f1f-4b43-9c5f-529c9f9763f7\") " pod="openshift-ingress/e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm"
Oct 30 13:15:00.132739 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.125223    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qskw\" (UniqueName: \"kubernetes.io/projected/93fa52c8-91ad-40ef-a3b0-46c740175647-kube-api-access-8qskw\") pod \"e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n\" (UID: \"93fa52c8-91ad-40ef-a3b0-46c740175647\") " pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n"
Oct 30 13:15:00.151892 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.151298    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfpbm\" (UniqueName: \"kubernetes.io/projected/9ef21594-2f1f-4b43-9c5f-529c9f9763f7-kube-api-access-gfpbm\") pod \"e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm\" (UID: \"9ef21594-2f1f-4b43-9c5f-529c9f9763f7\") " pod="openshift-ingress/e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm"
Oct 30 13:15:00.151892 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.151838    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ef21594-2f1f-4b43-9c5f-529c9f9763f7-istio-envoy\") pod \"e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm\" (UID: \"9ef21594-2f1f-4b43-9c5f-529c9f9763f7\") " pod="openshift-ingress/e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm"
Oct 30 13:15:00.152625 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.152562    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ef21594-2f1f-4b43-9c5f-529c9f9763f7-istio-podinfo\") pod \"e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm\" (UID: \"9ef21594-2f1f-4b43-9c5f-529c9f9763f7\") " pod="openshift-ingress/e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm"
Oct 30 13:15:00.153234 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.153188    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ef21594-2f1f-4b43-9c5f-529c9f9763f7-istio-token\") pod \"e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm\" (UID: \"9ef21594-2f1f-4b43-9c5f-529c9f9763f7\") " pod="openshift-ingress/e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm"
Oct 30 13:15:00.230814 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.230347    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-df6c3661-7f764d9794-zph7s" podUID="e50284eb-016e-4bc8-bf70-b3eb2815493c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:15:00.259360 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.241036    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d7e62b-29889435-2fb6n"]
Oct 30 13:15:00.273415 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.272287    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d7e62b-29889435-2fb6n"
Oct 30 13:15:00.292681 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.292644    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d7e62b-29889435-2fb6n"]
Oct 30 13:15:00.323366 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.317479    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\""
Oct 30 13:15:00.406628 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.403588    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n"
Oct 30 13:15:00.428627 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.407729    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cb58b8-29889435-25tf8"]
Oct 30 13:15:00.428627 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.418685    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2e90cd6-1c18-4edb-ae15-7caed4920fe5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d7e62b-29889435-2fb6n\" (UID: \"b2e90cd6-1c18-4edb-ae15-7caed4920fe5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d7e62b-29889435-2fb6n"
Oct 30 13:15:00.428627 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.418725    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdz42\" (UniqueName: \"kubernetes.io/projected/b2e90cd6-1c18-4edb-ae15-7caed4920fe5-kube-api-access-zdz42\") pod \"maas-api-key-cleanup-e2e-worker-w4-d7e62b-29889435-2fb6n\" (UID: \"b2e90cd6-1c18-4edb-ae15-7caed4920fe5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d7e62b-29889435-2fb6n"
Oct 30 13:15:00.451642 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.431503    4364 generic.go:356] "Generic (PLEG): container finished" podID="db45cea6-0b96-46b2-94ff-48caa8747410" containerID="82d874cf78bd4de431a430ad9cc436dde8678c678fabea99abe352c627a0e7b2" exitCode=0
Oct 30 13:15:00.451642 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.431758    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q"
Oct 30 13:15:00.451642 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.443538    4364 generic.go:356] "Generic (PLEG): container finished" podID="9059a7c7-81e0-4f77-b2dd-2e44f5c10d06" containerID="aa8476f1323579494f81445c97c4f7b9bf1bb9c2704b99d7d691701a3ced19db" exitCode=0
Oct 30 13:15:00.451642 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.443750    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425"
Oct 30 13:15:00.461820 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.461193    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm"
Oct 30 13:15:00.473855 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.473506    4364 generic.go:356] "Generic (PLEG): container finished" podID="3f1549da-ef30-4001-9eec-35b1040de3b6" containerID="8245749a780d8d79e3a62f3120c8bfb5456abe2999f435a12322852a787eae13" exitCode=0
Oct 30 13:15:00.473855 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.473671    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww"
Oct 30 13:15:00.537951 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.535958    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2e90cd6-1c18-4edb-ae15-7caed4920fe5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d7e62b-29889435-2fb6n\" (UID: \"b2e90cd6-1c18-4edb-ae15-7caed4920fe5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d7e62b-29889435-2fb6n"
Oct 30 13:15:00.537951 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.536005    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdz42\" (UniqueName: \"kubernetes.io/projected/b2e90cd6-1c18-4edb-ae15-7caed4920fe5-kube-api-access-zdz42\") pod \"maas-api-key-cleanup-e2e-worker-w4-d7e62b-29889435-2fb6n\" (UID: \"b2e90cd6-1c18-4edb-ae15-7caed4920fe5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d7e62b-29889435-2fb6n"
Oct 30 13:15:00.537951 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.536939    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889435-kgl8r"]
Oct 30 13:15:00.537951 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.537477    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2e90cd6-1c18-4edb-ae15-7caed4920fe5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d7e62b-29889435-2fb6n\" (UID: \"b2e90cd6-1c18-4edb-ae15-7caed4920fe5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d7e62b-29889435-2fb6n"
Oct 30 13:15:00.543243 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.541755    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cb58b8-29889435-25tf8"
Oct 30 13:15:00.616884 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.616847    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q" event={"ID":"db45cea6-0b96-46b2-94ff-48caa8747410","Type":"ContainerDied","Data":"82d874cf78bd4de431a430ad9cc436dde8678c678fabea99abe352c627a0e7b2"}
Oct 30 13:15:00.617014 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.616914    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889435-c6pcv"]
Oct 30 13:15:00.618964 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.618941    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889435-kgl8r"
Oct 30 13:15:00.631152 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.631126    4364 scope.go:122] "RemoveContainer" containerID="82d874cf78bd4de431a430ad9cc436dde8678c678fabea99abe352c627a0e7b2"
Oct 30 13:15:00.639285 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.639004    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c581513-81ee-42a4-ad45-2ae012744caf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-cb58b8-29889435-25tf8\" (UID: \"1c581513-81ee-42a4-ad45-2ae012744caf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cb58b8-29889435-25tf8"
Oct 30 13:15:00.639285 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.639221    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ngv5\" (UniqueName: \"kubernetes.io/projected/1c581513-81ee-42a4-ad45-2ae012744caf-kube-api-access-9ngv5\") pod \"maas-api-key-cleanup-e2e-worker-w2-cb58b8-29889435-25tf8\" (UID: \"1c581513-81ee-42a4-ad45-2ae012744caf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cb58b8-29889435-25tf8"
Oct 30 13:15:00.641452 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.641423    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdz42\" (UniqueName: \"kubernetes.io/projected/b2e90cd6-1c18-4edb-ae15-7caed4920fe5-kube-api-access-zdz42\") pod \"maas-api-key-cleanup-e2e-worker-w4-d7e62b-29889435-2fb6n\" (UID: \"b2e90cd6-1c18-4edb-ae15-7caed4920fe5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d7e62b-29889435-2fb6n"
Oct 30 13:15:00.644842 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.644818    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q" event={"ID":"db45cea6-0b96-46b2-94ff-48caa8747410","Type":"ContainerDied","Data":"1ea2b0dc533b14111a786acda78bbc0d21565a66f0ffcd671d259a0870ed72ec"}
Oct 30 13:15:00.644934 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.644862    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889435-km6bf"]
Oct 30 13:15:00.645137 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.645119    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889435-c6pcv"
Oct 30 13:15:00.650724 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.650426    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889435-kgl8r"]
Oct 30 13:15:00.650724 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.650469    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cb58b8-29889435-25tf8"]
Oct 30 13:15:00.650724 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.650496    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425" event={"ID":"9059a7c7-81e0-4f77-b2dd-2e44f5c10d06","Type":"ContainerDied","Data":"aa8476f1323579494f81445c97c4f7b9bf1bb9c2704b99d7d691701a3ced19db"}
Oct 30 13:15:00.650724 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.650516    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425" event={"ID":"9059a7c7-81e0-4f77-b2dd-2e44f5c10d06","Type":"ContainerDied","Data":"1878efcbb7be7a5b1617a6575b516dcda284c86e351c736035d0c1d0d3e8f534"}
Oct 30 13:15:00.650724 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.650528    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww" event={"ID":"3f1549da-ef30-4001-9eec-35b1040de3b6","Type":"ContainerDied","Data":"8245749a780d8d79e3a62f3120c8bfb5456abe2999f435a12322852a787eae13"}
Oct 30 13:15:00.650724 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.650557    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889435-c6pcv"]
Oct 30 13:15:00.650724 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.650583    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889435-km6bf"]
Oct 30 13:15:00.651330 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.651130    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww" event={"ID":"3f1549da-ef30-4001-9eec-35b1040de3b6","Type":"ContainerDied","Data":"9479077bbf563bda5f61013c379a0c4f66ca43b1e97822a4afaae8a09135e95f"}
Oct 30 13:15:00.651330 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.651225    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889435-km6bf"
Oct 30 13:15:00.660343 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.656775    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-8b77b0-29889435-5cb9r"]
Oct 30 13:15:00.666164 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.666018    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-8b77b0-29889435-5cb9r"
Oct 30 13:15:00.668729 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.668671    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889435-vhbwb"]
Oct 30 13:15:00.674585 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.674568    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889435-vhbwb"
Oct 30 13:15:00.680017 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.679992    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d833bd-29889435-wkpwn"]
Oct 30 13:15:00.687898 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.687530    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d833bd-29889435-wkpwn"
Oct 30 13:15:00.688422 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.688400    4364 scope.go:122] "RemoveContainer" containerID="82d874cf78bd4de431a430ad9cc436dde8678c678fabea99abe352c627a0e7b2"
Oct 30 13:15:00.690045 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:15:00.689563    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82d874cf78bd4de431a430ad9cc436dde8678c678fabea99abe352c627a0e7b2\": container with ID starting with 82d874cf78bd4de431a430ad9cc436dde8678c678fabea99abe352c627a0e7b2 not found: ID does not exist" containerID="82d874cf78bd4de431a430ad9cc436dde8678c678fabea99abe352c627a0e7b2"
Oct 30 13:15:00.690045 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.689649    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82d874cf78bd4de431a430ad9cc436dde8678c678fabea99abe352c627a0e7b2"} err="failed to get container status \"82d874cf78bd4de431a430ad9cc436dde8678c678fabea99abe352c627a0e7b2\": rpc error: code = NotFound desc = could not find container \"82d874cf78bd4de431a430ad9cc436dde8678c678fabea99abe352c627a0e7b2\": container with ID starting with 82d874cf78bd4de431a430ad9cc436dde8678c678fabea99abe352c627a0e7b2 not found: ID does not exist"
Oct 30 13:15:00.690045 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.689681    4364 scope.go:122] "RemoveContainer" containerID="aa8476f1323579494f81445c97c4f7b9bf1bb9c2704b99d7d691701a3ced19db"
Oct 30 13:15:00.693682 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.692930    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5a6281-29889435-9sk2f"]
Oct 30 13:15:00.700240 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.700223    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5a6281-29889435-9sk2f"
Oct 30 13:15:00.703010 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.702991    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-659d72-29889435-zl5cx"]
Oct 30 13:15:00.709741 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.709727    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-659d72-29889435-zl5cx"
Oct 30 13:15:00.725858 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.723921    4364 scope.go:122] "RemoveContainer" containerID="aa8476f1323579494f81445c97c4f7b9bf1bb9c2704b99d7d691701a3ced19db"
Oct 30 13:15:00.725858 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.724215    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f0a31b-29889435-65zmv"]
Oct 30 13:15:00.727564 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:15:00.727547    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa8476f1323579494f81445c97c4f7b9bf1bb9c2704b99d7d691701a3ced19db\": container with ID starting with aa8476f1323579494f81445c97c4f7b9bf1bb9c2704b99d7d691701a3ced19db not found: ID does not exist" containerID="aa8476f1323579494f81445c97c4f7b9bf1bb9c2704b99d7d691701a3ced19db"
Oct 30 13:15:00.727712 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.727687    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa8476f1323579494f81445c97c4f7b9bf1bb9c2704b99d7d691701a3ced19db"} err="failed to get container status \"aa8476f1323579494f81445c97c4f7b9bf1bb9c2704b99d7d691701a3ced19db\": rpc error: code = NotFound desc = could not find container \"aa8476f1323579494f81445c97c4f7b9bf1bb9c2704b99d7d691701a3ced19db\": container with ID starting with aa8476f1323579494f81445c97c4f7b9bf1bb9c2704b99d7d691701a3ced19db not found: ID does not exist"
Oct 30 13:15:00.727808 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.727796    4364 scope.go:122] "RemoveContainer" containerID="8245749a780d8d79e3a62f3120c8bfb5456abe2999f435a12322852a787eae13"
Oct 30 13:15:00.729167 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.729151    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f0a31b-29889435-65zmv"
Oct 30 13:15:00.732924 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.732817    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889435-vhbwb"]
Oct 30 13:15:00.744642 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.744620    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-8b77b0-29889435-5cb9r"]
Oct 30 13:15:00.747893 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.747871    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkpcw\" (UniqueName: \"kubernetes.io/projected/d75b5256-b21b-47d6-8acb-45cb0afe1190-kube-api-access-zkpcw\") pod \"maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889435-c6pcv\" (UID: \"d75b5256-b21b-47d6-8acb-45cb0afe1190\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889435-c6pcv"
Oct 30 13:15:00.748646 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.748626    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbd3c692-16b9-44ea-98a9-1aa8852f7100-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889435-km6bf\" (UID: \"cbd3c692-16b9-44ea-98a9-1aa8852f7100\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889435-km6bf"
Oct 30 13:15:00.748895 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.748838    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxd56\" (UniqueName: \"kubernetes.io/projected/cbd3c692-16b9-44ea-98a9-1aa8852f7100-kube-api-access-zxd56\") pod \"maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889435-km6bf\" (UID: \"cbd3c692-16b9-44ea-98a9-1aa8852f7100\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889435-km6bf"
Oct 30 13:15:00.749250 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.749231    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcrmh\" (UniqueName: \"kubernetes.io/projected/77ada86d-3625-4786-a01d-2c422fdf7049-kube-api-access-wcrmh\") pod \"maas-api-key-cleanup-e2e-worker-w1-52a649-29889435-vhbwb\" (UID: \"77ada86d-3625-4786-a01d-2c422fdf7049\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889435-vhbwb"
Oct 30 13:15:00.749516 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.749499    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ngv5\" (UniqueName: \"kubernetes.io/projected/1c581513-81ee-42a4-ad45-2ae012744caf-kube-api-access-9ngv5\") pod \"maas-api-key-cleanup-e2e-worker-w2-cb58b8-29889435-25tf8\" (UID: \"1c581513-81ee-42a4-ad45-2ae012744caf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cb58b8-29889435-25tf8"
Oct 30 13:15:00.751941 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.751373    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5a6281-29889435-9sk2f"]
Oct 30 13:15:00.753772 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.753753    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d75b5256-b21b-47d6-8acb-45cb0afe1190-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889435-c6pcv\" (UID: \"d75b5256-b21b-47d6-8acb-45cb0afe1190\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889435-c6pcv"
Oct 30 13:15:00.754448 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.754428    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b7b85af-fbe9-4a34-b0d6-16068595c558-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-8b77b0-29889435-5cb9r\" (UID: \"5b7b85af-fbe9-4a34-b0d6-16068595c558\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-8b77b0-29889435-5cb9r"
Oct 30 13:15:00.754692 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.754667    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqq49\" (UniqueName: \"kubernetes.io/projected/0187e742-2686-400f-bb53-c74b0d65b5a8-kube-api-access-dqq49\") pod \"maas-api-key-cleanup-e2e-worker-w3-d833bd-29889435-wkpwn\" (UID: \"0187e742-2686-400f-bb53-c74b0d65b5a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d833bd-29889435-wkpwn"
Oct 30 13:15:00.754933 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.754915    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dabc5857-b065-4205-a160-48118225ee82-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889435-kgl8r\" (UID: \"dabc5857-b065-4205-a160-48118225ee82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889435-kgl8r"
Oct 30 13:15:00.755100 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.755085    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77ada86d-3625-4786-a01d-2c422fdf7049-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-52a649-29889435-vhbwb\" (UID: \"77ada86d-3625-4786-a01d-2c422fdf7049\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889435-vhbwb"
Oct 30 13:15:00.755289 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.755270    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmqhq\" (UniqueName: \"kubernetes.io/projected/5b7b85af-fbe9-4a34-b0d6-16068595c558-kube-api-access-vmqhq\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-8b77b0-29889435-5cb9r\" (UID: \"5b7b85af-fbe9-4a34-b0d6-16068595c558\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-8b77b0-29889435-5cb9r"
Oct 30 13:15:00.755548 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.755531    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c581513-81ee-42a4-ad45-2ae012744caf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-cb58b8-29889435-25tf8\" (UID: \"1c581513-81ee-42a4-ad45-2ae012744caf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cb58b8-29889435-25tf8"
Oct 30 13:15:00.756395 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.756377    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5q97\" (UniqueName: \"kubernetes.io/projected/dabc5857-b065-4205-a160-48118225ee82-kube-api-access-t5q97\") pod \"maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889435-kgl8r\" (UID: \"dabc5857-b065-4205-a160-48118225ee82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889435-kgl8r"
Oct 30 13:15:00.756664 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.756644    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0187e742-2686-400f-bb53-c74b0d65b5a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-d833bd-29889435-wkpwn\" (UID: \"0187e742-2686-400f-bb53-c74b0d65b5a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d833bd-29889435-wkpwn"
Oct 30 13:15:00.756883 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.756331    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c581513-81ee-42a4-ad45-2ae012744caf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-cb58b8-29889435-25tf8\" (UID: \"1c581513-81ee-42a4-ad45-2ae012744caf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cb58b8-29889435-25tf8"
Oct 30 13:15:00.760843 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.760724    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-659d72-29889435-zl5cx"]
Oct 30 13:15:00.766069 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.766004    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d833bd-29889435-wkpwn"]
Oct 30 13:15:00.771949 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.771934    4364 scope.go:122] "RemoveContainer" containerID="8245749a780d8d79e3a62f3120c8bfb5456abe2999f435a12322852a787eae13"
Oct 30 13:15:00.772420 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:15:00.772399    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8245749a780d8d79e3a62f3120c8bfb5456abe2999f435a12322852a787eae13\": container with ID starting with 8245749a780d8d79e3a62f3120c8bfb5456abe2999f435a12322852a787eae13 not found: ID does not exist" containerID="8245749a780d8d79e3a62f3120c8bfb5456abe2999f435a12322852a787eae13"
Oct 30 13:15:00.772550 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.772528    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8245749a780d8d79e3a62f3120c8bfb5456abe2999f435a12322852a787eae13"} err="failed to get container status \"8245749a780d8d79e3a62f3120c8bfb5456abe2999f435a12322852a787eae13\": rpc error: code = NotFound desc = could not find container \"8245749a780d8d79e3a62f3120c8bfb5456abe2999f435a12322852a787eae13\": container with ID starting with 8245749a780d8d79e3a62f3120c8bfb5456abe2999f435a12322852a787eae13 not found: ID does not exist"
Oct 30 13:15:00.773830 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.773773    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f0a31b-29889435-65zmv"]
Oct 30 13:15:00.774013 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.773968    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ngv5\" (UniqueName: \"kubernetes.io/projected/1c581513-81ee-42a4-ad45-2ae012744caf-kube-api-access-9ngv5\") pod \"maas-api-key-cleanup-e2e-worker-w2-cb58b8-29889435-25tf8\" (UID: \"1c581513-81ee-42a4-ad45-2ae012744caf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cb58b8-29889435-25tf8"
Oct 30 13:15:00.860478 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.860426    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b7b85af-fbe9-4a34-b0d6-16068595c558-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-8b77b0-29889435-5cb9r\" (UID: \"5b7b85af-fbe9-4a34-b0d6-16068595c558\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-8b77b0-29889435-5cb9r"
Oct 30 13:15:00.860778 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.860742    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqq49\" (UniqueName: \"kubernetes.io/projected/0187e742-2686-400f-bb53-c74b0d65b5a8-kube-api-access-dqq49\") pod \"maas-api-key-cleanup-e2e-worker-w3-d833bd-29889435-wkpwn\" (UID: \"0187e742-2686-400f-bb53-c74b0d65b5a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d833bd-29889435-wkpwn"
Oct 30 13:15:00.860999 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.860981    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2bgn\" (UniqueName: \"kubernetes.io/projected/98232f3c-5612-48bf-b272-439425a1fe37-kube-api-access-s2bgn\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-5a6281-29889435-9sk2f\" (UID: \"98232f3c-5612-48bf-b272-439425a1fe37\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5a6281-29889435-9sk2f"
Oct 30 13:15:00.861168 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.861146    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dabc5857-b065-4205-a160-48118225ee82-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889435-kgl8r\" (UID: \"dabc5857-b065-4205-a160-48118225ee82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889435-kgl8r"
Oct 30 13:15:00.861305 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.861290    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77ada86d-3625-4786-a01d-2c422fdf7049-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-52a649-29889435-vhbwb\" (UID: \"77ada86d-3625-4786-a01d-2c422fdf7049\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889435-vhbwb"
Oct 30 13:15:00.861473 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.861460    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmqhq\" (UniqueName: \"kubernetes.io/projected/5b7b85af-fbe9-4a34-b0d6-16068595c558-kube-api-access-vmqhq\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-8b77b0-29889435-5cb9r\" (UID: \"5b7b85af-fbe9-4a34-b0d6-16068595c558\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-8b77b0-29889435-5cb9r"
Oct 30 13:15:00.861702 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.861672    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ec36a29-9692-4a74-8297-c73fff6ab164-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-f0a31b-29889435-65zmv\" (UID: \"4ec36a29-9692-4a74-8297-c73fff6ab164\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f0a31b-29889435-65zmv"
Oct 30 13:15:00.861847 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.861832    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98232f3c-5612-48bf-b272-439425a1fe37-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-5a6281-29889435-9sk2f\" (UID: \"98232f3c-5612-48bf-b272-439425a1fe37\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5a6281-29889435-9sk2f"
Oct 30 13:15:00.862006 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.861980    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5q97\" (UniqueName: \"kubernetes.io/projected/dabc5857-b065-4205-a160-48118225ee82-kube-api-access-t5q97\") pod \"maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889435-kgl8r\" (UID: \"dabc5857-b065-4205-a160-48118225ee82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889435-kgl8r"
Oct 30 13:15:00.865683 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.865661    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0187e742-2686-400f-bb53-c74b0d65b5a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-d833bd-29889435-wkpwn\" (UID: \"0187e742-2686-400f-bb53-c74b0d65b5a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d833bd-29889435-wkpwn"
Oct 30 13:15:00.865963 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.865922    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkpcw\" (UniqueName: \"kubernetes.io/projected/d75b5256-b21b-47d6-8acb-45cb0afe1190-kube-api-access-zkpcw\") pod \"maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889435-c6pcv\" (UID: \"d75b5256-b21b-47d6-8acb-45cb0afe1190\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889435-c6pcv"
Oct 30 13:15:00.866141 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.866124    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbd3c692-16b9-44ea-98a9-1aa8852f7100-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889435-km6bf\" (UID: \"cbd3c692-16b9-44ea-98a9-1aa8852f7100\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889435-km6bf"
Oct 30 13:15:00.866330 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.866313    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxd56\" (UniqueName: \"kubernetes.io/projected/cbd3c692-16b9-44ea-98a9-1aa8852f7100-kube-api-access-zxd56\") pod \"maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889435-km6bf\" (UID: \"cbd3c692-16b9-44ea-98a9-1aa8852f7100\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889435-km6bf"
Oct 30 13:15:00.866519 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.866482    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcrmh\" (UniqueName: \"kubernetes.io/projected/77ada86d-3625-4786-a01d-2c422fdf7049-kube-api-access-wcrmh\") pod \"maas-api-key-cleanup-e2e-worker-w1-52a649-29889435-vhbwb\" (UID: \"77ada86d-3625-4786-a01d-2c422fdf7049\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889435-vhbwb"
Oct 30 13:15:00.866902 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.866863    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9p5h\" (UniqueName: \"kubernetes.io/projected/4ec36a29-9692-4a74-8297-c73fff6ab164-kube-api-access-k9p5h\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-f0a31b-29889435-65zmv\" (UID: \"4ec36a29-9692-4a74-8297-c73fff6ab164\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f0a31b-29889435-65zmv"
Oct 30 13:15:00.867088 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.867071    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d75b5256-b21b-47d6-8acb-45cb0afe1190-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889435-c6pcv\" (UID: \"d75b5256-b21b-47d6-8acb-45cb0afe1190\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889435-c6pcv"
Oct 30 13:15:00.868264 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.868246    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dc41559-3131-4c6d-bef5-f5cae1150954-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-659d72-29889435-zl5cx\" (UID: \"6dc41559-3131-4c6d-bef5-f5cae1150954\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-659d72-29889435-zl5cx"
Oct 30 13:15:00.868348 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.868271    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcq2b\" (UniqueName: \"kubernetes.io/projected/6dc41559-3131-4c6d-bef5-f5cae1150954-kube-api-access-mcq2b\") pod \"maas-api-key-cleanup-e2e-api-a-659d72-29889435-zl5cx\" (UID: \"6dc41559-3131-4c6d-bef5-f5cae1150954\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-659d72-29889435-zl5cx"
Oct 30 13:15:00.868411 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.863253    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b7b85af-fbe9-4a34-b0d6-16068595c558-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-8b77b0-29889435-5cb9r\" (UID: \"5b7b85af-fbe9-4a34-b0d6-16068595c558\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-8b77b0-29889435-5cb9r"
Oct 30 13:15:00.869696 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.864420    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77ada86d-3625-4786-a01d-2c422fdf7049-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-52a649-29889435-vhbwb\" (UID: \"77ada86d-3625-4786-a01d-2c422fdf7049\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889435-vhbwb"
Oct 30 13:15:00.869696 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.869156    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0187e742-2686-400f-bb53-c74b0d65b5a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-d833bd-29889435-wkpwn\" (UID: \"0187e742-2686-400f-bb53-c74b0d65b5a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d833bd-29889435-wkpwn"
Oct 30 13:15:00.869696 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.867125    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dabc5857-b065-4205-a160-48118225ee82-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889435-kgl8r\" (UID: \"dabc5857-b065-4205-a160-48118225ee82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889435-kgl8r"
Oct 30 13:15:00.869696 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.868107    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d75b5256-b21b-47d6-8acb-45cb0afe1190-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889435-c6pcv\" (UID: \"d75b5256-b21b-47d6-8acb-45cb0afe1190\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889435-c6pcv"
Oct 30 13:15:00.869979 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.869757    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbd3c692-16b9-44ea-98a9-1aa8852f7100-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889435-km6bf\" (UID: \"cbd3c692-16b9-44ea-98a9-1aa8852f7100\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889435-km6bf"
Oct 30 13:15:00.877662 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.877645    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425"]
Oct 30 13:15:00.920380 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.920346    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pp425"]
Oct 30 13:15:00.930485 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.930266    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d7e62b-29889435-2fb6n"
Oct 30 13:15:00.935557 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.935500    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkpcw\" (UniqueName: \"kubernetes.io/projected/d75b5256-b21b-47d6-8acb-45cb0afe1190-kube-api-access-zkpcw\") pod \"maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889435-c6pcv\" (UID: \"d75b5256-b21b-47d6-8acb-45cb0afe1190\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889435-c6pcv"
Oct 30 13:15:00.935557 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.935513    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmqhq\" (UniqueName: \"kubernetes.io/projected/5b7b85af-fbe9-4a34-b0d6-16068595c558-kube-api-access-vmqhq\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-8b77b0-29889435-5cb9r\" (UID: \"5b7b85af-fbe9-4a34-b0d6-16068595c558\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-8b77b0-29889435-5cb9r"
Oct 30 13:15:00.935557 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.935525    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5q97\" (UniqueName: \"kubernetes.io/projected/dabc5857-b065-4205-a160-48118225ee82-kube-api-access-t5q97\") pod \"maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889435-kgl8r\" (UID: \"dabc5857-b065-4205-a160-48118225ee82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889435-kgl8r"
Oct 30 13:15:00.935557 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.935525    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxd56\" (UniqueName: \"kubernetes.io/projected/cbd3c692-16b9-44ea-98a9-1aa8852f7100-kube-api-access-zxd56\") pod \"maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889435-km6bf\" (UID: \"cbd3c692-16b9-44ea-98a9-1aa8852f7100\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889435-km6bf"
Oct 30 13:15:00.936125 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.936103    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqq49\" (UniqueName: \"kubernetes.io/projected/0187e742-2686-400f-bb53-c74b0d65b5a8-kube-api-access-dqq49\") pod \"maas-api-key-cleanup-e2e-worker-w3-d833bd-29889435-wkpwn\" (UID: \"0187e742-2686-400f-bb53-c74b0d65b5a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d833bd-29889435-wkpwn"
Oct 30 13:15:00.936901 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.936884    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcrmh\" (UniqueName: \"kubernetes.io/projected/77ada86d-3625-4786-a01d-2c422fdf7049-kube-api-access-wcrmh\") pod \"maas-api-key-cleanup-e2e-worker-w1-52a649-29889435-vhbwb\" (UID: \"77ada86d-3625-4786-a01d-2c422fdf7049\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889435-vhbwb"
Oct 30 13:15:00.940028 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.940003    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww"]
Oct 30 13:15:00.945985 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.945957    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-wjgww"]
Oct 30 13:15:00.971072 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.971041    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ec36a29-9692-4a74-8297-c73fff6ab164-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-f0a31b-29889435-65zmv\" (UID: \"4ec36a29-9692-4a74-8297-c73fff6ab164\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f0a31b-29889435-65zmv"
Oct 30 13:15:00.971185 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.971098    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98232f3c-5612-48bf-b272-439425a1fe37-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-5a6281-29889435-9sk2f\" (UID: \"98232f3c-5612-48bf-b272-439425a1fe37\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5a6281-29889435-9sk2f"
Oct 30 13:15:00.971388 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.971365    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9p5h\" (UniqueName: \"kubernetes.io/projected/4ec36a29-9692-4a74-8297-c73fff6ab164-kube-api-access-k9p5h\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-f0a31b-29889435-65zmv\" (UID: \"4ec36a29-9692-4a74-8297-c73fff6ab164\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f0a31b-29889435-65zmv"
Oct 30 13:15:00.971675 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.971654    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dc41559-3131-4c6d-bef5-f5cae1150954-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-659d72-29889435-zl5cx\" (UID: \"6dc41559-3131-4c6d-bef5-f5cae1150954\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-659d72-29889435-zl5cx"
Oct 30 13:15:00.971831 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.971808    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcq2b\" (UniqueName: \"kubernetes.io/projected/6dc41559-3131-4c6d-bef5-f5cae1150954-kube-api-access-mcq2b\") pod \"maas-api-key-cleanup-e2e-api-a-659d72-29889435-zl5cx\" (UID: \"6dc41559-3131-4c6d-bef5-f5cae1150954\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-659d72-29889435-zl5cx"
Oct 30 13:15:00.971934 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.971844    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ec36a29-9692-4a74-8297-c73fff6ab164-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-f0a31b-29889435-65zmv\" (UID: \"4ec36a29-9692-4a74-8297-c73fff6ab164\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f0a31b-29889435-65zmv"
Oct 30 13:15:00.972003 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.971971    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98232f3c-5612-48bf-b272-439425a1fe37-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-5a6281-29889435-9sk2f\" (UID: \"98232f3c-5612-48bf-b272-439425a1fe37\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5a6281-29889435-9sk2f"
Oct 30 13:15:00.972239 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.972210    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2bgn\" (UniqueName: \"kubernetes.io/projected/98232f3c-5612-48bf-b272-439425a1fe37-kube-api-access-s2bgn\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-5a6281-29889435-9sk2f\" (UID: \"98232f3c-5612-48bf-b272-439425a1fe37\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5a6281-29889435-9sk2f"
Oct 30 13:15:00.972335 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.972284    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dc41559-3131-4c6d-bef5-f5cae1150954-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-659d72-29889435-zl5cx\" (UID: \"6dc41559-3131-4c6d-bef5-f5cae1150954\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-659d72-29889435-zl5cx"
Oct 30 13:15:00.977256 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.977218    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f1549da-ef30-4001-9eec-35b1040de3b6" path="/var/lib/kubelet/pods/3f1549da-ef30-4001-9eec-35b1040de3b6/volumes"
Oct 30 13:15:00.978128 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:00.978104    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9059a7c7-81e0-4f77-b2dd-2e44f5c10d06" path="/var/lib/kubelet/pods/9059a7c7-81e0-4f77-b2dd-2e44f5c10d06/volumes"
Oct 30 13:15:01.014717 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:01.014229    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cb58b8-29889435-25tf8"
Oct 30 13:15:01.015391 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:01.015059    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2bgn\" (UniqueName: \"kubernetes.io/projected/98232f3c-5612-48bf-b272-439425a1fe37-kube-api-access-s2bgn\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-5a6281-29889435-9sk2f\" (UID: \"98232f3c-5612-48bf-b272-439425a1fe37\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5a6281-29889435-9sk2f"
Oct 30 13:15:01.016443 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:01.016321    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcq2b\" (UniqueName: \"kubernetes.io/projected/6dc41559-3131-4c6d-bef5-f5cae1150954-kube-api-access-mcq2b\") pod \"maas-api-key-cleanup-e2e-api-a-659d72-29889435-zl5cx\" (UID: \"6dc41559-3131-4c6d-bef5-f5cae1150954\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-659d72-29889435-zl5cx"
Oct 30 13:15:01.017193 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:01.016921    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9p5h\" (UniqueName: \"kubernetes.io/projected/4ec36a29-9692-4a74-8297-c73fff6ab164-kube-api-access-k9p5h\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-f0a31b-29889435-65zmv\" (UID: \"4ec36a29-9692-4a74-8297-c73fff6ab164\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f0a31b-29889435-65zmv"
Oct 30 13:15:01.023088 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:15:01.022853    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93fa52c8_91ad_40ef_a3b0_46c740175647.slice/crio-adc70631b7d9be779aad9e349f1ee5db3843836aa2136a9e7a81ebcd51e5a586 WatchSource:0}: Error finding container adc70631b7d9be779aad9e349f1ee5db3843836aa2136a9e7a81ebcd51e5a586: Status 404 returned error can't find the container with id adc70631b7d9be779aad9e349f1ee5db3843836aa2136a9e7a81ebcd51e5a586
Oct 30 13:15:01.028413 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:01.027316    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n"]
Oct 30 13:15:01.029264 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:01.029246    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889435-kgl8r"
Oct 30 13:15:01.046044 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:01.044390    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm"]
Oct 30 13:15:01.065173 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:01.059847    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889435-c6pcv"
Oct 30 13:15:01.065613 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:01.065434    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889435-km6bf"
Oct 30 13:15:01.068826 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:01.067346    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:15:01.068826 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:01.067501    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:15:01.068826 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:01.067626    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:15:01.081103 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:01.081079    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-8b77b0-29889435-5cb9r"
Oct 30 13:15:01.090910 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:01.090835    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:15:01.095151 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:01.095116    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:15:01.095409 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:01.095382    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:15:01.110527 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:01.110497    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889435-vhbwb"
Oct 30 13:15:01.170943 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:01.170727    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d833bd-29889435-wkpwn"
Oct 30 13:15:01.209457 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:01.209400    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q"]
Oct 30 13:15:01.223891 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:01.220054    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-z8r9q"]
Oct 30 13:15:01.244325 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:01.243634    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-659d72-29889435-zl5cx"
Oct 30 13:15:01.245200 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:01.244971    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f0a31b-29889435-65zmv"
Oct 30 13:15:01.245200 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:01.245154    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5a6281-29889435-9sk2f"
Oct 30 13:15:01.308852 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:01.308816    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d7e62b-29889435-2fb6n"]
Oct 30 13:15:01.570459 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:01.570362    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d7e62b-29889435-2fb6n" event={"ID":"b2e90cd6-1c18-4edb-ae15-7caed4920fe5","Type":"ContainerStarted","Data":"fdd19f0cabd7ab7173cba85ca2c2d358d9b9e1cd5532f466cf2bbdda1c6d7114"}
Oct 30 13:15:01.572463 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:01.572373    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm" event={"ID":"9ef21594-2f1f-4b43-9c5f-529c9f9763f7","Type":"ContainerStarted","Data":"3a77f3235de1d4a14463b962dd9ce64acb54589f3205fff191d293a0cd610a7f"}
Oct 30 13:15:01.574089 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:01.574017    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n" event={"ID":"93fa52c8-91ad-40ef-a3b0-46c740175647","Type":"ContainerStarted","Data":"adc70631b7d9be779aad9e349f1ee5db3843836aa2136a9e7a81ebcd51e5a586"}
Oct 30 13:15:01.632652 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:01.632533    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cb58b8-29889435-25tf8"]
Oct 30 13:15:02.121475 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:15:02.121433    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbd3c692_16b9_44ea_98a9_1aa8852f7100.slice/crio-3c98c533eb319fb150d2bf69069295342232a5dd82043b079ddbce36936cfc63 WatchSource:0}: Error finding container 3c98c533eb319fb150d2bf69069295342232a5dd82043b079ddbce36936cfc63: Status 404 returned error can't find the container with id 3c98c533eb319fb150d2bf69069295342232a5dd82043b079ddbce36936cfc63
Oct 30 13:15:02.126698 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:02.126665    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889435-km6bf"]
Oct 30 13:15:02.133934 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:02.133910    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889435-kgl8r"]
Oct 30 13:15:02.176615 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:02.176512    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889435-c6pcv"]
Oct 30 13:15:02.235318 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:02.235285    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889435-vhbwb"]
Oct 30 13:15:02.250024 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:02.249907    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-3778dacd-b89667ff4-jm528" podUID="8fac787c-599b-4969-af01-42832173ed13" containerName="payload-processing" containerID="cri-o://8d3c89fe016a68e7c799e799e27509d1e818a133cd17bfabef4d89329648f87f" gracePeriod=25
Oct 30 13:15:02.330532 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:02.330471    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-8b77b0-29889435-5cb9r"]
Oct 30 13:15:02.428647 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:02.427892    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5a6281-29889435-9sk2f"]
Oct 30 13:15:02.472403 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:02.472349    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d833bd-29889435-wkpwn"]
Oct 30 13:15:02.525310 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:02.508826    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-659d72-29889435-zl5cx"]
Oct 30 13:15:02.671218 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:02.643285    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f0a31b-29889435-65zmv"]
Oct 30 13:15:03.075918 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:03.075826    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db45cea6-0b96-46b2-94ff-48caa8747410" path="/var/lib/kubelet/pods/db45cea6-0b96-46b2-94ff-48caa8747410/volumes"
Oct 30 13:15:03.076219 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:03.076186    4364 scope.go:122] "RemoveContainer" containerID="27865853a846f10754277684421877d3005b65020dcd9062d73ac18faaf0f6f5"
Oct 30 13:15:03.076379 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:03.076353    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d7e62b-29889435-2fb6n" event={"ID":"b2e90cd6-1c18-4edb-ae15-7caed4920fe5","Type":"ContainerStarted","Data":"27865853a846f10754277684421877d3005b65020dcd9062d73ac18faaf0f6f5"}
Oct 30 13:15:03.161234 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:03.161183    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889435-vhbwb" event={"ID":"77ada86d-3625-4786-a01d-2c422fdf7049","Type":"ContainerStarted","Data":"924ecd9596496f9711f6c36fc622b36da824c71ef7eb6b423c5b329a9d312cb1"}
Oct 30 13:15:03.165614 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:03.165571    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5a6281-29889435-9sk2f" event={"ID":"98232f3c-5612-48bf-b272-439425a1fe37","Type":"ContainerStarted","Data":"42db23c8a3ff3271dc8030c22884dafd700e76b4bc4b71f914611b88948711b3"}
Oct 30 13:15:03.245347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:03.201400    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm" event={"ID":"9ef21594-2f1f-4b43-9c5f-529c9f9763f7","Type":"ContainerStarted","Data":"e07d871ed6035bbaa3a9a1cf1365580f2ebcbb398ca33e4cebc9e511aa7a3527"}
Oct 30 13:15:03.277471 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:03.276031    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cb58b8-29889435-25tf8" event={"ID":"1c581513-81ee-42a4-ad45-2ae012744caf","Type":"ContainerStarted","Data":"e6dba3c3054e25aad6f8838764f007c247b83dd0a6baae525a288c3bbcbdd14f"}
Oct 30 13:15:03.277471 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:03.276097    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cb58b8-29889435-25tf8" event={"ID":"1c581513-81ee-42a4-ad45-2ae012744caf","Type":"ContainerStarted","Data":"0566ed2317f37da33d7b3b3e101560f9aa1ac3405bc15ecbe83e89e7025c0b29"}
Oct 30 13:15:03.277471 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:03.276626    4364 scope.go:122] "RemoveContainer" containerID="e6dba3c3054e25aad6f8838764f007c247b83dd0a6baae525a288c3bbcbdd14f"
Oct 30 13:15:03.468561 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:03.468489    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm"
Oct 30 13:15:03.713272 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:03.712770    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm"
Oct 30 13:15:03.971707 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:03.943055    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm" podStartSLOduration=6.943034259 podStartE2EDuration="6.943034259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:14:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:03.338625162 +0000 UTC m=+1402.504989811" watchObservedRunningTime="2026-10-30 13:15:03.943034259 +0000 UTC m=+1403.109398908"
Oct 30 13:15:04.147504 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:04.142273    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n" event={"ID":"93fa52c8-91ad-40ef-a3b0-46c740175647","Type":"ContainerStarted","Data":"8842099cc4a6b23133e36bc48ce1c11bfc89522294efe950a46721e681ce1455"}
Oct 30 13:15:04.365071 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:04.352518    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n" podStartSLOduration=10.352475259 podStartE2EDuration="10.352475259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:14:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:04.281068722 +0000 UTC m=+1403.447433371" watchObservedRunningTime="2026-10-30 13:15:04.352475259 +0000 UTC m=+1403.518839906"
Oct 30 13:15:04.365071 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:04.364337    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3778dacd-b89667ff4-jm528"
Oct 30 13:15:04.371359 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:04.371330    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889435-kgl8r" event={"ID":"dabc5857-b065-4205-a160-48118225ee82","Type":"ContainerStarted","Data":"6f5f3107a2f675c2a9c5a9fbc410a051dfdd8a4f613ec15f92ec30af95ac9b75"}
Oct 30 13:15:04.434947 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:04.421905    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n"
Oct 30 13:15:04.450980 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:04.435984    4364 patch_prober.go:29] interesting pod/e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 13:15:04.450980 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:04.436043    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n" podUID="93fa52c8-91ad-40ef-a3b0-46c740175647" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 13:15:04.460241 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:04.460212    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fac787c-599b-4969-af01-42832173ed13-kube-api-access-j4824\" (UniqueName: \"kubernetes.io/projected/8fac787c-599b-4969-af01-42832173ed13-kube-api-access-j4824\") pod \"8fac787c-599b-4969-af01-42832173ed13\" (UID: \"8fac787c-599b-4969-af01-42832173ed13\") "
Oct 30 13:15:04.460564 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:04.460542    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fac787c-599b-4969-af01-42832173ed13-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fac787c-599b-4969-af01-42832173ed13-plugins-config-volume\") pod \"8fac787c-599b-4969-af01-42832173ed13\" (UID: \"8fac787c-599b-4969-af01-42832173ed13\") "
Oct 30 13:15:04.463233 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:04.463198    4364 generic.go:356] "Generic (PLEG): container finished" podID="8fac787c-599b-4969-af01-42832173ed13" containerID="8d3c89fe016a68e7c799e799e27509d1e818a133cd17bfabef4d89329648f87f" exitCode=0
Oct 30 13:15:04.463327 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:04.463286    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3778dacd-b89667ff4-jm528" event={"ID":"8fac787c-599b-4969-af01-42832173ed13","Type":"ContainerDied","Data":"8d3c89fe016a68e7c799e799e27509d1e818a133cd17bfabef4d89329648f87f"}
Oct 30 13:15:04.463400 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:04.463339    4364 scope.go:122] "RemoveContainer" containerID="8d3c89fe016a68e7c799e799e27509d1e818a133cd17bfabef4d89329648f87f"
Oct 30 13:15:04.464959 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:04.464928    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fac787c-599b-4969-af01-42832173ed13-kube-api-access-j4824" pod "8fac787c-599b-4969-af01-42832173ed13" (UID: "8fac787c-599b-4969-af01-42832173ed13"). InnerVolumeSpecName "kube-api-access-j4824". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:04.467266 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:04.466706    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3778dacd-b89667ff4-jm528"
Oct 30 13:15:04.468729 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:04.468660    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-659d72-29889435-zl5cx" event={"ID":"6dc41559-3131-4c6d-bef5-f5cae1150954","Type":"ContainerStarted","Data":"2d7a47256dcf5c29d5fcff3096955f396ccf179f2a7ced31d7cbfb97aa463a8f"}
Oct 30 13:15:04.472582 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:04.471848    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889435-km6bf" event={"ID":"cbd3c692-16b9-44ea-98a9-1aa8852f7100","Type":"ContainerStarted","Data":"3c98c533eb319fb150d2bf69069295342232a5dd82043b079ddbce36936cfc63"}
Oct 30 13:15:04.480054 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:04.479691    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-8b77b0-29889435-5cb9r" event={"ID":"5b7b85af-fbe9-4a34-b0d6-16068595c558","Type":"ContainerStarted","Data":"4d06c9fa98f567597984fdb65f8c26915e00fd2e693f6ffcc2e967ce64f9c719"}
Oct 30 13:15:04.493755 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:04.484675    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889435-c6pcv" event={"ID":"d75b5256-b21b-47d6-8acb-45cb0afe1190","Type":"ContainerStarted","Data":"dbfcc8dd089c3e2c0ed92a8a900a96a5ccbbd4c02b6a581c42ccb25bcf18832c"}
Oct 30 13:15:04.497297 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:04.497225    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fac787c-599b-4969-af01-42832173ed13-plugins-config-volume" pod "8fac787c-599b-4969-af01-42832173ed13" (UID: "8fac787c-599b-4969-af01-42832173ed13"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:04.554220 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:04.554166    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889435-kgl8r" podStartSLOduration=4.5541483209999996 podStartE2EDuration="4.554148321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:04.500325096 +0000 UTC m=+1403.666689746" watchObservedRunningTime="2026-10-30 13:15:04.554148321 +0000 UTC m=+1403.720512985"
Oct 30 13:15:04.564253 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:04.564233    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4824\" (UniqueName: \"kubernetes.io/projected/8fac787c-599b-4969-af01-42832173ed13-kube-api-access-j4824\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:15:04.564354 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:04.564256    4364 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fac787c-599b-4969-af01-42832173ed13-plugins-config-volume\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:15:04.638688 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:04.638623    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889435-c6pcv" podStartSLOduration=4.638588719 podStartE2EDuration="4.638588719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:04.577781642 +0000 UTC m=+1403.744146301" watchObservedRunningTime="2026-10-30 13:15:04.638588719 +0000 UTC m=+1403.804953367"
Oct 30 13:15:04.645761 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:04.645708    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889435-km6bf" podStartSLOduration=4.645690776 podStartE2EDuration="4.645690776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:04.637345777 +0000 UTC m=+1403.803710425" watchObservedRunningTime="2026-10-30 13:15:04.645690776 +0000 UTC m=+1403.812055424"
Oct 30 13:15:04.846756 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:04.845827    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3778dacd-b89667ff4-jm528"]
Oct 30 13:15:04.851519 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:04.851476    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3778dacd-b89667ff4-jm528"]
Oct 30 13:15:05.085707 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:05.085149    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fac787c-599b-4969-af01-42832173ed13" path="/var/lib/kubelet/pods/8fac787c-599b-4969-af01-42832173ed13/volumes"
Oct 30 13:15:05.407442 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:05.406959    4364 patch_prober.go:29] interesting pod/e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 13:15:05.407442 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:05.407018    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n" podUID="93fa52c8-91ad-40ef-a3b0-46c740175647" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 13:15:05.539805 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:05.539770    4364 generic.go:356] "Generic (PLEG): container finished" podID="5b7b85af-fbe9-4a34-b0d6-16068595c558" containerID="4e901155e06a81fb310aaecf3e591b75c181d5b0ecbf481d5628433b0d058cb6" exitCode=60
Oct 30 13:15:05.539992 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:05.539850    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-8b77b0-29889435-5cb9r" event={"ID":"5b7b85af-fbe9-4a34-b0d6-16068595c558","Type":"ContainerDied","Data":"4e901155e06a81fb310aaecf3e591b75c181d5b0ecbf481d5628433b0d058cb6"}
Oct 30 13:15:05.540205 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:05.540187    4364 scope.go:122] "RemoveContainer" containerID="4e901155e06a81fb310aaecf3e591b75c181d5b0ecbf481d5628433b0d058cb6"
Oct 30 13:15:05.541583 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:05.541560    4364 generic.go:356] "Generic (PLEG): container finished" podID="d75b5256-b21b-47d6-8acb-45cb0afe1190" containerID="b8f98a8e318fcdb5380c1e93a1f947fee4e758db4dd773cf265d1070e2b5eb44" exitCode=60
Oct 30 13:15:05.542016 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:05.541726    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889435-c6pcv" event={"ID":"d75b5256-b21b-47d6-8acb-45cb0afe1190","Type":"ContainerDied","Data":"b8f98a8e318fcdb5380c1e93a1f947fee4e758db4dd773cf265d1070e2b5eb44"}
Oct 30 13:15:05.542403 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:05.542257    4364 scope.go:122] "RemoveContainer" containerID="b8f98a8e318fcdb5380c1e93a1f947fee4e758db4dd773cf265d1070e2b5eb44"
Oct 30 13:15:05.544966 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:05.544935    4364 generic.go:356] "Generic (PLEG): container finished" podID="b2e90cd6-1c18-4edb-ae15-7caed4920fe5" containerID="27865853a846f10754277684421877d3005b65020dcd9062d73ac18faaf0f6f5" exitCode=60
Oct 30 13:15:05.545045 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:05.545029    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d7e62b-29889435-2fb6n" event={"ID":"b2e90cd6-1c18-4edb-ae15-7caed4920fe5","Type":"ContainerDied","Data":"27865853a846f10754277684421877d3005b65020dcd9062d73ac18faaf0f6f5"}
Oct 30 13:15:05.545085 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:05.545073    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d7e62b-29889435-2fb6n" event={"ID":"b2e90cd6-1c18-4edb-ae15-7caed4920fe5","Type":"ContainerStarted","Data":"92feb3f4bc8d127003e5a6567766912a96d8f3af53e3f80e3229e0d5462e1da8"}
Oct 30 13:15:05.547423 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:05.547404    4364 generic.go:356] "Generic (PLEG): container finished" podID="77ada86d-3625-4786-a01d-2c422fdf7049" containerID="e45c456462e4d04efcf0e7d7470c8ba92dfb10cdf689abbb97c06b36db6bf01e" exitCode=60
Oct 30 13:15:05.547531 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:05.547494    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889435-vhbwb" event={"ID":"77ada86d-3625-4786-a01d-2c422fdf7049","Type":"ContainerDied","Data":"e45c456462e4d04efcf0e7d7470c8ba92dfb10cdf689abbb97c06b36db6bf01e"}
Oct 30 13:15:05.547744 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:05.547726    4364 scope.go:122] "RemoveContainer" containerID="e45c456462e4d04efcf0e7d7470c8ba92dfb10cdf689abbb97c06b36db6bf01e"
Oct 30 13:15:05.549209 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:05.549190    4364 generic.go:356] "Generic (PLEG): container finished" podID="98232f3c-5612-48bf-b272-439425a1fe37" containerID="8b51d66d66dbbf29081f30eeea6dc509df73c1730d992f4a106d7251bcfe8d7e" exitCode=60
Oct 30 13:15:05.549297 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:05.549280    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5a6281-29889435-9sk2f" event={"ID":"98232f3c-5612-48bf-b272-439425a1fe37","Type":"ContainerDied","Data":"8b51d66d66dbbf29081f30eeea6dc509df73c1730d992f4a106d7251bcfe8d7e"}
Oct 30 13:15:05.549560 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:05.549544    4364 scope.go:122] "RemoveContainer" containerID="8b51d66d66dbbf29081f30eeea6dc509df73c1730d992f4a106d7251bcfe8d7e"
Oct 30 13:15:05.551383 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:05.551365    4364 generic.go:356] "Generic (PLEG): container finished" podID="4ec36a29-9692-4a74-8297-c73fff6ab164" containerID="1652878b5c16a1d3ec4ff5a2aea5d08b9f2f4e51258a7aa34c2e8a122a815806" exitCode=60
Oct 30 13:15:05.551467 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:05.551443    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f0a31b-29889435-65zmv" event={"ID":"4ec36a29-9692-4a74-8297-c73fff6ab164","Type":"ContainerDied","Data":"1652878b5c16a1d3ec4ff5a2aea5d08b9f2f4e51258a7aa34c2e8a122a815806"}
Oct 30 13:15:05.551541 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:05.551502    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f0a31b-29889435-65zmv" event={"ID":"4ec36a29-9692-4a74-8297-c73fff6ab164","Type":"ContainerStarted","Data":"0a4696d2c18f37d11163446ed0692d096c57aa37bf30394b32f8f939c9ac6ee2"}
Oct 30 13:15:05.551709 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:05.551688    4364 scope.go:122] "RemoveContainer" containerID="1652878b5c16a1d3ec4ff5a2aea5d08b9f2f4e51258a7aa34c2e8a122a815806"
Oct 30 13:15:05.554072 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:05.554054    4364 generic.go:356] "Generic (PLEG): container finished" podID="1c581513-81ee-42a4-ad45-2ae012744caf" containerID="e6dba3c3054e25aad6f8838764f007c247b83dd0a6baae525a288c3bbcbdd14f" exitCode=60
Oct 30 13:15:05.554214 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:05.554198    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cb58b8-29889435-25tf8" event={"ID":"1c581513-81ee-42a4-ad45-2ae012744caf","Type":"ContainerDied","Data":"e6dba3c3054e25aad6f8838764f007c247b83dd0a6baae525a288c3bbcbdd14f"}
Oct 30 13:15:05.554258 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:05.554247    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cb58b8-29889435-25tf8" event={"ID":"1c581513-81ee-42a4-ad45-2ae012744caf","Type":"ContainerStarted","Data":"96a44356b20d787084159b054005ee91abae1620cb057a8a87748beee8692e84"}
Oct 30 13:15:05.557163 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:05.557144    4364 generic.go:356] "Generic (PLEG): container finished" podID="0187e742-2686-400f-bb53-c74b0d65b5a8" containerID="3a21574b8077d3e82622e57d34465d4550b26d4ac645b0b912429cdd24f687e9" exitCode=60
Oct 30 13:15:05.557247 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:05.557215    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d833bd-29889435-wkpwn" event={"ID":"0187e742-2686-400f-bb53-c74b0d65b5a8","Type":"ContainerDied","Data":"3a21574b8077d3e82622e57d34465d4550b26d4ac645b0b912429cdd24f687e9"}
Oct 30 13:15:05.557308 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:05.557249    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d833bd-29889435-wkpwn" event={"ID":"0187e742-2686-400f-bb53-c74b0d65b5a8","Type":"ContainerStarted","Data":"b38caacca2b0512753f113ca2c9cb0a1104c0a2685838f34be247e5aea98dd46"}
Oct 30 13:15:05.557495 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:05.557478    4364 scope.go:122] "RemoveContainer" containerID="3a21574b8077d3e82622e57d34465d4550b26d4ac645b0b912429cdd24f687e9"
Oct 30 13:15:05.559037 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:05.559016    4364 generic.go:356] "Generic (PLEG): container finished" podID="dabc5857-b065-4205-a160-48118225ee82" containerID="f549002fa94a2422220f0560c31fa3d6539368f9a1fea53ee7fa092a86a05a37" exitCode=60
Oct 30 13:15:05.559093 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:05.559070    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889435-kgl8r" event={"ID":"dabc5857-b065-4205-a160-48118225ee82","Type":"ContainerDied","Data":"f549002fa94a2422220f0560c31fa3d6539368f9a1fea53ee7fa092a86a05a37"}
Oct 30 13:15:05.559273 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:05.559257    4364 scope.go:122] "RemoveContainer" containerID="f549002fa94a2422220f0560c31fa3d6539368f9a1fea53ee7fa092a86a05a37"
Oct 30 13:15:05.561636 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:05.561615    4364 generic.go:356] "Generic (PLEG): container finished" podID="6dc41559-3131-4c6d-bef5-f5cae1150954" containerID="6bfba78bc6be9d52266a8c0a66146a7136154bfa2e83d9878971d5c20f043c39" exitCode=60
Oct 30 13:15:05.561724 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:05.561707    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-659d72-29889435-zl5cx" event={"ID":"6dc41559-3131-4c6d-bef5-f5cae1150954","Type":"ContainerDied","Data":"6bfba78bc6be9d52266a8c0a66146a7136154bfa2e83d9878971d5c20f043c39"}
Oct 30 13:15:05.561900 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:05.561883    4364 scope.go:122] "RemoveContainer" containerID="6bfba78bc6be9d52266a8c0a66146a7136154bfa2e83d9878971d5c20f043c39"
Oct 30 13:15:05.563151 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:05.563125    4364 generic.go:356] "Generic (PLEG): container finished" podID="cbd3c692-16b9-44ea-98a9-1aa8852f7100" containerID="88a606fc9b05d61618c238f1ad097014556b66e5a379683e0cec9cfab572eeae" exitCode=60
Oct 30 13:15:05.563246 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:05.563192    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889435-km6bf" event={"ID":"cbd3c692-16b9-44ea-98a9-1aa8852f7100","Type":"ContainerDied","Data":"88a606fc9b05d61618c238f1ad097014556b66e5a379683e0cec9cfab572eeae"}
Oct 30 13:15:05.563429 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:05.563409    4364 scope.go:122] "RemoveContainer" containerID="88a606fc9b05d61618c238f1ad097014556b66e5a379683e0cec9cfab572eeae"
Oct 30 13:15:05.563578 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:05.563560    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm"
Oct 30 13:15:05.564705 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:05.564688    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm"
Oct 30 13:15:05.849457 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:05.849399    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cb58b8-29889435-25tf8" podStartSLOduration=5.849380054 podStartE2EDuration="5.849380054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:05.835417382 +0000 UTC m=+1405.001782031" watchObservedRunningTime="2026-10-30 13:15:05.849380054 +0000 UTC m=+1405.015744703"
Oct 30 13:15:06.361686 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:06.360875    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d7e62b-29889435-2fb6n" podStartSLOduration=6.360854637 podStartE2EDuration="6.360854637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:06.242449274 +0000 UTC m=+1405.408813922" watchObservedRunningTime="2026-10-30 13:15:06.360854637 +0000 UTC m=+1405.527219284"
Oct 30 13:15:06.416759 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:06.406479    4364 patch_prober.go:29] interesting pod/e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 13:15:06.416759 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:06.406531    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n" podUID="93fa52c8-91ad-40ef-a3b0-46c740175647" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 13:15:06.609620 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:06.601465    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d833bd-29889435-wkpwn" event={"ID":"0187e742-2686-400f-bb53-c74b0d65b5a8","Type":"ContainerStarted","Data":"40917ca068e4a1e4e7b6836a650476e53943feb3b173f8b050ed7cd8fcff8480"}
Oct 30 13:15:06.635033 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:06.634985    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889435-kgl8r" event={"ID":"dabc5857-b065-4205-a160-48118225ee82","Type":"ContainerStarted","Data":"dd1dd833ef19ece80becc3685decd5681e3be2c7d869d9c86d05baf0f88f3c42"}
Oct 30 13:15:06.638632 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:06.638584    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-659d72-29889435-zl5cx" event={"ID":"6dc41559-3131-4c6d-bef5-f5cae1150954","Type":"ContainerStarted","Data":"b3d63cfdce1e9382be2bc3c4aadd4659cd51214760b613a4cac32dae977073db"}
Oct 30 13:15:06.641434 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:06.641409    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889435-km6bf" event={"ID":"cbd3c692-16b9-44ea-98a9-1aa8852f7100","Type":"ContainerStarted","Data":"0a2c43ece3593a9cc3d7f345241ec126ccb4822306efa117bf638c9cb7238160"}
Oct 30 13:15:06.644331 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:06.644293    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-8b77b0-29889435-5cb9r" event={"ID":"5b7b85af-fbe9-4a34-b0d6-16068595c558","Type":"ContainerStarted","Data":"fdde710636496e08450681aea7334adba2dcbcc4e49ad61c83335c21ba8c7c2c"}
Oct 30 13:15:06.647232 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:06.647211    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889435-c6pcv" event={"ID":"d75b5256-b21b-47d6-8acb-45cb0afe1190","Type":"ContainerStarted","Data":"59213a626651e495ec97d13088b697667f2308db10667f55c68e157ac23e7b04"}
Oct 30 13:15:06.649032 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:06.649011    4364 generic.go:356] "Generic (PLEG): container finished" podID="b2e90cd6-1c18-4edb-ae15-7caed4920fe5" containerID="92feb3f4bc8d127003e5a6567766912a96d8f3af53e3f80e3229e0d5462e1da8" exitCode=60
Oct 30 13:15:06.649122 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:06.649100    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d7e62b-29889435-2fb6n" event={"ID":"b2e90cd6-1c18-4edb-ae15-7caed4920fe5","Type":"ContainerDied","Data":"92feb3f4bc8d127003e5a6567766912a96d8f3af53e3f80e3229e0d5462e1da8"}
Oct 30 13:15:06.649186 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:06.649134    4364 scope.go:122] "RemoveContainer" containerID="27865853a846f10754277684421877d3005b65020dcd9062d73ac18faaf0f6f5"
Oct 30 13:15:06.649241 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:06.649225    4364 scope.go:122] "RemoveContainer" containerID="92feb3f4bc8d127003e5a6567766912a96d8f3af53e3f80e3229e0d5462e1da8"
Oct 30 13:15:06.649475 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:15:06.649457    4364 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-d7e62b-29889435-2fb6n_odh-ai-gateway-infra(b2e90cd6-1c18-4edb-ae15-7caed4920fe5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d7e62b-29889435-2fb6n" podUID="b2e90cd6-1c18-4edb-ae15-7caed4920fe5"
Oct 30 13:15:06.651147 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:06.651124    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889435-vhbwb" event={"ID":"77ada86d-3625-4786-a01d-2c422fdf7049","Type":"ContainerStarted","Data":"0e215363a50b60c725692aec841e2fee46e1d069c1f2896907e0b01c792e94a0"}
Oct 30 13:15:06.653125 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:06.653100    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5a6281-29889435-9sk2f" event={"ID":"98232f3c-5612-48bf-b272-439425a1fe37","Type":"ContainerStarted","Data":"50c541a0d3bbb0e2c3e06f944cc8d244bc60fc8a3c011dc26ac2dda924a18d23"}
Oct 30 13:15:06.654905 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:06.654880    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f0a31b-29889435-65zmv" event={"ID":"4ec36a29-9692-4a74-8297-c73fff6ab164","Type":"ContainerStarted","Data":"96ff74f504e1e06f47dd22d5fe887e0c1a89aa88c31ccacd90022b1712a6997c"}
Oct 30 13:15:06.656956 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:06.656928    4364 generic.go:356] "Generic (PLEG): container finished" podID="1c581513-81ee-42a4-ad45-2ae012744caf" containerID="96a44356b20d787084159b054005ee91abae1620cb057a8a87748beee8692e84" exitCode=60
Oct 30 13:15:06.657036 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:06.657007    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cb58b8-29889435-25tf8" event={"ID":"1c581513-81ee-42a4-ad45-2ae012744caf","Type":"ContainerDied","Data":"96a44356b20d787084159b054005ee91abae1620cb057a8a87748beee8692e84"}
Oct 30 13:15:06.657180 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:06.657164    4364 scope.go:122] "RemoveContainer" containerID="96a44356b20d787084159b054005ee91abae1620cb057a8a87748beee8692e84"
Oct 30 13:15:06.657501 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:15:06.657474    4364 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-cb58b8-29889435-25tf8_odh-ai-gateway-infra(1c581513-81ee-42a4-ad45-2ae012744caf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cb58b8-29889435-25tf8" podUID="1c581513-81ee-42a4-ad45-2ae012744caf"
Oct 30 13:15:06.662566 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:06.662546    4364 scope.go:122] "RemoveContainer" containerID="e6dba3c3054e25aad6f8838764f007c247b83dd0a6baae525a288c3bbcbdd14f"
Oct 30 13:15:06.664853 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:06.664818    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d833bd-29889435-wkpwn" podStartSLOduration=6.664807817 podStartE2EDuration="6.664807817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:06.660395119 +0000 UTC m=+1405.826759767" watchObservedRunningTime="2026-10-30 13:15:06.664807817 +0000 UTC m=+1405.831172463"
Oct 30 13:15:06.719039 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:06.718970    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-659d72-29889435-zl5cx" podStartSLOduration=6.718951444 podStartE2EDuration="6.718951444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:06.703646775 +0000 UTC m=+1405.870011423" watchObservedRunningTime="2026-10-30 13:15:06.718951444 +0000 UTC m=+1405.885316093"
Oct 30 13:15:06.746769 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:06.746720    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-8b77b0-29889435-5cb9r" podStartSLOduration=6.7467051510000005 podStartE2EDuration="6.746705151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:06.744336548 +0000 UTC m=+1405.910701217" watchObservedRunningTime="2026-10-30 13:15:06.746705151 +0000 UTC m=+1405.913069799"
Oct 30 13:15:06.850830 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:06.850786    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f0a31b-29889435-65zmv" podStartSLOduration=6.850770036 podStartE2EDuration="6.850770036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:06.848669419 +0000 UTC m=+1406.015034066" watchObservedRunningTime="2026-10-30 13:15:06.850770036 +0000 UTC m=+1406.017134685"
Oct 30 13:15:06.851390 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:06.851368    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5a6281-29889435-9sk2f" podStartSLOduration=6.851361278 podStartE2EDuration="6.851361278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:06.807882112 +0000 UTC m=+1405.974246760" watchObservedRunningTime="2026-10-30 13:15:06.851361278 +0000 UTC m=+1406.017725926"
Oct 30 13:15:07.111653 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:07.111182    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889435-vhbwb" podStartSLOduration=7.11116096 podStartE2EDuration="7.11116096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:07.089759324 +0000 UTC m=+1406.256123973" watchObservedRunningTime="2026-10-30 13:15:07.11116096 +0000 UTC m=+1406.277525609"
Oct 30 13:15:07.411622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:07.411461    4364 patch_prober.go:29] interesting pod/e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 13:15:07.411622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:07.411531    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n" podUID="93fa52c8-91ad-40ef-a3b0-46c740175647" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 13:15:07.700085 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:07.699996    4364 scope.go:122] "RemoveContainer" containerID="b3d63cfdce1e9382be2bc3c4aadd4659cd51214760b613a4cac32dae977073db"
Oct 30 13:15:07.701191 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:15:07.701161    4364 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-659d72-29889435-zl5cx_odh-ai-gateway-infra(6dc41559-3131-4c6d-bef5-f5cae1150954)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-659d72-29889435-zl5cx" podUID="6dc41559-3131-4c6d-bef5-f5cae1150954"
Oct 30 13:15:07.702077 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:07.702055    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-659d72-29889435-zl5cx" event={"ID":"6dc41559-3131-4c6d-bef5-f5cae1150954","Type":"ContainerDied","Data":"b3d63cfdce1e9382be2bc3c4aadd4659cd51214760b613a4cac32dae977073db"}
Oct 30 13:15:07.702328 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:07.702312    4364 scope.go:122] "RemoveContainer" containerID="6bfba78bc6be9d52266a8c0a66146a7136154bfa2e83d9878971d5c20f043c39"
Oct 30 13:15:07.708627 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:07.699406    4364 generic.go:356] "Generic (PLEG): container finished" podID="6dc41559-3131-4c6d-bef5-f5cae1150954" containerID="b3d63cfdce1e9382be2bc3c4aadd4659cd51214760b613a4cac32dae977073db" exitCode=60
Oct 30 13:15:07.709149 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:07.709119    4364 generic.go:356] "Generic (PLEG): container finished" podID="cbd3c692-16b9-44ea-98a9-1aa8852f7100" containerID="0a2c43ece3593a9cc3d7f345241ec126ccb4822306efa117bf638c9cb7238160" exitCode=60
Oct 30 13:15:07.709215 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:07.709173    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889435-km6bf" event={"ID":"cbd3c692-16b9-44ea-98a9-1aa8852f7100","Type":"ContainerDied","Data":"0a2c43ece3593a9cc3d7f345241ec126ccb4822306efa117bf638c9cb7238160"}
Oct 30 13:15:07.710652 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:07.709648    4364 scope.go:122] "RemoveContainer" containerID="0a2c43ece3593a9cc3d7f345241ec126ccb4822306efa117bf638c9cb7238160"
Oct 30 13:15:07.710652 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:15:07.709953    4364 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889435-km6bf_odh-ai-gateway-infra(cbd3c692-16b9-44ea-98a9-1aa8852f7100)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889435-km6bf" podUID="cbd3c692-16b9-44ea-98a9-1aa8852f7100"
Oct 30 13:15:07.712451 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:07.712430    4364 generic.go:356] "Generic (PLEG): container finished" podID="5b7b85af-fbe9-4a34-b0d6-16068595c558" containerID="fdde710636496e08450681aea7334adba2dcbcc4e49ad61c83335c21ba8c7c2c" exitCode=60
Oct 30 13:15:07.712521 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:07.712500    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-8b77b0-29889435-5cb9r" event={"ID":"5b7b85af-fbe9-4a34-b0d6-16068595c558","Type":"ContainerDied","Data":"fdde710636496e08450681aea7334adba2dcbcc4e49ad61c83335c21ba8c7c2c"}
Oct 30 13:15:07.712793 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:07.712770    4364 scope.go:122] "RemoveContainer" containerID="fdde710636496e08450681aea7334adba2dcbcc4e49ad61c83335c21ba8c7c2c"
Oct 30 13:15:07.713073 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:15:07.713053    4364 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-8b77b0-29889435-5cb9r_odh-ai-gateway-infra(5b7b85af-fbe9-4a34-b0d6-16068595c558)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-8b77b0-29889435-5cb9r" podUID="5b7b85af-fbe9-4a34-b0d6-16068595c558"
Oct 30 13:15:07.714540 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:07.714517    4364 generic.go:356] "Generic (PLEG): container finished" podID="d75b5256-b21b-47d6-8acb-45cb0afe1190" containerID="59213a626651e495ec97d13088b697667f2308db10667f55c68e157ac23e7b04" exitCode=60
Oct 30 13:15:07.714677 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:07.714613    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889435-c6pcv" event={"ID":"d75b5256-b21b-47d6-8acb-45cb0afe1190","Type":"ContainerDied","Data":"59213a626651e495ec97d13088b697667f2308db10667f55c68e157ac23e7b04"}
Oct 30 13:15:07.714902 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:07.714883    4364 scope.go:122] "RemoveContainer" containerID="59213a626651e495ec97d13088b697667f2308db10667f55c68e157ac23e7b04"
Oct 30 13:15:07.715208 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:15:07.715189    4364 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889435-c6pcv_odh-ai-gateway-infra(d75b5256-b21b-47d6-8acb-45cb0afe1190)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889435-c6pcv" podUID="d75b5256-b21b-47d6-8acb-45cb0afe1190"
Oct 30 13:15:07.723179 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:07.723161    4364 scope.go:122] "RemoveContainer" containerID="88a606fc9b05d61618c238f1ad097014556b66e5a379683e0cec9cfab572eeae"
Oct 30 13:15:07.723566 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:07.723549    4364 scope.go:122] "RemoveContainer" containerID="92feb3f4bc8d127003e5a6567766912a96d8f3af53e3f80e3229e0d5462e1da8"
Oct 30 13:15:07.723910 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:15:07.723887    4364 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-d7e62b-29889435-2fb6n_odh-ai-gateway-infra(b2e90cd6-1c18-4edb-ae15-7caed4920fe5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d7e62b-29889435-2fb6n" podUID="b2e90cd6-1c18-4edb-ae15-7caed4920fe5"
Oct 30 13:15:07.725375 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:07.725355    4364 generic.go:356] "Generic (PLEG): container finished" podID="77ada86d-3625-4786-a01d-2c422fdf7049" containerID="0e215363a50b60c725692aec841e2fee46e1d069c1f2896907e0b01c792e94a0" exitCode=60
Oct 30 13:15:07.725465 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:07.725445    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889435-vhbwb" event={"ID":"77ada86d-3625-4786-a01d-2c422fdf7049","Type":"ContainerDied","Data":"0e215363a50b60c725692aec841e2fee46e1d069c1f2896907e0b01c792e94a0"}
Oct 30 13:15:07.725698 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:07.725680    4364 scope.go:122] "RemoveContainer" containerID="0e215363a50b60c725692aec841e2fee46e1d069c1f2896907e0b01c792e94a0"
Oct 30 13:15:07.725962 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:15:07.725943    4364 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-52a649-29889435-vhbwb_odh-ai-gateway-infra(77ada86d-3625-4786-a01d-2c422fdf7049)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889435-vhbwb" podUID="77ada86d-3625-4786-a01d-2c422fdf7049"
Oct 30 13:15:07.727537 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:07.727517    4364 generic.go:356] "Generic (PLEG): container finished" podID="98232f3c-5612-48bf-b272-439425a1fe37" containerID="50c541a0d3bbb0e2c3e06f944cc8d244bc60fc8a3c011dc26ac2dda924a18d23" exitCode=60
Oct 30 13:15:07.727651 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:07.727611    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5a6281-29889435-9sk2f" event={"ID":"98232f3c-5612-48bf-b272-439425a1fe37","Type":"ContainerDied","Data":"50c541a0d3bbb0e2c3e06f944cc8d244bc60fc8a3c011dc26ac2dda924a18d23"}
Oct 30 13:15:07.727780 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:07.727764    4364 scope.go:122] "RemoveContainer" containerID="50c541a0d3bbb0e2c3e06f944cc8d244bc60fc8a3c011dc26ac2dda924a18d23"
Oct 30 13:15:07.727995 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:15:07.727974    4364 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-5a6281-29889435-9sk2f_odh-ai-gateway-infra(98232f3c-5612-48bf-b272-439425a1fe37)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5a6281-29889435-9sk2f" podUID="98232f3c-5612-48bf-b272-439425a1fe37"
Oct 30 13:15:07.729658 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:07.729634    4364 generic.go:356] "Generic (PLEG): container finished" podID="4ec36a29-9692-4a74-8297-c73fff6ab164" containerID="96ff74f504e1e06f47dd22d5fe887e0c1a89aa88c31ccacd90022b1712a6997c" exitCode=60
Oct 30 13:15:07.729756 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:07.729738    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f0a31b-29889435-65zmv" event={"ID":"4ec36a29-9692-4a74-8297-c73fff6ab164","Type":"ContainerDied","Data":"96ff74f504e1e06f47dd22d5fe887e0c1a89aa88c31ccacd90022b1712a6997c"}
Oct 30 13:15:07.729926 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:07.729911    4364 scope.go:122] "RemoveContainer" containerID="96ff74f504e1e06f47dd22d5fe887e0c1a89aa88c31ccacd90022b1712a6997c"
Oct 30 13:15:07.730191 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:15:07.730167    4364 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-f0a31b-29889435-65zmv_odh-ai-gateway-infra(4ec36a29-9692-4a74-8297-c73fff6ab164)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f0a31b-29889435-65zmv" podUID="4ec36a29-9692-4a74-8297-c73fff6ab164"
Oct 30 13:15:07.731796 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:07.731773    4364 scope.go:122] "RemoveContainer" containerID="96a44356b20d787084159b054005ee91abae1620cb057a8a87748beee8692e84"
Oct 30 13:15:07.732083 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:15:07.732014    4364 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-cb58b8-29889435-25tf8_odh-ai-gateway-infra(1c581513-81ee-42a4-ad45-2ae012744caf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cb58b8-29889435-25tf8" podUID="1c581513-81ee-42a4-ad45-2ae012744caf"
Oct 30 13:15:07.733653 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:07.733630    4364 generic.go:356] "Generic (PLEG): container finished" podID="0187e742-2686-400f-bb53-c74b0d65b5a8" containerID="40917ca068e4a1e4e7b6836a650476e53943feb3b173f8b050ed7cd8fcff8480" exitCode=60
Oct 30 13:15:07.733745 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:07.733652    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d833bd-29889435-wkpwn" event={"ID":"0187e742-2686-400f-bb53-c74b0d65b5a8","Type":"ContainerDied","Data":"40917ca068e4a1e4e7b6836a650476e53943feb3b173f8b050ed7cd8fcff8480"}
Oct 30 13:15:07.733954 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:07.733934    4364 scope.go:122] "RemoveContainer" containerID="40917ca068e4a1e4e7b6836a650476e53943feb3b173f8b050ed7cd8fcff8480"
Oct 30 13:15:07.734233 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:15:07.734208    4364 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-d833bd-29889435-wkpwn_odh-ai-gateway-infra(0187e742-2686-400f-bb53-c74b0d65b5a8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d833bd-29889435-wkpwn" podUID="0187e742-2686-400f-bb53-c74b0d65b5a8"
Oct 30 13:15:07.736576 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:07.736554    4364 generic.go:356] "Generic (PLEG): container finished" podID="dabc5857-b065-4205-a160-48118225ee82" containerID="dd1dd833ef19ece80becc3685decd5681e3be2c7d869d9c86d05baf0f88f3c42" exitCode=60
Oct 30 13:15:07.736692 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:07.736632    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889435-kgl8r" event={"ID":"dabc5857-b065-4205-a160-48118225ee82","Type":"ContainerDied","Data":"dd1dd833ef19ece80becc3685decd5681e3be2c7d869d9c86d05baf0f88f3c42"}
Oct 30 13:15:07.736897 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:07.736880    4364 scope.go:122] "RemoveContainer" containerID="dd1dd833ef19ece80becc3685decd5681e3be2c7d869d9c86d05baf0f88f3c42"
Oct 30 13:15:07.737182 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:15:07.737160    4364 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889435-kgl8r_odh-ai-gateway-infra(dabc5857-b065-4205-a160-48118225ee82)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889435-kgl8r" podUID="dabc5857-b065-4205-a160-48118225ee82"
Oct 30 13:15:07.737797 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:07.737780    4364 scope.go:122] "RemoveContainer" containerID="4e901155e06a81fb310aaecf3e591b75c181d5b0ecbf481d5628433b0d058cb6"
Oct 30 13:15:07.753390 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:07.753369    4364 scope.go:122] "RemoveContainer" containerID="b8f98a8e318fcdb5380c1e93a1f947fee4e758db4dd773cf265d1070e2b5eb44"
Oct 30 13:15:07.769716 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:07.769697    4364 scope.go:122] "RemoveContainer" containerID="e45c456462e4d04efcf0e7d7470c8ba92dfb10cdf689abbb97c06b36db6bf01e"
Oct 30 13:15:07.822106 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:07.822043    4364 scope.go:122] "RemoveContainer" containerID="8b51d66d66dbbf29081f30eeea6dc509df73c1730d992f4a106d7251bcfe8d7e"
Oct 30 13:15:07.846664 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:07.846639    4364 scope.go:122] "RemoveContainer" containerID="1652878b5c16a1d3ec4ff5a2aea5d08b9f2f4e51258a7aa34c2e8a122a815806"
Oct 30 13:15:07.860071 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:07.860036    4364 scope.go:122] "RemoveContainer" containerID="3a21574b8077d3e82622e57d34465d4550b26d4ac645b0b912429cdd24f687e9"
Oct 30 13:15:07.876714 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:07.876691    4364 scope.go:122] "RemoveContainer" containerID="f549002fa94a2422220f0560c31fa3d6539368f9a1fea53ee7fa092a86a05a37"
Oct 30 13:15:07.976457 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:07.976383    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2daeae-5f484f5b8-wzx8f"]
Oct 30 13:15:07.977346 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:07.977324    4364 cpu_manager.go:414] "containerMap: removing container" podUID="8fac787c-599b-4969-af01-42832173ed13" containerName="payload-processing"
Oct 30 13:15:07.977346 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:07.977343    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fac787c-599b-4969-af01-42832173ed13" containerName="payload-processing"
Oct 30 13:15:07.977503 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:07.977477    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fac787c-599b-4969-af01-42832173ed13" containerName="payload-processing"
Oct 30 13:15:08.002397 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:08.002370    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2daeae-5f484f5b8-wzx8f"
Oct 30 13:15:08.018551 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:08.017795    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-2daeae\""
Oct 30 13:15:08.027300 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:08.027274    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2daeae-5f484f5b8-wzx8f"]
Oct 30 13:15:08.034585 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:08.034556    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b770e4ef-cfb0-41d8-90dc-2b95d06f4c9e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2daeae-5f484f5b8-wzx8f\" (UID: \"b770e4ef-cfb0-41d8-90dc-2b95d06f4c9e\") " pod="openshift-ingress/payload-processing-e2e-api-b-2daeae-5f484f5b8-wzx8f"
Oct 30 13:15:08.034699 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:08.034639    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8k55\" (UniqueName: \"kubernetes.io/projected/b770e4ef-cfb0-41d8-90dc-2b95d06f4c9e-kube-api-access-v8k55\") pod \"payload-processing-e2e-api-b-2daeae-5f484f5b8-wzx8f\" (UID: \"b770e4ef-cfb0-41d8-90dc-2b95d06f4c9e\") " pod="openshift-ingress/payload-processing-e2e-api-b-2daeae-5f484f5b8-wzx8f"
Oct 30 13:15:08.165170 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:08.165129    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b770e4ef-cfb0-41d8-90dc-2b95d06f4c9e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2daeae-5f484f5b8-wzx8f\" (UID: \"b770e4ef-cfb0-41d8-90dc-2b95d06f4c9e\") " pod="openshift-ingress/payload-processing-e2e-api-b-2daeae-5f484f5b8-wzx8f"
Oct 30 13:15:08.165338 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:08.165212    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8k55\" (UniqueName: \"kubernetes.io/projected/b770e4ef-cfb0-41d8-90dc-2b95d06f4c9e-kube-api-access-v8k55\") pod \"payload-processing-e2e-api-b-2daeae-5f484f5b8-wzx8f\" (UID: \"b770e4ef-cfb0-41d8-90dc-2b95d06f4c9e\") " pod="openshift-ingress/payload-processing-e2e-api-b-2daeae-5f484f5b8-wzx8f"
Oct 30 13:15:08.179668 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:08.179629    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b770e4ef-cfb0-41d8-90dc-2b95d06f4c9e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2daeae-5f484f5b8-wzx8f\" (UID: \"b770e4ef-cfb0-41d8-90dc-2b95d06f4c9e\") " pod="openshift-ingress/payload-processing-e2e-api-b-2daeae-5f484f5b8-wzx8f"
Oct 30 13:15:08.403733 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:08.384916    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8k55\" (UniqueName: \"kubernetes.io/projected/b770e4ef-cfb0-41d8-90dc-2b95d06f4c9e-kube-api-access-v8k55\") pod \"payload-processing-e2e-api-b-2daeae-5f484f5b8-wzx8f\" (UID: \"b770e4ef-cfb0-41d8-90dc-2b95d06f4c9e\") " pod="openshift-ingress/payload-processing-e2e-api-b-2daeae-5f484f5b8-wzx8f"
Oct 30 13:15:08.446629 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:08.445719    4364 patch_prober.go:29] interesting pod/e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 13:15:08.446629 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:08.445781    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n" podUID="93fa52c8-91ad-40ef-a3b0-46c740175647" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 13:15:08.638120 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:08.638086    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2daeae-5f484f5b8-wzx8f"
Oct 30 13:15:08.899220 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:08.899189    4364 scope.go:122] "RemoveContainer" containerID="0e215363a50b60c725692aec841e2fee46e1d069c1f2896907e0b01c792e94a0"
Oct 30 13:15:08.900635 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:15:08.900586    4364 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-52a649-29889435-vhbwb_odh-ai-gateway-infra(77ada86d-3625-4786-a01d-2c422fdf7049)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889435-vhbwb" podUID="77ada86d-3625-4786-a01d-2c422fdf7049"
Oct 30 13:15:08.918804 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:08.918710    4364 scope.go:122] "RemoveContainer" containerID="50c541a0d3bbb0e2c3e06f944cc8d244bc60fc8a3c011dc26ac2dda924a18d23"
Oct 30 13:15:08.921363 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:15:08.921326    4364 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-5a6281-29889435-9sk2f_odh-ai-gateway-infra(98232f3c-5612-48bf-b272-439425a1fe37)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5a6281-29889435-9sk2f" podUID="98232f3c-5612-48bf-b272-439425a1fe37"
Oct 30 13:15:08.940742 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:08.940714    4364 scope.go:122] "RemoveContainer" containerID="96ff74f504e1e06f47dd22d5fe887e0c1a89aa88c31ccacd90022b1712a6997c"
Oct 30 13:15:08.951583 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:15:08.951037    4364 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-f0a31b-29889435-65zmv_odh-ai-gateway-infra(4ec36a29-9692-4a74-8297-c73fff6ab164)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f0a31b-29889435-65zmv" podUID="4ec36a29-9692-4a74-8297-c73fff6ab164"
Oct 30 13:15:09.064839 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:09.063163    4364 scope.go:122] "RemoveContainer" containerID="40917ca068e4a1e4e7b6836a650476e53943feb3b173f8b050ed7cd8fcff8480"
Oct 30 13:15:09.064839 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:15:09.064201    4364 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-d833bd-29889435-wkpwn_odh-ai-gateway-infra(0187e742-2686-400f-bb53-c74b0d65b5a8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d833bd-29889435-wkpwn" podUID="0187e742-2686-400f-bb53-c74b0d65b5a8"
Oct 30 13:15:09.068695 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:09.065371    4364 scope.go:122] "RemoveContainer" containerID="dd1dd833ef19ece80becc3685decd5681e3be2c7d869d9c86d05baf0f88f3c42"
Oct 30 13:15:09.068695 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:09.065754    4364 scope.go:122] "RemoveContainer" containerID="b3d63cfdce1e9382be2bc3c4aadd4659cd51214760b613a4cac32dae977073db"
Oct 30 13:15:09.068695 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:15:09.066028    4364 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889435-kgl8r_odh-ai-gateway-infra(dabc5857-b065-4205-a160-48118225ee82)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889435-kgl8r" podUID="dabc5857-b065-4205-a160-48118225ee82"
Oct 30 13:15:09.068695 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:09.066641    4364 scope.go:122] "RemoveContainer" containerID="0a2c43ece3593a9cc3d7f345241ec126ccb4822306efa117bf638c9cb7238160"
Oct 30 13:15:09.068695 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:15:09.067152    4364 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889435-km6bf_odh-ai-gateway-infra(cbd3c692-16b9-44ea-98a9-1aa8852f7100)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889435-km6bf" podUID="cbd3c692-16b9-44ea-98a9-1aa8852f7100"
Oct 30 13:15:09.068695 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:15:09.066639    4364 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-659d72-29889435-zl5cx_odh-ai-gateway-infra(6dc41559-3131-4c6d-bef5-f5cae1150954)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-659d72-29889435-zl5cx" podUID="6dc41559-3131-4c6d-bef5-f5cae1150954"
Oct 30 13:15:09.068695 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:09.067384    4364 scope.go:122] "RemoveContainer" containerID="fdde710636496e08450681aea7334adba2dcbcc4e49ad61c83335c21ba8c7c2c"
Oct 30 13:15:09.068695 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:09.067696    4364 scope.go:122] "RemoveContainer" containerID="59213a626651e495ec97d13088b697667f2308db10667f55c68e157ac23e7b04"
Oct 30 13:15:09.068695 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:15:09.068110    4364 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-8b77b0-29889435-5cb9r_odh-ai-gateway-infra(5b7b85af-fbe9-4a34-b0d6-16068595c558)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-8b77b0-29889435-5cb9r" podUID="5b7b85af-fbe9-4a34-b0d6-16068595c558"
Oct 30 13:15:09.068695 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:15:09.068212    4364 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889435-c6pcv_odh-ai-gateway-infra(d75b5256-b21b-47d6-8acb-45cb0afe1190)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889435-c6pcv" podUID="d75b5256-b21b-47d6-8acb-45cb0afe1190"
Oct 30 13:15:09.189356 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:09.189271    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-df6c3661-5866c5b5c5-99krg" podUID="552839a6-adf1-4cb3-adf3-c75500d71782" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:15:09.361627 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:09.361582    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2daeae-5f484f5b8-wzx8f"]
Oct 30 13:15:09.784979 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:09.784894    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n"
Oct 30 13:15:09.785659 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:09.785633    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n"
Oct 30 13:15:09.786271 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:09.786247    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n"
Oct 30 13:15:10.086546 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:10.086456    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2daeae-5f484f5b8-wzx8f" event={"ID":"b770e4ef-cfb0-41d8-90dc-2b95d06f4c9e","Type":"ContainerStarted","Data":"9098920df2c800094b3dd393a78cd78e61e724684bab34f423da74faa0cdcf91"}
Oct 30 13:15:10.086546 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:10.086521    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2daeae-5f484f5b8-wzx8f" event={"ID":"b770e4ef-cfb0-41d8-90dc-2b95d06f4c9e","Type":"ContainerStarted","Data":"21ee457e0a243777f2662c9c86c7e6130d82d7ba8224fc65ae1075f2ebaf485a"}
Oct 30 13:15:10.086958 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:10.086557    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-2daeae-5f484f5b8-wzx8f"
Oct 30 13:15:10.132649 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:10.132024    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-2daeae-5f484f5b8-wzx8f" podStartSLOduration=3.132006257 podStartE2EDuration="3.132006257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:10.130504082 +0000 UTC m=+1409.296868731" watchObservedRunningTime="2026-10-30 13:15:10.132006257 +0000 UTC m=+1409.298370906"
Oct 30 13:15:10.199204 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:10.199155    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-df6c3661-7f764d9794-zph7s" podUID="e50284eb-016e-4bc8-bf70-b3eb2815493c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:15:11.101958 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:11.101254    4364 generic.go:356] "Generic (PLEG): container finished" podID="e50284eb-016e-4bc8-bf70-b3eb2815493c" containerID="26c3a8eac05adbbf7801fbcca634ec6809557ac4043655bd20db49bf72a42b2f" exitCode=0
Oct 30 13:15:11.101958 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:11.101467    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-df6c3661-7f764d9794-zph7s" event={"ID":"e50284eb-016e-4bc8-bf70-b3eb2815493c","Type":"ContainerDied","Data":"26c3a8eac05adbbf7801fbcca634ec6809557ac4043655bd20db49bf72a42b2f"}
Oct 30 13:15:11.349429 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:11.349399    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-df6c3661-7f764d9794-zph7s"
Oct 30 13:15:11.409272 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:11.408748    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2daeae-5b6fb8f887-22zjb"]
Oct 30 13:15:11.410456 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:11.409751    4364 cpu_manager.go:414] "containerMap: removing container" podUID="e50284eb-016e-4bc8-bf70-b3eb2815493c" containerName="payload-processing"
Oct 30 13:15:11.410456 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:11.409768    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e50284eb-016e-4bc8-bf70-b3eb2815493c" containerName="payload-processing"
Oct 30 13:15:11.410456 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:11.409921    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="e50284eb-016e-4bc8-bf70-b3eb2815493c" containerName="payload-processing"
Oct 30 13:15:11.412813 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:11.412793    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2daeae-5b6fb8f887-22zjb"
Oct 30 13:15:11.417431 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:11.417398    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-df6c3661-5866c5b5c5-99krg"
Oct 30 13:15:11.443902 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:11.443870    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-2daeae-dockercfg-4g9f9\""
Oct 30 13:15:11.481748 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:11.481713    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e50284eb-016e-4bc8-bf70-b3eb2815493c-kube-api-access-zcg6n\" (UniqueName: \"kubernetes.io/projected/e50284eb-016e-4bc8-bf70-b3eb2815493c-kube-api-access-zcg6n\") pod \"e50284eb-016e-4bc8-bf70-b3eb2815493c\" (UID: \"e50284eb-016e-4bc8-bf70-b3eb2815493c\") "
Oct 30 13:15:11.502914 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:11.499862    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e50284eb-016e-4bc8-bf70-b3eb2815493c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e50284eb-016e-4bc8-bf70-b3eb2815493c-plugins-config-volume\") pod \"e50284eb-016e-4bc8-bf70-b3eb2815493c\" (UID: \"e50284eb-016e-4bc8-bf70-b3eb2815493c\") "
Oct 30 13:15:11.519095 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:11.507202    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e50284eb-016e-4bc8-bf70-b3eb2815493c-plugins-config-volume" pod "e50284eb-016e-4bc8-bf70-b3eb2815493c" (UID: "e50284eb-016e-4bc8-bf70-b3eb2815493c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:11.520899 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:11.520859    4364 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e50284eb-016e-4bc8-bf70-b3eb2815493c-plugins-config-volume\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:15:11.555691 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:11.553359    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2daeae-5b6fb8f887-22zjb"]
Oct 30 13:15:11.566851 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:11.566814    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e50284eb-016e-4bc8-bf70-b3eb2815493c-kube-api-access-zcg6n" pod "e50284eb-016e-4bc8-bf70-b3eb2815493c" (UID: "e50284eb-016e-4bc8-bf70-b3eb2815493c"). InnerVolumeSpecName "kube-api-access-zcg6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:11.604578 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:15:11.604542    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:15:11.622696 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:11.622668    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/552839a6-adf1-4cb3-adf3-c75500d71782-kube-api-access-vbn4q\" (UniqueName: \"kubernetes.io/projected/552839a6-adf1-4cb3-adf3-c75500d71782-kube-api-access-vbn4q\") pod \"552839a6-adf1-4cb3-adf3-c75500d71782\" (UID: \"552839a6-adf1-4cb3-adf3-c75500d71782\") "
Oct 30 13:15:11.622867 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:11.622734    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/552839a6-adf1-4cb3-adf3-c75500d71782-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/552839a6-adf1-4cb3-adf3-c75500d71782-plugins-config-volume\") pod \"552839a6-adf1-4cb3-adf3-c75500d71782\" (UID: \"552839a6-adf1-4cb3-adf3-c75500d71782\") "
Oct 30 13:15:11.623208 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:11.623124    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/552839a6-adf1-4cb3-adf3-c75500d71782-plugins-config-volume" pod "552839a6-adf1-4cb3-adf3-c75500d71782" (UID: "552839a6-adf1-4cb3-adf3-c75500d71782"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:11.623470 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:11.623443    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59wrg\" (UniqueName: \"kubernetes.io/projected/dfb4ff14-8dac-4cd5-944a-2aee68792901-kube-api-access-59wrg\") pod \"payload-pre-processing-e2e-api-b-2daeae-5b6fb8f887-22zjb\" (UID: \"dfb4ff14-8dac-4cd5-944a-2aee68792901\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2daeae-5b6fb8f887-22zjb"
Oct 30 13:15:11.623670 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:11.623650    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfb4ff14-8dac-4cd5-944a-2aee68792901-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2daeae-5b6fb8f887-22zjb\" (UID: \"dfb4ff14-8dac-4cd5-944a-2aee68792901\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2daeae-5b6fb8f887-22zjb"
Oct 30 13:15:11.624012 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:11.623993    4364 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/552839a6-adf1-4cb3-adf3-c75500d71782-plugins-config-volume\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:15:11.624059 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:11.624019    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcg6n\" (UniqueName: \"kubernetes.io/projected/e50284eb-016e-4bc8-bf70-b3eb2815493c-kube-api-access-zcg6n\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:15:11.625063 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:11.625042    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/552839a6-adf1-4cb3-adf3-c75500d71782-kube-api-access-vbn4q" pod "552839a6-adf1-4cb3-adf3-c75500d71782" (UID: "552839a6-adf1-4cb3-adf3-c75500d71782"). InnerVolumeSpecName "kube-api-access-vbn4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:11.725114 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:11.725085    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59wrg\" (UniqueName: \"kubernetes.io/projected/dfb4ff14-8dac-4cd5-944a-2aee68792901-kube-api-access-59wrg\") pod \"payload-pre-processing-e2e-api-b-2daeae-5b6fb8f887-22zjb\" (UID: \"dfb4ff14-8dac-4cd5-944a-2aee68792901\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2daeae-5b6fb8f887-22zjb"
Oct 30 13:15:11.725240 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:11.725162    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfb4ff14-8dac-4cd5-944a-2aee68792901-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2daeae-5b6fb8f887-22zjb\" (UID: \"dfb4ff14-8dac-4cd5-944a-2aee68792901\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2daeae-5b6fb8f887-22zjb"
Oct 30 13:15:11.725410 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:11.725394    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbn4q\" (UniqueName: \"kubernetes.io/projected/552839a6-adf1-4cb3-adf3-c75500d71782-kube-api-access-vbn4q\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:15:11.725880 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:11.725860    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfb4ff14-8dac-4cd5-944a-2aee68792901-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2daeae-5b6fb8f887-22zjb\" (UID: \"dfb4ff14-8dac-4cd5-944a-2aee68792901\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2daeae-5b6fb8f887-22zjb"
Oct 30 13:15:11.748677 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:11.748654    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59wrg\" (UniqueName: \"kubernetes.io/projected/dfb4ff14-8dac-4cd5-944a-2aee68792901-kube-api-access-59wrg\") pod \"payload-pre-processing-e2e-api-b-2daeae-5b6fb8f887-22zjb\" (UID: \"dfb4ff14-8dac-4cd5-944a-2aee68792901\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2daeae-5b6fb8f887-22zjb"
Oct 30 13:15:12.041971 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:12.041947    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2daeae-5b6fb8f887-22zjb"
Oct 30 13:15:12.118613 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:12.118574    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-df6c3661-7f764d9794-zph7s" event={"ID":"e50284eb-016e-4bc8-bf70-b3eb2815493c","Type":"ContainerDied","Data":"628debe20ed039a2e79295a7e0aca0e80c2d0eb367c12abd8afe01dfd7f4c41d"}
Oct 30 13:15:12.118999 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:12.118681    4364 scope.go:122] "RemoveContainer" containerID="26c3a8eac05adbbf7801fbcca634ec6809557ac4043655bd20db49bf72a42b2f"
Oct 30 13:15:12.118999 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:12.118861    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-df6c3661-7f764d9794-zph7s"
Oct 30 13:15:12.132266 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:12.132243    4364 generic.go:356] "Generic (PLEG): container finished" podID="552839a6-adf1-4cb3-adf3-c75500d71782" containerID="16fa7ae7eec2b13d0d6406c1e7bb31b05664dfc6c45ca82fafc3e99fe6a4bc7d" exitCode=0
Oct 30 13:15:12.132395 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:12.132383    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-df6c3661-5866c5b5c5-99krg" event={"ID":"552839a6-adf1-4cb3-adf3-c75500d71782","Type":"ContainerDied","Data":"16fa7ae7eec2b13d0d6406c1e7bb31b05664dfc6c45ca82fafc3e99fe6a4bc7d"}
Oct 30 13:15:12.132438 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:12.132417    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-df6c3661-5866c5b5c5-99krg" event={"ID":"552839a6-adf1-4cb3-adf3-c75500d71782","Type":"ContainerDied","Data":"fddcc9d7bf951892421050e2290f1fb84c65e1f47c80bc32a7c38c370638160a"}
Oct 30 13:15:12.132479 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:12.132465    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-df6c3661-5866c5b5c5-99krg"
Oct 30 13:15:12.158894 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:12.158866    4364 scope.go:122] "RemoveContainer" containerID="16fa7ae7eec2b13d0d6406c1e7bb31b05664dfc6c45ca82fafc3e99fe6a4bc7d"
Oct 30 13:15:12.171015 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:12.170991    4364 scope.go:122] "RemoveContainer" containerID="16fa7ae7eec2b13d0d6406c1e7bb31b05664dfc6c45ca82fafc3e99fe6a4bc7d"
Oct 30 13:15:12.171275 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:15:12.171252    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16fa7ae7eec2b13d0d6406c1e7bb31b05664dfc6c45ca82fafc3e99fe6a4bc7d\": container with ID starting with 16fa7ae7eec2b13d0d6406c1e7bb31b05664dfc6c45ca82fafc3e99fe6a4bc7d not found: ID does not exist" containerID="16fa7ae7eec2b13d0d6406c1e7bb31b05664dfc6c45ca82fafc3e99fe6a4bc7d"
Oct 30 13:15:12.171360 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:12.171286    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16fa7ae7eec2b13d0d6406c1e7bb31b05664dfc6c45ca82fafc3e99fe6a4bc7d"} err="failed to get container status \"16fa7ae7eec2b13d0d6406c1e7bb31b05664dfc6c45ca82fafc3e99fe6a4bc7d\": rpc error: code = NotFound desc = could not find container \"16fa7ae7eec2b13d0d6406c1e7bb31b05664dfc6c45ca82fafc3e99fe6a4bc7d\": container with ID starting with 16fa7ae7eec2b13d0d6406c1e7bb31b05664dfc6c45ca82fafc3e99fe6a4bc7d not found: ID does not exist"
Oct 30 13:15:12.183642 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:12.183584    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-df6c3661-7f764d9794-zph7s"]
Oct 30 13:15:12.201672 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:12.201640    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-df6c3661-7f764d9794-zph7s"]
Oct 30 13:15:12.248840 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:12.248808    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-df6c3661-5866c5b5c5-99krg"]
Oct 30 13:15:12.258253 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:12.258176    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-df6c3661-5866c5b5c5-99krg"]
Oct 30 13:15:12.359113 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:15:12.359080    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfb4ff14_8dac_4cd5_944a_2aee68792901.slice/crio-5b7182cc0b2934e9c9f98c61873f61e6e7d164685647616c2f4e831a40759deb WatchSource:0}: Error finding container 5b7182cc0b2934e9c9f98c61873f61e6e7d164685647616c2f4e831a40759deb: Status 404 returned error can't find the container with id 5b7182cc0b2934e9c9f98c61873f61e6e7d164685647616c2f4e831a40759deb
Oct 30 13:15:12.362009 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:12.361982    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2daeae-5b6fb8f887-22zjb"]
Oct 30 13:15:12.973229 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:12.973197    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="552839a6-adf1-4cb3-adf3-c75500d71782" path="/var/lib/kubelet/pods/552839a6-adf1-4cb3-adf3-c75500d71782/volumes"
Oct 30 13:15:12.973667 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:12.973652    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e50284eb-016e-4bc8-bf70-b3eb2815493c" path="/var/lib/kubelet/pods/e50284eb-016e-4bc8-bf70-b3eb2815493c/volumes"
Oct 30 13:15:13.143991 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:13.143958    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2daeae-5b6fb8f887-22zjb" event={"ID":"dfb4ff14-8dac-4cd5-944a-2aee68792901","Type":"ContainerStarted","Data":"fd76b622842d35ed4b232c19a127cc56d00de9a5038cbd5d744442ac4e2032d8"}
Oct 30 13:15:13.144422 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:13.144029    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2daeae-5b6fb8f887-22zjb"
Oct 30 13:15:13.144422 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:13.144054    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2daeae-5b6fb8f887-22zjb" event={"ID":"dfb4ff14-8dac-4cd5-944a-2aee68792901","Type":"ContainerStarted","Data":"5b7182cc0b2934e9c9f98c61873f61e6e7d164685647616c2f4e831a40759deb"}
Oct 30 13:15:13.146779 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:13.146758    4364 generic.go:356] "Generic (PLEG): container finished" podID="cd6e6964-3e54-48f1-a184-8db6ef093a7b" containerID="5bcc210571b46ecb21eae031ebf90fca908e5b4d1d757005c3eec4e48ba9b6be" exitCode=0
Oct 30 13:15:13.146883 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:13.146841    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx" event={"ID":"cd6e6964-3e54-48f1-a184-8db6ef093a7b","Type":"ContainerDied","Data":"5bcc210571b46ecb21eae031ebf90fca908e5b4d1d757005c3eec4e48ba9b6be"}
Oct 30 13:15:13.147235 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:13.147221    4364 scope.go:122] "RemoveContainer" containerID="5bcc210571b46ecb21eae031ebf90fca908e5b4d1d757005c3eec4e48ba9b6be"
Oct 30 13:15:13.148928 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:13.148902    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:15:13.148989 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:13.148966    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:15:13.149024 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:13.148998    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:15:13.186497 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:13.186447    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2daeae-5b6fb8f887-22zjb" podStartSLOduration=5.186434928 podStartE2EDuration="5.186434928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:13.184185858 +0000 UTC m=+1412.350550504" watchObservedRunningTime="2026-10-30 13:15:13.186434928 +0000 UTC m=+1412.352799576"
Oct 30 13:15:14.159824 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:14.159790    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx" event={"ID":"cd6e6964-3e54-48f1-a184-8db6ef093a7b","Type":"ContainerStarted","Data":"841e79331eff502018036c927e124e95de3674c0570c3c1be13453a228aaf208"}
Oct 30 13:15:14.828527 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:14.828490    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx"
Oct 30 13:15:14.828769 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:14.828647    4364 patch_prober.go:29] interesting pod/e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Oct 30 13:15:14.828769 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:14.828690    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx" podUID="cd6e6964-3e54-48f1-a184-8db6ef093a7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Oct 30 13:15:14.913400 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:14.913368    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-774b5079-rnk2n"]
Oct 30 13:15:14.914384 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:14.914366    4364 cpu_manager.go:414] "containerMap: removing container" podUID="552839a6-adf1-4cb3-adf3-c75500d71782" containerName="payload-pre-processing"
Oct 30 13:15:14.914447 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:14.914385    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="552839a6-adf1-4cb3-adf3-c75500d71782" containerName="payload-pre-processing"
Oct 30 13:15:14.914548 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:14.914538    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="552839a6-adf1-4cb3-adf3-c75500d71782" containerName="payload-pre-processing"
Oct 30 13:15:14.917707 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:14.917691    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-774b5079-rnk2n"
Oct 30 13:15:14.939856 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:14.939831    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-774b5079-rnk2n"]
Oct 30 13:15:14.944591 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:14.944536    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:15:15.005931 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:15.005898    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ada2dea-42f8-4811-8109-87b14971d203-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-774b5079-rnk2n\" (UID: \"2ada2dea-42f8-4811-8109-87b14971d203\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-774b5079-rnk2n"
Oct 30 13:15:15.006198 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:15.006177    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ada2dea-42f8-4811-8109-87b14971d203-maas-api-xks-ca\") pod \"maas-api-revoke-sub-774b5079-rnk2n\" (UID: \"2ada2dea-42f8-4811-8109-87b14971d203\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-774b5079-rnk2n"
Oct 30 13:15:15.006323 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:15.006292    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfskt\" (UniqueName: \"kubernetes.io/projected/2ada2dea-42f8-4811-8109-87b14971d203-kube-api-access-tfskt\") pod \"maas-api-revoke-sub-774b5079-rnk2n\" (UID: \"2ada2dea-42f8-4811-8109-87b14971d203\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-774b5079-rnk2n"
Oct 30 13:15:15.108894 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:15.108799    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ada2dea-42f8-4811-8109-87b14971d203-maas-api-xks-ca\") pod \"maas-api-revoke-sub-774b5079-rnk2n\" (UID: \"2ada2dea-42f8-4811-8109-87b14971d203\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-774b5079-rnk2n"
Oct 30 13:15:15.109068 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:15.108918    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfskt\" (UniqueName: \"kubernetes.io/projected/2ada2dea-42f8-4811-8109-87b14971d203-kube-api-access-tfskt\") pod \"maas-api-revoke-sub-774b5079-rnk2n\" (UID: \"2ada2dea-42f8-4811-8109-87b14971d203\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-774b5079-rnk2n"
Oct 30 13:15:15.109068 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:15.109039    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ada2dea-42f8-4811-8109-87b14971d203-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-774b5079-rnk2n\" (UID: \"2ada2dea-42f8-4811-8109-87b14971d203\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-774b5079-rnk2n"
Oct 30 13:15:15.110021 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:15.109997    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ada2dea-42f8-4811-8109-87b14971d203-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-774b5079-rnk2n\" (UID: \"2ada2dea-42f8-4811-8109-87b14971d203\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-774b5079-rnk2n"
Oct 30 13:15:15.120763 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:15.120736    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ada2dea-42f8-4811-8109-87b14971d203-maas-api-xks-ca\") pod \"maas-api-revoke-sub-774b5079-rnk2n\" (UID: \"2ada2dea-42f8-4811-8109-87b14971d203\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-774b5079-rnk2n"
Oct 30 13:15:15.135837 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:15.135817    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfskt\" (UniqueName: \"kubernetes.io/projected/2ada2dea-42f8-4811-8109-87b14971d203-kube-api-access-tfskt\") pod \"maas-api-revoke-sub-774b5079-rnk2n\" (UID: \"2ada2dea-42f8-4811-8109-87b14971d203\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-774b5079-rnk2n"
Oct 30 13:15:15.231525 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:15.231497    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-774b5079-rnk2n"
Oct 30 13:15:15.826446 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:15:15.826415    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ada2dea_42f8_4811_8109_87b14971d203.slice/crio-dfb40850212f8e3a103ce4829dc342772f601701c838ec446183838b853cb91a WatchSource:0}: Error finding container dfb40850212f8e3a103ce4829dc342772f601701c838ec446183838b853cb91a: Status 404 returned error can't find the container with id dfb40850212f8e3a103ce4829dc342772f601701c838ec446183838b853cb91a
Oct 30 13:15:15.827901 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:15.827812    4364 patch_prober.go:29] interesting pod/e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Oct 30 13:15:15.827901 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:15.827874    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx" podUID="cd6e6964-3e54-48f1-a184-8db6ef093a7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Oct 30 13:15:15.828457 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:15.828435    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-774b5079-rnk2n"]
Oct 30 13:15:16.205495 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:16.205451    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-774b5079-rnk2n" event={"ID":"2ada2dea-42f8-4811-8109-87b14971d203","Type":"ContainerStarted","Data":"dfb40850212f8e3a103ce4829dc342772f601701c838ec446183838b853cb91a"}
Oct 30 13:15:16.837625 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:16.835209    4364 patch_prober.go:29] interesting pod/e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Oct 30 13:15:16.837625 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:16.835267    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx" podUID="cd6e6964-3e54-48f1-a184-8db6ef093a7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Oct 30 13:15:16.962652 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:16.962593    4364 scope.go:122] "RemoveContainer" containerID="92feb3f4bc8d127003e5a6567766912a96d8f3af53e3f80e3229e0d5462e1da8"
Oct 30 13:15:16.962809 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:16.962742    4364 scope.go:122] "RemoveContainer" containerID="96a44356b20d787084159b054005ee91abae1620cb057a8a87748beee8692e84"
Oct 30 13:15:17.217723 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:17.217696    4364 generic.go:356] "Generic (PLEG): container finished" podID="2ada2dea-42f8-4811-8109-87b14971d203" containerID="ed885b2a549c22047b0e1ed8a282fcdc976343bf43bfea0834e24c9dc1c435fc" exitCode=0
Oct 30 13:15:17.217877 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:17.217775    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-774b5079-rnk2n" event={"ID":"2ada2dea-42f8-4811-8109-87b14971d203","Type":"ContainerDied","Data":"ed885b2a549c22047b0e1ed8a282fcdc976343bf43bfea0834e24c9dc1c435fc"}
Oct 30 13:15:17.828530 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:17.828443    4364 patch_prober.go:29] interesting pod/e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Oct 30 13:15:17.828530 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:17.828503    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx" podUID="cd6e6964-3e54-48f1-a184-8db6ef093a7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Oct 30 13:15:17.963748 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:17.963715    4364 scope.go:122] "RemoveContainer" containerID="b3d63cfdce1e9382be2bc3c4aadd4659cd51214760b613a4cac32dae977073db"
Oct 30 13:15:17.964112 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:17.963818    4364 scope.go:122] "RemoveContainer" containerID="0e215363a50b60c725692aec841e2fee46e1d069c1f2896907e0b01c792e94a0"
Oct 30 13:15:17.964254 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:17.964226    4364 scope.go:122] "RemoveContainer" containerID="dd1dd833ef19ece80becc3685decd5681e3be2c7d869d9c86d05baf0f88f3c42"
Oct 30 13:15:17.964410 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:17.964302    4364 scope.go:122] "RemoveContainer" containerID="40917ca068e4a1e4e7b6836a650476e53943feb3b173f8b050ed7cd8fcff8480"
Oct 30 13:15:17.964652 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:17.964520    4364 scope.go:122] "RemoveContainer" containerID="96ff74f504e1e06f47dd22d5fe887e0c1a89aa88c31ccacd90022b1712a6997c"
Oct 30 13:15:17.965581 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:17.965563    4364 scope.go:122] "RemoveContainer" containerID="50c541a0d3bbb0e2c3e06f944cc8d244bc60fc8a3c011dc26ac2dda924a18d23"
Oct 30 13:15:18.231221 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:18.231190    4364 generic.go:356] "Generic (PLEG): container finished" podID="b2e90cd6-1c18-4edb-ae15-7caed4920fe5" containerID="70e17a839d66814e96180873cf24cc871de2ef29de91cd9ed9ec63c1dd828a08" exitCode=60
Oct 30 13:15:18.231357 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:18.231257    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d7e62b-29889435-2fb6n" event={"ID":"b2e90cd6-1c18-4edb-ae15-7caed4920fe5","Type":"ContainerDied","Data":"70e17a839d66814e96180873cf24cc871de2ef29de91cd9ed9ec63c1dd828a08"}
Oct 30 13:15:18.231357 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:18.231339    4364 scope.go:122] "RemoveContainer" containerID="92feb3f4bc8d127003e5a6567766912a96d8f3af53e3f80e3229e0d5462e1da8"
Oct 30 13:15:18.231819 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:18.231799    4364 scope.go:122] "RemoveContainer" containerID="70e17a839d66814e96180873cf24cc871de2ef29de91cd9ed9ec63c1dd828a08"
Oct 30 13:15:18.232232 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:15:18.232208    4364 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-d7e62b-29889435-2fb6n_odh-ai-gateway-infra(b2e90cd6-1c18-4edb-ae15-7caed4920fe5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d7e62b-29889435-2fb6n" podUID="b2e90cd6-1c18-4edb-ae15-7caed4920fe5"
Oct 30 13:15:18.233833 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:18.233813    4364 generic.go:356] "Generic (PLEG): container finished" podID="1c581513-81ee-42a4-ad45-2ae012744caf" containerID="32efcb9a52693c0232b45529ba89df050032a175c895fd86ddb73a77acd01018" exitCode=60
Oct 30 13:15:18.233927 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:18.233871    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cb58b8-29889435-25tf8" event={"ID":"1c581513-81ee-42a4-ad45-2ae012744caf","Type":"ContainerDied","Data":"32efcb9a52693c0232b45529ba89df050032a175c895fd86ddb73a77acd01018"}
Oct 30 13:15:18.234315 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:18.234301    4364 scope.go:122] "RemoveContainer" containerID="32efcb9a52693c0232b45529ba89df050032a175c895fd86ddb73a77acd01018"
Oct 30 13:15:18.234689 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:15:18.234665    4364 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-cb58b8-29889435-25tf8_odh-ai-gateway-infra(1c581513-81ee-42a4-ad45-2ae012744caf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cb58b8-29889435-25tf8" podUID="1c581513-81ee-42a4-ad45-2ae012744caf"
Oct 30 13:15:18.248240 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:18.248219    4364 scope.go:122] "RemoveContainer" containerID="96a44356b20d787084159b054005ee91abae1620cb057a8a87748beee8692e84"
Oct 30 13:15:18.473278 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:18.473258    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-774b5079-rnk2n"
Oct 30 13:15:18.505918 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:18.497129    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ada2dea-42f8-4811-8109-87b14971d203-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ada2dea-42f8-4811-8109-87b14971d203-maas-api-xks-ca\") pod \"2ada2dea-42f8-4811-8109-87b14971d203\" (UID: \"2ada2dea-42f8-4811-8109-87b14971d203\") "
Oct 30 13:15:18.505918 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:18.497335    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ada2dea-42f8-4811-8109-87b14971d203-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ada2dea-42f8-4811-8109-87b14971d203-maas-api-openshift-ca\") pod \"2ada2dea-42f8-4811-8109-87b14971d203\" (UID: \"2ada2dea-42f8-4811-8109-87b14971d203\") "
Oct 30 13:15:18.505918 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:18.497677    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ada2dea-42f8-4811-8109-87b14971d203-kube-api-access-tfskt\" (UniqueName: \"kubernetes.io/projected/2ada2dea-42f8-4811-8109-87b14971d203-kube-api-access-tfskt\") pod \"2ada2dea-42f8-4811-8109-87b14971d203\" (UID: \"2ada2dea-42f8-4811-8109-87b14971d203\") "
Oct 30 13:15:18.505918 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:18.500250    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ada2dea-42f8-4811-8109-87b14971d203-maas-api-openshift-ca" pod "2ada2dea-42f8-4811-8109-87b14971d203" (UID: "2ada2dea-42f8-4811-8109-87b14971d203"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:18.509444 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:18.509411    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ada2dea-42f8-4811-8109-87b14971d203-kube-api-access-tfskt" pod "2ada2dea-42f8-4811-8109-87b14971d203" (UID: "2ada2dea-42f8-4811-8109-87b14971d203"). InnerVolumeSpecName "kube-api-access-tfskt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:18.510468 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:18.510444    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ada2dea-42f8-4811-8109-87b14971d203-maas-api-xks-ca" pod "2ada2dea-42f8-4811-8109-87b14971d203" (UID: "2ada2dea-42f8-4811-8109-87b14971d203"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:15:18.610646 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:18.610091    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ada2dea-42f8-4811-8109-87b14971d203-maas-api-openshift-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:15:18.610646 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:18.610213    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfskt\" (UniqueName: \"kubernetes.io/projected/2ada2dea-42f8-4811-8109-87b14971d203-kube-api-access-tfskt\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:15:18.610646 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:18.610233    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ada2dea-42f8-4811-8109-87b14971d203-maas-api-xks-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:15:18.828127 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:18.828099    4364 patch_prober.go:29] interesting pod/e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Oct 30 13:15:18.828235 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:18.828154    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx" podUID="cd6e6964-3e54-48f1-a184-8db6ef093a7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Oct 30 13:15:18.963031 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:18.962953    4364 scope.go:122] "RemoveContainer" containerID="0a2c43ece3593a9cc3d7f345241ec126ccb4822306efa117bf638c9cb7238160"
Oct 30 13:15:18.963300 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:18.963267    4364 scope.go:122] "RemoveContainer" containerID="fdde710636496e08450681aea7334adba2dcbcc4e49ad61c83335c21ba8c7c2c"
Oct 30 13:15:19.251949 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:19.251918    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889435-vhbwb" event={"ID":"77ada86d-3625-4786-a01d-2c422fdf7049","Type":"ContainerStarted","Data":"13c06286d932f956d386ba7383375dbc27fb0013855758f49c7dab31db8fd469"}
Oct 30 13:15:19.255558 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:19.255533    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5a6281-29889435-9sk2f" event={"ID":"98232f3c-5612-48bf-b272-439425a1fe37","Type":"ContainerStarted","Data":"1afec301abd694af74bdbe2251c47260bcc3b7f6e907d37df96be4b25821a23d"}
Oct 30 13:15:19.258080 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:19.258055    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f0a31b-29889435-65zmv" event={"ID":"4ec36a29-9692-4a74-8297-c73fff6ab164","Type":"ContainerStarted","Data":"4923bbb89b9f1e2289b77b86e672c5226d46bafbab65df1119d894d9fcddcb2b"}
Oct 30 13:15:19.260750 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:19.260728    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-774b5079-rnk2n" event={"ID":"2ada2dea-42f8-4811-8109-87b14971d203","Type":"ContainerDied","Data":"dfb40850212f8e3a103ce4829dc342772f601701c838ec446183838b853cb91a"}
Oct 30 13:15:19.260822 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:19.260764    4364 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dfb40850212f8e3a103ce4829dc342772f601701c838ec446183838b853cb91a"
Oct 30 13:15:19.260822 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:19.260806    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-774b5079-rnk2n"
Oct 30 13:15:19.263201 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:19.263178    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d833bd-29889435-wkpwn" event={"ID":"0187e742-2686-400f-bb53-c74b0d65b5a8","Type":"ContainerStarted","Data":"d4139d97029671ae03c6f0c173dc2fcd6929a7778cb8857d33da731df8f8db0f"}
Oct 30 13:15:19.265309 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:19.265287    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889435-kgl8r" event={"ID":"dabc5857-b065-4205-a160-48118225ee82","Type":"ContainerStarted","Data":"b940753f8f10b447075db1564ad2acaeca683deda842a4a3ff5ef24ff3f3dda3"}
Oct 30 13:15:19.267215 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:19.267194    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-659d72-29889435-zl5cx" event={"ID":"6dc41559-3131-4c6d-bef5-f5cae1150954","Type":"ContainerStarted","Data":"501ec58cf77dd51b8bf0728b99f207e20aa70bc60f7fe8aaa55e407f8046f2ab"}
Oct 30 13:15:19.333729 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:19.333679    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d7e62b-29889435-2fb6n"]
Oct 30 13:15:19.359132 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:19.359101    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cb58b8-29889435-25tf8"]
Oct 30 13:15:19.830723 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:19.830689    4364 patch_prober.go:29] interesting pod/e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Oct 30 13:15:19.830860 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:19.830801    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx" podUID="cd6e6964-3e54-48f1-a184-8db6ef093a7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Oct 30 13:15:19.836642 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:19.836310    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d7e62b-29889435-2fb6n"
Oct 30 13:15:19.913288 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:19.913231    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2e90cd6-1c18-4edb-ae15-7caed4920fe5-kube-api-access-zdz42\" (UniqueName: \"kubernetes.io/projected/b2e90cd6-1c18-4edb-ae15-7caed4920fe5-kube-api-access-zdz42\") pod \"b2e90cd6-1c18-4edb-ae15-7caed4920fe5\" (UID: \"b2e90cd6-1c18-4edb-ae15-7caed4920fe5\") "
Oct 30 13:15:19.913520 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:19.913496    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2e90cd6-1c18-4edb-ae15-7caed4920fe5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2e90cd6-1c18-4edb-ae15-7caed4920fe5-maas-api-service-ca\") pod \"b2e90cd6-1c18-4edb-ae15-7caed4920fe5\" (UID: \"b2e90cd6-1c18-4edb-ae15-7caed4920fe5\") "
Oct 30 13:15:19.914910 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:19.914864    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2e90cd6-1c18-4edb-ae15-7caed4920fe5-maas-api-service-ca" pod "b2e90cd6-1c18-4edb-ae15-7caed4920fe5" (UID: "b2e90cd6-1c18-4edb-ae15-7caed4920fe5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:19.919068 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:19.919023    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2e90cd6-1c18-4edb-ae15-7caed4920fe5-kube-api-access-zdz42" pod "b2e90cd6-1c18-4edb-ae15-7caed4920fe5" (UID: "b2e90cd6-1c18-4edb-ae15-7caed4920fe5"). InnerVolumeSpecName "kube-api-access-zdz42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:19.963086 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:19.963055    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cb58b8-29889435-25tf8"
Oct 30 13:15:20.016141 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:20.016100    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c581513-81ee-42a4-ad45-2ae012744caf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c581513-81ee-42a4-ad45-2ae012744caf-maas-api-service-ca\") pod \"1c581513-81ee-42a4-ad45-2ae012744caf\" (UID: \"1c581513-81ee-42a4-ad45-2ae012744caf\") "
Oct 30 13:15:20.016371 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:20.016350    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c581513-81ee-42a4-ad45-2ae012744caf-kube-api-access-9ngv5\" (UniqueName: \"kubernetes.io/projected/1c581513-81ee-42a4-ad45-2ae012744caf-kube-api-access-9ngv5\") pod \"1c581513-81ee-42a4-ad45-2ae012744caf\" (UID: \"1c581513-81ee-42a4-ad45-2ae012744caf\") "
Oct 30 13:15:20.016519 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:20.016493    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c581513-81ee-42a4-ad45-2ae012744caf-maas-api-service-ca" pod "1c581513-81ee-42a4-ad45-2ae012744caf" (UID: "1c581513-81ee-42a4-ad45-2ae012744caf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:20.017458 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:20.017434    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdz42\" (UniqueName: \"kubernetes.io/projected/b2e90cd6-1c18-4edb-ae15-7caed4920fe5-kube-api-access-zdz42\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:15:20.017458 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:20.017461    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c581513-81ee-42a4-ad45-2ae012744caf-maas-api-service-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:15:20.017684 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:20.017476    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2e90cd6-1c18-4edb-ae15-7caed4920fe5-maas-api-service-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:15:20.019369 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:20.019336    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c581513-81ee-42a4-ad45-2ae012744caf-kube-api-access-9ngv5" pod "1c581513-81ee-42a4-ad45-2ae012744caf" (UID: "1c581513-81ee-42a4-ad45-2ae012744caf"). InnerVolumeSpecName "kube-api-access-9ngv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:20.119212 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:20.119187    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ngv5\" (UniqueName: \"kubernetes.io/projected/1c581513-81ee-42a4-ad45-2ae012744caf-kube-api-access-9ngv5\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:15:20.277729 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:20.277699    4364 generic.go:356] "Generic (PLEG): container finished" podID="98232f3c-5612-48bf-b272-439425a1fe37" containerID="1afec301abd694af74bdbe2251c47260bcc3b7f6e907d37df96be4b25821a23d" exitCode=60
Oct 30 13:15:20.278125 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:20.277785    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5a6281-29889435-9sk2f" event={"ID":"98232f3c-5612-48bf-b272-439425a1fe37","Type":"ContainerDied","Data":"1afec301abd694af74bdbe2251c47260bcc3b7f6e907d37df96be4b25821a23d"}
Oct 30 13:15:20.278125 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:20.277841    4364 scope.go:122] "RemoveContainer" containerID="50c541a0d3bbb0e2c3e06f944cc8d244bc60fc8a3c011dc26ac2dda924a18d23"
Oct 30 13:15:20.278257 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:20.278217    4364 scope.go:122] "RemoveContainer" containerID="1afec301abd694af74bdbe2251c47260bcc3b7f6e907d37df96be4b25821a23d"
Oct 30 13:15:20.279009 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:15:20.278692    4364 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-5a6281-29889435-9sk2f_odh-ai-gateway-infra(98232f3c-5612-48bf-b272-439425a1fe37)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5a6281-29889435-9sk2f" podUID="98232f3c-5612-48bf-b272-439425a1fe37"
Oct 30 13:15:20.280329 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:20.280306    4364 generic.go:356] "Generic (PLEG): container finished" podID="4ec36a29-9692-4a74-8297-c73fff6ab164" containerID="4923bbb89b9f1e2289b77b86e672c5226d46bafbab65df1119d894d9fcddcb2b" exitCode=60
Oct 30 13:15:20.280444 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:20.280377    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f0a31b-29889435-65zmv" event={"ID":"4ec36a29-9692-4a74-8297-c73fff6ab164","Type":"ContainerDied","Data":"4923bbb89b9f1e2289b77b86e672c5226d46bafbab65df1119d894d9fcddcb2b"}
Oct 30 13:15:20.280789 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:20.280772    4364 scope.go:122] "RemoveContainer" containerID="4923bbb89b9f1e2289b77b86e672c5226d46bafbab65df1119d894d9fcddcb2b"
Oct 30 13:15:20.281229 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:15:20.281207    4364 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-f0a31b-29889435-65zmv_odh-ai-gateway-infra(4ec36a29-9692-4a74-8297-c73fff6ab164)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f0a31b-29889435-65zmv" podUID="4ec36a29-9692-4a74-8297-c73fff6ab164"
Oct 30 13:15:20.281962 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:20.281936    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cb58b8-29889435-25tf8"
Oct 30 13:15:20.282061 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:20.281980    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cb58b8-29889435-25tf8" event={"ID":"1c581513-81ee-42a4-ad45-2ae012744caf","Type":"ContainerDied","Data":"0566ed2317f37da33d7b3b3e101560f9aa1ac3405bc15ecbe83e89e7025c0b29"}
Oct 30 13:15:20.284236 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:20.284213    4364 generic.go:356] "Generic (PLEG): container finished" podID="0187e742-2686-400f-bb53-c74b0d65b5a8" containerID="d4139d97029671ae03c6f0c173dc2fcd6929a7778cb8857d33da731df8f8db0f" exitCode=60
Oct 30 13:15:20.284324 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:20.284287    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d833bd-29889435-wkpwn" event={"ID":"0187e742-2686-400f-bb53-c74b0d65b5a8","Type":"ContainerDied","Data":"d4139d97029671ae03c6f0c173dc2fcd6929a7778cb8857d33da731df8f8db0f"}
Oct 30 13:15:20.284697 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:20.284678    4364 scope.go:122] "RemoveContainer" containerID="d4139d97029671ae03c6f0c173dc2fcd6929a7778cb8857d33da731df8f8db0f"
Oct 30 13:15:20.285040 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:15:20.285016    4364 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-d833bd-29889435-wkpwn_odh-ai-gateway-infra(0187e742-2686-400f-bb53-c74b0d65b5a8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d833bd-29889435-wkpwn" podUID="0187e742-2686-400f-bb53-c74b0d65b5a8"
Oct 30 13:15:20.286471 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:20.286452    4364 generic.go:356] "Generic (PLEG): container finished" podID="dabc5857-b065-4205-a160-48118225ee82" containerID="b940753f8f10b447075db1564ad2acaeca683deda842a4a3ff5ef24ff3f3dda3" exitCode=60
Oct 30 13:15:20.286569 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:20.286522    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889435-kgl8r" event={"ID":"dabc5857-b065-4205-a160-48118225ee82","Type":"ContainerDied","Data":"b940753f8f10b447075db1564ad2acaeca683deda842a4a3ff5ef24ff3f3dda3"}
Oct 30 13:15:20.286986 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:20.286900    4364 scope.go:122] "RemoveContainer" containerID="b940753f8f10b447075db1564ad2acaeca683deda842a4a3ff5ef24ff3f3dda3"
Oct 30 13:15:20.287266 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:15:20.287241    4364 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889435-kgl8r_odh-ai-gateway-infra(dabc5857-b065-4205-a160-48118225ee82)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889435-kgl8r" podUID="dabc5857-b065-4205-a160-48118225ee82"
Oct 30 13:15:20.288718 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:20.288697    4364 generic.go:356] "Generic (PLEG): container finished" podID="6dc41559-3131-4c6d-bef5-f5cae1150954" containerID="501ec58cf77dd51b8bf0728b99f207e20aa70bc60f7fe8aaa55e407f8046f2ab" exitCode=60
Oct 30 13:15:20.288809 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:20.288772    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-659d72-29889435-zl5cx" event={"ID":"6dc41559-3131-4c6d-bef5-f5cae1150954","Type":"ContainerDied","Data":"501ec58cf77dd51b8bf0728b99f207e20aa70bc60f7fe8aaa55e407f8046f2ab"}
Oct 30 13:15:20.289135 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:20.289117    4364 scope.go:122] "RemoveContainer" containerID="501ec58cf77dd51b8bf0728b99f207e20aa70bc60f7fe8aaa55e407f8046f2ab"
Oct 30 13:15:20.289448 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:15:20.289429    4364 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-659d72-29889435-zl5cx_odh-ai-gateway-infra(6dc41559-3131-4c6d-bef5-f5cae1150954)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-659d72-29889435-zl5cx" podUID="6dc41559-3131-4c6d-bef5-f5cae1150954"
Oct 30 13:15:20.291910 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:20.291887    4364 generic.go:356] "Generic (PLEG): container finished" podID="cbd3c692-16b9-44ea-98a9-1aa8852f7100" containerID="ec3647d8c74f2f2e0b017190a6e71ac936db4aae368022edf47cf69d2cacf467" exitCode=60
Oct 30 13:15:20.292052 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:20.291960    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889435-km6bf" event={"ID":"cbd3c692-16b9-44ea-98a9-1aa8852f7100","Type":"ContainerDied","Data":"ec3647d8c74f2f2e0b017190a6e71ac936db4aae368022edf47cf69d2cacf467"}
Oct 30 13:15:20.292395 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:20.292374    4364 scope.go:122] "RemoveContainer" containerID="ec3647d8c74f2f2e0b017190a6e71ac936db4aae368022edf47cf69d2cacf467"
Oct 30 13:15:20.292782 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:15:20.292758    4364 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889435-km6bf_odh-ai-gateway-infra(cbd3c692-16b9-44ea-98a9-1aa8852f7100)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889435-km6bf" podUID="cbd3c692-16b9-44ea-98a9-1aa8852f7100"
Oct 30 13:15:20.293979 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:20.293953    4364 scope.go:122] "RemoveContainer" containerID="96ff74f504e1e06f47dd22d5fe887e0c1a89aa88c31ccacd90022b1712a6997c"
Oct 30 13:15:20.294410 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:20.294391    4364 generic.go:356] "Generic (PLEG): container finished" podID="5b7b85af-fbe9-4a34-b0d6-16068595c558" containerID="4f697d15eaa899c5edc57005dc97c14248194da66e5ea37c5e8f15b6d93c5a3a" exitCode=60
Oct 30 13:15:20.294489 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:20.294452    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-8b77b0-29889435-5cb9r" event={"ID":"5b7b85af-fbe9-4a34-b0d6-16068595c558","Type":"ContainerDied","Data":"4f697d15eaa899c5edc57005dc97c14248194da66e5ea37c5e8f15b6d93c5a3a"}
Oct 30 13:15:20.294844 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:20.294826    4364 scope.go:122] "RemoveContainer" containerID="4f697d15eaa899c5edc57005dc97c14248194da66e5ea37c5e8f15b6d93c5a3a"
Oct 30 13:15:20.295119 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:15:20.295100    4364 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-8b77b0-29889435-5cb9r_odh-ai-gateway-infra(5b7b85af-fbe9-4a34-b0d6-16068595c558)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-8b77b0-29889435-5cb9r" podUID="5b7b85af-fbe9-4a34-b0d6-16068595c558"
Oct 30 13:15:20.296084 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:20.296069    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d7e62b-29889435-2fb6n"
Oct 30 13:15:20.296084 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:20.296077    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d7e62b-29889435-2fb6n" event={"ID":"b2e90cd6-1c18-4edb-ae15-7caed4920fe5","Type":"ContainerDied","Data":"fdd19f0cabd7ab7173cba85ca2c2d358d9b9e1cd5532f466cf2bbdda1c6d7114"}
Oct 30 13:15:20.298328 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:20.298301    4364 generic.go:356] "Generic (PLEG): container finished" podID="77ada86d-3625-4786-a01d-2c422fdf7049" containerID="13c06286d932f956d386ba7383375dbc27fb0013855758f49c7dab31db8fd469" exitCode=60
Oct 30 13:15:20.298414 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:20.298358    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889435-vhbwb" event={"ID":"77ada86d-3625-4786-a01d-2c422fdf7049","Type":"ContainerDied","Data":"13c06286d932f956d386ba7383375dbc27fb0013855758f49c7dab31db8fd469"}
Oct 30 13:15:20.298753 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:20.298735    4364 scope.go:122] "RemoveContainer" containerID="13c06286d932f956d386ba7383375dbc27fb0013855758f49c7dab31db8fd469"
Oct 30 13:15:20.299052 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:15:20.299031    4364 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-52a649-29889435-vhbwb_odh-ai-gateway-infra(77ada86d-3625-4786-a01d-2c422fdf7049)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889435-vhbwb" podUID="77ada86d-3625-4786-a01d-2c422fdf7049"
Oct 30 13:15:20.307456 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:20.307434    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889435-vhbwb"]
Oct 30 13:15:20.308863 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:20.308846    4364 scope.go:122] "RemoveContainer" containerID="32efcb9a52693c0232b45529ba89df050032a175c895fd86ddb73a77acd01018"
Oct 30 13:15:20.319287 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:20.319257    4364 scope.go:122] "RemoveContainer" containerID="40917ca068e4a1e4e7b6836a650476e53943feb3b173f8b050ed7cd8fcff8480"
Oct 30 13:15:20.332696 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:20.332677    4364 scope.go:122] "RemoveContainer" containerID="dd1dd833ef19ece80becc3685decd5681e3be2c7d869d9c86d05baf0f88f3c42"
Oct 30 13:15:20.344476 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:20.344460    4364 scope.go:122] "RemoveContainer" containerID="b3d63cfdce1e9382be2bc3c4aadd4659cd51214760b613a4cac32dae977073db"
Oct 30 13:15:20.356651 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:20.356633    4364 scope.go:122] "RemoveContainer" containerID="0a2c43ece3593a9cc3d7f345241ec126ccb4822306efa117bf638c9cb7238160"
Oct 30 13:15:20.368809 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:20.368790    4364 scope.go:122] "RemoveContainer" containerID="fdde710636496e08450681aea7334adba2dcbcc4e49ad61c83335c21ba8c7c2c"
Oct 30 13:15:20.383804 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:20.383785    4364 scope.go:122] "RemoveContainer" containerID="70e17a839d66814e96180873cf24cc871de2ef29de91cd9ed9ec63c1dd828a08"
Oct 30 13:15:20.385963 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:20.385941    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f0a31b-29889435-65zmv"]
Oct 30 13:15:20.396062 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:20.396040    4364 scope.go:122] "RemoveContainer" containerID="0e215363a50b60c725692aec841e2fee46e1d069c1f2896907e0b01c792e94a0"
Oct 30 13:15:20.426101 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:20.426078    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889435-kgl8r"]
Oct 30 13:15:20.455523 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:20.455496    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d7e62b-29889435-2fb6n"]
Oct 30 13:15:20.462145 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:20.462126    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d7e62b-29889435-2fb6n"]
Oct 30 13:15:20.483297 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:20.483274    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d833bd-29889435-wkpwn"]
Oct 30 13:15:20.574838 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:20.574772    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-659d72-29889435-zl5cx"]
Oct 30 13:15:20.624728 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:20.624695    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cb58b8-29889435-25tf8"]
Oct 30 13:15:20.629491 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:20.629471    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cb58b8-29889435-25tf8"]
Oct 30 13:15:20.651958 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:20.651937    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5a6281-29889435-9sk2f"]
Oct 30 13:15:20.678084 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:20.678064    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx"
Oct 30 13:15:20.832900 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:20.832818    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx"
Oct 30 13:15:20.969885 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:20.968707    4364 scope.go:122] "RemoveContainer" containerID="59213a626651e495ec97d13088b697667f2308db10667f55c68e157ac23e7b04"
Oct 30 13:15:20.977420 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:20.977397    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c581513-81ee-42a4-ad45-2ae012744caf" path="/var/lib/kubelet/pods/1c581513-81ee-42a4-ad45-2ae012744caf/volumes"
Oct 30 13:15:20.977880 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:20.977865    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2e90cd6-1c18-4edb-ae15-7caed4920fe5" path="/var/lib/kubelet/pods/b2e90cd6-1c18-4edb-ae15-7caed4920fe5/volumes"
Oct 30 13:15:21.105572 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:21.105501    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-2daeae-5f484f5b8-wzx8f"
Oct 30 13:15:21.328648 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:21.328556    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx"
Oct 30 13:15:21.623316 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:21.622938    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-8b77b0-29889435-5cb9r"]
Oct 30 13:15:21.784524 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:21.784436    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889435-vhbwb"
Oct 30 13:15:21.788763 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:21.788514    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889435-km6bf"]
Oct 30 13:15:21.876540 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:21.876461    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d833bd-29889435-wkpwn"
Oct 30 13:15:21.880818 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:21.880746    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-659d72-29889435-zl5cx"
Oct 30 13:15:21.889427 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:21.889401    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889435-kgl8r"
Oct 30 13:15:21.896465 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:21.896443    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f0a31b-29889435-65zmv"
Oct 30 13:15:21.897316 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:21.897294    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77ada86d-3625-4786-a01d-2c422fdf7049-kube-api-access-wcrmh\" (UniqueName: \"kubernetes.io/projected/77ada86d-3625-4786-a01d-2c422fdf7049-kube-api-access-wcrmh\") pod \"77ada86d-3625-4786-a01d-2c422fdf7049\" (UID: \"77ada86d-3625-4786-a01d-2c422fdf7049\") "
Oct 30 13:15:21.897452 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:21.897437    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77ada86d-3625-4786-a01d-2c422fdf7049-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77ada86d-3625-4786-a01d-2c422fdf7049-maas-api-service-ca\") pod \"77ada86d-3625-4786-a01d-2c422fdf7049\" (UID: \"77ada86d-3625-4786-a01d-2c422fdf7049\") "
Oct 30 13:15:21.898247 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:21.898206    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77ada86d-3625-4786-a01d-2c422fdf7049-maas-api-service-ca" pod "77ada86d-3625-4786-a01d-2c422fdf7049" (UID: "77ada86d-3625-4786-a01d-2c422fdf7049"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:21.900352 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:21.900313    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77ada86d-3625-4786-a01d-2c422fdf7049-kube-api-access-wcrmh" pod "77ada86d-3625-4786-a01d-2c422fdf7049" (UID: "77ada86d-3625-4786-a01d-2c422fdf7049"). InnerVolumeSpecName "kube-api-access-wcrmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:21.903765 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:21.903744    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5a6281-29889435-9sk2f"
Oct 30 13:15:21.999451 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:21.999417    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dc41559-3131-4c6d-bef5-f5cae1150954-kube-api-access-mcq2b\" (UniqueName: \"kubernetes.io/projected/6dc41559-3131-4c6d-bef5-f5cae1150954-kube-api-access-mcq2b\") pod \"6dc41559-3131-4c6d-bef5-f5cae1150954\" (UID: \"6dc41559-3131-4c6d-bef5-f5cae1150954\") "
Oct 30 13:15:21.999594 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:21.999473    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dc41559-3131-4c6d-bef5-f5cae1150954-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dc41559-3131-4c6d-bef5-f5cae1150954-maas-api-service-ca\") pod \"6dc41559-3131-4c6d-bef5-f5cae1150954\" (UID: \"6dc41559-3131-4c6d-bef5-f5cae1150954\") "
Oct 30 13:15:21.999594 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:21.999573    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0187e742-2686-400f-bb53-c74b0d65b5a8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0187e742-2686-400f-bb53-c74b0d65b5a8-maas-api-service-ca\") pod \"0187e742-2686-400f-bb53-c74b0d65b5a8\" (UID: \"0187e742-2686-400f-bb53-c74b0d65b5a8\") "
Oct 30 13:15:21.999734 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:21.999700    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dabc5857-b065-4205-a160-48118225ee82-kube-api-access-t5q97\" (UniqueName: \"kubernetes.io/projected/dabc5857-b065-4205-a160-48118225ee82-kube-api-access-t5q97\") pod \"dabc5857-b065-4205-a160-48118225ee82\" (UID: \"dabc5857-b065-4205-a160-48118225ee82\") "
Oct 30 13:15:21.999795 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:21.999775    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ec36a29-9692-4a74-8297-c73fff6ab164-kube-api-access-k9p5h\" (UniqueName: \"kubernetes.io/projected/4ec36a29-9692-4a74-8297-c73fff6ab164-kube-api-access-k9p5h\") pod \"4ec36a29-9692-4a74-8297-c73fff6ab164\" (UID: \"4ec36a29-9692-4a74-8297-c73fff6ab164\") "
Oct 30 13:15:21.999861 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:21.999842    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ec36a29-9692-4a74-8297-c73fff6ab164-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ec36a29-9692-4a74-8297-c73fff6ab164-maas-api-service-ca\") pod \"4ec36a29-9692-4a74-8297-c73fff6ab164\" (UID: \"4ec36a29-9692-4a74-8297-c73fff6ab164\") "
Oct 30 13:15:21.999911 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:21.999878    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98232f3c-5612-48bf-b272-439425a1fe37-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98232f3c-5612-48bf-b272-439425a1fe37-maas-api-service-ca\") pod \"98232f3c-5612-48bf-b272-439425a1fe37\" (UID: \"98232f3c-5612-48bf-b272-439425a1fe37\") "
Oct 30 13:15:22.000144 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.000127    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0187e742-2686-400f-bb53-c74b0d65b5a8-kube-api-access-dqq49\" (UniqueName: \"kubernetes.io/projected/0187e742-2686-400f-bb53-c74b0d65b5a8-kube-api-access-dqq49\") pod \"0187e742-2686-400f-bb53-c74b0d65b5a8\" (UID: \"0187e742-2686-400f-bb53-c74b0d65b5a8\") "
Oct 30 13:15:22.000200 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.000165    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98232f3c-5612-48bf-b272-439425a1fe37-kube-api-access-s2bgn\" (UniqueName: \"kubernetes.io/projected/98232f3c-5612-48bf-b272-439425a1fe37-kube-api-access-s2bgn\") pod \"98232f3c-5612-48bf-b272-439425a1fe37\" (UID: \"98232f3c-5612-48bf-b272-439425a1fe37\") "
Oct 30 13:15:22.000257 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.000241    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dabc5857-b065-4205-a160-48118225ee82-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dabc5857-b065-4205-a160-48118225ee82-maas-api-service-ca\") pod \"dabc5857-b065-4205-a160-48118225ee82\" (UID: \"dabc5857-b065-4205-a160-48118225ee82\") "
Oct 30 13:15:22.000746 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.000716    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dc41559-3131-4c6d-bef5-f5cae1150954-maas-api-service-ca" pod "6dc41559-3131-4c6d-bef5-f5cae1150954" (UID: "6dc41559-3131-4c6d-bef5-f5cae1150954"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:22.001117 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.001072    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dabc5857-b065-4205-a160-48118225ee82-maas-api-service-ca" pod "dabc5857-b065-4205-a160-48118225ee82" (UID: "dabc5857-b065-4205-a160-48118225ee82"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:22.001209 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.001172    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcrmh\" (UniqueName: \"kubernetes.io/projected/77ada86d-3625-4786-a01d-2c422fdf7049-kube-api-access-wcrmh\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:15:22.001209 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.001197    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77ada86d-3625-4786-a01d-2c422fdf7049-maas-api-service-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:15:22.001324 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.001211    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dc41559-3131-4c6d-bef5-f5cae1150954-maas-api-service-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:15:22.001481 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.001430    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0187e742-2686-400f-bb53-c74b0d65b5a8-maas-api-service-ca" pod "0187e742-2686-400f-bb53-c74b0d65b5a8" (UID: "0187e742-2686-400f-bb53-c74b0d65b5a8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:22.001631 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.001510    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98232f3c-5612-48bf-b272-439425a1fe37-maas-api-service-ca" pod "98232f3c-5612-48bf-b272-439425a1fe37" (UID: "98232f3c-5612-48bf-b272-439425a1fe37"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:22.002085 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.002061    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ec36a29-9692-4a74-8297-c73fff6ab164-maas-api-service-ca" pod "4ec36a29-9692-4a74-8297-c73fff6ab164" (UID: "4ec36a29-9692-4a74-8297-c73fff6ab164"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:22.005533 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.004174    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ec36a29-9692-4a74-8297-c73fff6ab164-kube-api-access-k9p5h" pod "4ec36a29-9692-4a74-8297-c73fff6ab164" (UID: "4ec36a29-9692-4a74-8297-c73fff6ab164"). InnerVolumeSpecName "kube-api-access-k9p5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:22.005533 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.005086    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0187e742-2686-400f-bb53-c74b0d65b5a8-kube-api-access-dqq49" pod "0187e742-2686-400f-bb53-c74b0d65b5a8" (UID: "0187e742-2686-400f-bb53-c74b0d65b5a8"). InnerVolumeSpecName "kube-api-access-dqq49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:22.012855 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.012731    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dc41559-3131-4c6d-bef5-f5cae1150954-kube-api-access-mcq2b" pod "6dc41559-3131-4c6d-bef5-f5cae1150954" (UID: "6dc41559-3131-4c6d-bef5-f5cae1150954"). InnerVolumeSpecName "kube-api-access-mcq2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:22.012855 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.012824    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dabc5857-b065-4205-a160-48118225ee82-kube-api-access-t5q97" pod "dabc5857-b065-4205-a160-48118225ee82" (UID: "dabc5857-b065-4205-a160-48118225ee82"). InnerVolumeSpecName "kube-api-access-t5q97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:22.013805 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.013772    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98232f3c-5612-48bf-b272-439425a1fe37-kube-api-access-s2bgn" pod "98232f3c-5612-48bf-b272-439425a1fe37" (UID: "98232f3c-5612-48bf-b272-439425a1fe37"). InnerVolumeSpecName "kube-api-access-s2bgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:22.023692 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.023669    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8af0fe-6c66bcb66d-lh75p"]
Oct 30 13:15:22.024970 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.024945    4364 cpu_manager.go:414] "containerMap: removing container" podUID="98232f3c-5612-48bf-b272-439425a1fe37" containerName="cleanup"
Oct 30 13:15:22.025072 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.024973    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98232f3c-5612-48bf-b272-439425a1fe37" containerName="cleanup"
Oct 30 13:15:22.025072 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.025001    4364 cpu_manager.go:414] "containerMap: removing container" podUID="77ada86d-3625-4786-a01d-2c422fdf7049" containerName="cleanup"
Oct 30 13:15:22.025072 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.025009    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77ada86d-3625-4786-a01d-2c422fdf7049" containerName="cleanup"
Oct 30 13:15:22.025072 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.025018    4364 cpu_manager.go:414] "containerMap: removing container" podUID="4ec36a29-9692-4a74-8297-c73fff6ab164" containerName="cleanup"
Oct 30 13:15:22.025072 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.025025    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ec36a29-9692-4a74-8297-c73fff6ab164" containerName="cleanup"
Oct 30 13:15:22.025072 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.025052    4364 cpu_manager.go:414] "containerMap: removing container" podUID="1c581513-81ee-42a4-ad45-2ae012744caf" containerName="cleanup"
Oct 30 13:15:22.025072 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.025060    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c581513-81ee-42a4-ad45-2ae012744caf" containerName="cleanup"
Oct 30 13:15:22.025072 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.025068    4364 cpu_manager.go:414] "containerMap: removing container" podUID="0187e742-2686-400f-bb53-c74b0d65b5a8" containerName="cleanup"
Oct 30 13:15:22.025464 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.025076    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0187e742-2686-400f-bb53-c74b0d65b5a8" containerName="cleanup"
Oct 30 13:15:22.025464 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.025099    4364 cpu_manager.go:414] "containerMap: removing container" podUID="b2e90cd6-1c18-4edb-ae15-7caed4920fe5" containerName="cleanup"
Oct 30 13:15:22.025464 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.025107    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2e90cd6-1c18-4edb-ae15-7caed4920fe5" containerName="cleanup"
Oct 30 13:15:22.025464 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.025117    4364 cpu_manager.go:414] "containerMap: removing container" podUID="6dc41559-3131-4c6d-bef5-f5cae1150954" containerName="cleanup"
Oct 30 13:15:22.025464 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.025126    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dc41559-3131-4c6d-bef5-f5cae1150954" containerName="cleanup"
Oct 30 13:15:22.025464 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.025135    4364 cpu_manager.go:414] "containerMap: removing container" podUID="b2e90cd6-1c18-4edb-ae15-7caed4920fe5" containerName="cleanup"
Oct 30 13:15:22.025464 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.025141    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2e90cd6-1c18-4edb-ae15-7caed4920fe5" containerName="cleanup"
Oct 30 13:15:22.025464 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.025156    4364 cpu_manager.go:414] "containerMap: removing container" podUID="dabc5857-b065-4205-a160-48118225ee82" containerName="cleanup"
Oct 30 13:15:22.025464 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.025163    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dabc5857-b065-4205-a160-48118225ee82" containerName="cleanup"
Oct 30 13:15:22.025464 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.025180    4364 cpu_manager.go:414] "containerMap: removing container" podUID="77ada86d-3625-4786-a01d-2c422fdf7049" containerName="cleanup"
Oct 30 13:15:22.025464 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.025187    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77ada86d-3625-4786-a01d-2c422fdf7049" containerName="cleanup"
Oct 30 13:15:22.025464 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.025205    4364 cpu_manager.go:414] "containerMap: removing container" podUID="4ec36a29-9692-4a74-8297-c73fff6ab164" containerName="cleanup"
Oct 30 13:15:22.025464 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.025212    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ec36a29-9692-4a74-8297-c73fff6ab164" containerName="cleanup"
Oct 30 13:15:22.025464 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.025223    4364 cpu_manager.go:414] "containerMap: removing container" podUID="77ada86d-3625-4786-a01d-2c422fdf7049" containerName="cleanup"
Oct 30 13:15:22.025464 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.025231    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77ada86d-3625-4786-a01d-2c422fdf7049" containerName="cleanup"
Oct 30 13:15:22.025464 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.025247    4364 cpu_manager.go:414] "containerMap: removing container" podUID="dabc5857-b065-4205-a160-48118225ee82" containerName="cleanup"
Oct 30 13:15:22.025464 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.025254    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dabc5857-b065-4205-a160-48118225ee82" containerName="cleanup"
Oct 30 13:15:22.025464 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.025263    4364 cpu_manager.go:414] "containerMap: removing container" podUID="98232f3c-5612-48bf-b272-439425a1fe37" containerName="cleanup"
Oct 30 13:15:22.025464 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.025272    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98232f3c-5612-48bf-b272-439425a1fe37" containerName="cleanup"
Oct 30 13:15:22.025464 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.025284    4364 cpu_manager.go:414] "containerMap: removing container" podUID="b2e90cd6-1c18-4edb-ae15-7caed4920fe5" containerName="cleanup"
Oct 30 13:15:22.025464 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.025291    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2e90cd6-1c18-4edb-ae15-7caed4920fe5" containerName="cleanup"
Oct 30 13:15:22.025464 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.025310    4364 cpu_manager.go:414] "containerMap: removing container" podUID="1c581513-81ee-42a4-ad45-2ae012744caf" containerName="cleanup"
Oct 30 13:15:22.025464 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.025317    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c581513-81ee-42a4-ad45-2ae012744caf" containerName="cleanup"
Oct 30 13:15:22.025464 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.025335    4364 cpu_manager.go:414] "containerMap: removing container" podUID="0187e742-2686-400f-bb53-c74b0d65b5a8" containerName="cleanup"
Oct 30 13:15:22.025464 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.025343    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0187e742-2686-400f-bb53-c74b0d65b5a8" containerName="cleanup"
Oct 30 13:15:22.025464 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.025353    4364 cpu_manager.go:414] "containerMap: removing container" podUID="1c581513-81ee-42a4-ad45-2ae012744caf" containerName="cleanup"
Oct 30 13:15:22.025464 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.025360    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c581513-81ee-42a4-ad45-2ae012744caf" containerName="cleanup"
Oct 30 13:15:22.025464 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.025372    4364 cpu_manager.go:414] "containerMap: removing container" podUID="2ada2dea-42f8-4811-8109-87b14971d203" containerName="revoke-keys"
Oct 30 13:15:22.025464 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.025380    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ada2dea-42f8-4811-8109-87b14971d203" containerName="revoke-keys"
Oct 30 13:15:22.025464 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.025399    4364 cpu_manager.go:414] "containerMap: removing container" podUID="6dc41559-3131-4c6d-bef5-f5cae1150954" containerName="cleanup"
Oct 30 13:15:22.025464 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.025405    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dc41559-3131-4c6d-bef5-f5cae1150954" containerName="cleanup"
Oct 30 13:15:22.025464 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.025417    4364 cpu_manager.go:414] "containerMap: removing container" podUID="dabc5857-b065-4205-a160-48118225ee82" containerName="cleanup"
Oct 30 13:15:22.025464 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.025424    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dabc5857-b065-4205-a160-48118225ee82" containerName="cleanup"
Oct 30 13:15:22.027005 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.025688    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ec36a29-9692-4a74-8297-c73fff6ab164" containerName="cleanup"
Oct 30 13:15:22.027005 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.025705    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="77ada86d-3625-4786-a01d-2c422fdf7049" containerName="cleanup"
Oct 30 13:15:22.027005 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.025719    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="dabc5857-b065-4205-a160-48118225ee82" containerName="cleanup"
Oct 30 13:15:22.027005 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.025727    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="98232f3c-5612-48bf-b272-439425a1fe37" containerName="cleanup"
Oct 30 13:15:22.027005 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.025735    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2e90cd6-1c18-4edb-ae15-7caed4920fe5" containerName="cleanup"
Oct 30 13:15:22.027005 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.025744    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c581513-81ee-42a4-ad45-2ae012744caf" containerName="cleanup"
Oct 30 13:15:22.027005 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.025754    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="0187e742-2686-400f-bb53-c74b0d65b5a8" containerName="cleanup"
Oct 30 13:15:22.027005 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.025762    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c581513-81ee-42a4-ad45-2ae012744caf" containerName="cleanup"
Oct 30 13:15:22.027005 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.025773    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ada2dea-42f8-4811-8109-87b14971d203" containerName="revoke-keys"
Oct 30 13:15:22.027005 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.025796    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dc41559-3131-4c6d-bef5-f5cae1150954" containerName="cleanup"
Oct 30 13:15:22.027005 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.025808    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="98232f3c-5612-48bf-b272-439425a1fe37" containerName="cleanup"
Oct 30 13:15:22.027005 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.025817    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ec36a29-9692-4a74-8297-c73fff6ab164" containerName="cleanup"
Oct 30 13:15:22.027005 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.025827    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="0187e742-2686-400f-bb53-c74b0d65b5a8" containerName="cleanup"
Oct 30 13:15:22.027005 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.025837    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2e90cd6-1c18-4edb-ae15-7caed4920fe5" containerName="cleanup"
Oct 30 13:15:22.027005 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.025845    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2e90cd6-1c18-4edb-ae15-7caed4920fe5" containerName="cleanup"
Oct 30 13:15:22.027005 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.025853    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dc41559-3131-4c6d-bef5-f5cae1150954" containerName="cleanup"
Oct 30 13:15:22.027005 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.025862    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="dabc5857-b065-4205-a160-48118225ee82" containerName="cleanup"
Oct 30 13:15:22.027005 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.025873    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="77ada86d-3625-4786-a01d-2c422fdf7049" containerName="cleanup"
Oct 30 13:15:22.036013 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.035990    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8af0fe-6c66bcb66d-lh75p"
Oct 30 13:15:22.056853 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.052285    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8af0fe-6c66bcb66d-lh75p"]
Oct 30 13:15:22.056853 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.052457    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-8af0fe-serving-cert\""
Oct 30 13:15:22.085489 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.085465    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-8b77b0-29889435-5cb9r"
Oct 30 13:15:22.107442 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.105353    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8e76a41-e80b-4b68-b18a-eae91fc341e6-maas-api-tls\") pod \"maas-api-e2e-api-a-8af0fe-6c66bcb66d-lh75p\" (UID: \"f8e76a41-e80b-4b68-b18a-eae91fc341e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8af0fe-6c66bcb66d-lh75p"
Oct 30 13:15:22.107442 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.105677    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rntgj\" (UniqueName: \"kubernetes.io/projected/f8e76a41-e80b-4b68-b18a-eae91fc341e6-kube-api-access-rntgj\") pod \"maas-api-e2e-api-a-8af0fe-6c66bcb66d-lh75p\" (UID: \"f8e76a41-e80b-4b68-b18a-eae91fc341e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8af0fe-6c66bcb66d-lh75p"
Oct 30 13:15:22.107442 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.105723    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8e76a41-e80b-4b68-b18a-eae91fc341e6-metrics-certs\") pod \"maas-api-e2e-api-a-8af0fe-6c66bcb66d-lh75p\" (UID: \"f8e76a41-e80b-4b68-b18a-eae91fc341e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8af0fe-6c66bcb66d-lh75p"
Oct 30 13:15:22.107442 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.105780    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8e76a41-e80b-4b68-b18a-eae91fc341e6-openshift-service-ca\") pod \"maas-api-e2e-api-a-8af0fe-6c66bcb66d-lh75p\" (UID: \"f8e76a41-e80b-4b68-b18a-eae91fc341e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8af0fe-6c66bcb66d-lh75p"
Oct 30 13:15:22.107442 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.105891    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqq49\" (UniqueName: \"kubernetes.io/projected/0187e742-2686-400f-bb53-c74b0d65b5a8-kube-api-access-dqq49\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:15:22.107442 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.105909    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2bgn\" (UniqueName: \"kubernetes.io/projected/98232f3c-5612-48bf-b272-439425a1fe37-kube-api-access-s2bgn\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:15:22.107442 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.105923    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dabc5857-b065-4205-a160-48118225ee82-maas-api-service-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:15:22.107442 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.107263    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcq2b\" (UniqueName: \"kubernetes.io/projected/6dc41559-3131-4c6d-bef5-f5cae1150954-kube-api-access-mcq2b\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:15:22.107442 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.107285    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0187e742-2686-400f-bb53-c74b0d65b5a8-maas-api-service-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:15:22.107442 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.107299    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5q97\" (UniqueName: \"kubernetes.io/projected/dabc5857-b065-4205-a160-48118225ee82-kube-api-access-t5q97\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:15:22.107442 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.107313    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9p5h\" (UniqueName: \"kubernetes.io/projected/4ec36a29-9692-4a74-8297-c73fff6ab164-kube-api-access-k9p5h\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:15:22.107442 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.107328    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ec36a29-9692-4a74-8297-c73fff6ab164-maas-api-service-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:15:22.107442 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.107342    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98232f3c-5612-48bf-b272-439425a1fe37-maas-api-service-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:15:22.128528 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.128505    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889435-km6bf"
Oct 30 13:15:22.208788 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.208761    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b7b85af-fbe9-4a34-b0d6-16068595c558-kube-api-access-vmqhq\" (UniqueName: \"kubernetes.io/projected/5b7b85af-fbe9-4a34-b0d6-16068595c558-kube-api-access-vmqhq\") pod \"5b7b85af-fbe9-4a34-b0d6-16068595c558\" (UID: \"5b7b85af-fbe9-4a34-b0d6-16068595c558\") "
Oct 30 13:15:22.208974 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.208802    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbd3c692-16b9-44ea-98a9-1aa8852f7100-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbd3c692-16b9-44ea-98a9-1aa8852f7100-maas-api-service-ca\") pod \"cbd3c692-16b9-44ea-98a9-1aa8852f7100\" (UID: \"cbd3c692-16b9-44ea-98a9-1aa8852f7100\") "
Oct 30 13:15:22.208974 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.208893    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbd3c692-16b9-44ea-98a9-1aa8852f7100-kube-api-access-zxd56\" (UniqueName: \"kubernetes.io/projected/cbd3c692-16b9-44ea-98a9-1aa8852f7100-kube-api-access-zxd56\") pod \"cbd3c692-16b9-44ea-98a9-1aa8852f7100\" (UID: \"cbd3c692-16b9-44ea-98a9-1aa8852f7100\") "
Oct 30 13:15:22.209155 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.209116    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b7b85af-fbe9-4a34-b0d6-16068595c558-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b7b85af-fbe9-4a34-b0d6-16068595c558-maas-api-service-ca\") pod \"5b7b85af-fbe9-4a34-b0d6-16068595c558\" (UID: \"5b7b85af-fbe9-4a34-b0d6-16068595c558\") "
Oct 30 13:15:22.209343 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.209308    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbd3c692-16b9-44ea-98a9-1aa8852f7100-maas-api-service-ca" pod "cbd3c692-16b9-44ea-98a9-1aa8852f7100" (UID: "cbd3c692-16b9-44ea-98a9-1aa8852f7100"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:22.209506 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.209481    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b7b85af-fbe9-4a34-b0d6-16068595c558-maas-api-service-ca" pod "5b7b85af-fbe9-4a34-b0d6-16068595c558" (UID: "5b7b85af-fbe9-4a34-b0d6-16068595c558"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:22.209788 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.209713    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8e76a41-e80b-4b68-b18a-eae91fc341e6-maas-api-tls\") pod \"maas-api-e2e-api-a-8af0fe-6c66bcb66d-lh75p\" (UID: \"f8e76a41-e80b-4b68-b18a-eae91fc341e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8af0fe-6c66bcb66d-lh75p"
Oct 30 13:15:22.210032 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.210011    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rntgj\" (UniqueName: \"kubernetes.io/projected/f8e76a41-e80b-4b68-b18a-eae91fc341e6-kube-api-access-rntgj\") pod \"maas-api-e2e-api-a-8af0fe-6c66bcb66d-lh75p\" (UID: \"f8e76a41-e80b-4b68-b18a-eae91fc341e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8af0fe-6c66bcb66d-lh75p"
Oct 30 13:15:22.210652 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.210066    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8e76a41-e80b-4b68-b18a-eae91fc341e6-metrics-certs\") pod \"maas-api-e2e-api-a-8af0fe-6c66bcb66d-lh75p\" (UID: \"f8e76a41-e80b-4b68-b18a-eae91fc341e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8af0fe-6c66bcb66d-lh75p"
Oct 30 13:15:22.210652 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.210482    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8e76a41-e80b-4b68-b18a-eae91fc341e6-openshift-service-ca\") pod \"maas-api-e2e-api-a-8af0fe-6c66bcb66d-lh75p\" (UID: \"f8e76a41-e80b-4b68-b18a-eae91fc341e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8af0fe-6c66bcb66d-lh75p"
Oct 30 13:15:22.210861 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.210778    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbd3c692-16b9-44ea-98a9-1aa8852f7100-maas-api-service-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:15:22.210861 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.210802    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b7b85af-fbe9-4a34-b0d6-16068595c558-maas-api-service-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:15:22.211433 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.211405    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8e76a41-e80b-4b68-b18a-eae91fc341e6-openshift-service-ca\") pod \"maas-api-e2e-api-a-8af0fe-6c66bcb66d-lh75p\" (UID: \"f8e76a41-e80b-4b68-b18a-eae91fc341e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8af0fe-6c66bcb66d-lh75p"
Oct 30 13:15:22.211549 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.211524    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b7b85af-fbe9-4a34-b0d6-16068595c558-kube-api-access-vmqhq" pod "5b7b85af-fbe9-4a34-b0d6-16068595c558" (UID: "5b7b85af-fbe9-4a34-b0d6-16068595c558"). InnerVolumeSpecName "kube-api-access-vmqhq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:22.212668 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.212644    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8e76a41-e80b-4b68-b18a-eae91fc341e6-maas-api-tls\") pod \"maas-api-e2e-api-a-8af0fe-6c66bcb66d-lh75p\" (UID: \"f8e76a41-e80b-4b68-b18a-eae91fc341e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8af0fe-6c66bcb66d-lh75p"
Oct 30 13:15:22.213179 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.213155    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbd3c692-16b9-44ea-98a9-1aa8852f7100-kube-api-access-zxd56" pod "cbd3c692-16b9-44ea-98a9-1aa8852f7100" (UID: "cbd3c692-16b9-44ea-98a9-1aa8852f7100"). InnerVolumeSpecName "kube-api-access-zxd56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:22.213685 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.213619    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8e76a41-e80b-4b68-b18a-eae91fc341e6-metrics-certs\") pod \"maas-api-e2e-api-a-8af0fe-6c66bcb66d-lh75p\" (UID: \"f8e76a41-e80b-4b68-b18a-eae91fc341e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8af0fe-6c66bcb66d-lh75p"
Oct 30 13:15:22.234848 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.234772    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rntgj\" (UniqueName: \"kubernetes.io/projected/f8e76a41-e80b-4b68-b18a-eae91fc341e6-kube-api-access-rntgj\") pod \"maas-api-e2e-api-a-8af0fe-6c66bcb66d-lh75p\" (UID: \"f8e76a41-e80b-4b68-b18a-eae91fc341e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8af0fe-6c66bcb66d-lh75p"
Oct 30 13:15:22.312399 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.312367    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxd56\" (UniqueName: \"kubernetes.io/projected/cbd3c692-16b9-44ea-98a9-1aa8852f7100-kube-api-access-zxd56\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:15:22.312399 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.312394    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmqhq\" (UniqueName: \"kubernetes.io/projected/5b7b85af-fbe9-4a34-b0d6-16068595c558-kube-api-access-vmqhq\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:15:22.336549 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.336525    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889435-vhbwb"
Oct 30 13:15:22.337007 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.336543    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889435-vhbwb" event={"ID":"77ada86d-3625-4786-a01d-2c422fdf7049","Type":"ContainerDied","Data":"924ecd9596496f9711f6c36fc622b36da824c71ef7eb6b423c5b329a9d312cb1"}
Oct 30 13:15:22.337007 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.336643    4364 scope.go:122] "RemoveContainer" containerID="13c06286d932f956d386ba7383375dbc27fb0013855758f49c7dab31db8fd469"
Oct 30 13:15:22.338371 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.338349    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5a6281-29889435-9sk2f" event={"ID":"98232f3c-5612-48bf-b272-439425a1fe37","Type":"ContainerDied","Data":"42db23c8a3ff3271dc8030c22884dafd700e76b4bc4b71f914611b88948711b3"}
Oct 30 13:15:22.338494 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.338356    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5a6281-29889435-9sk2f"
Oct 30 13:15:22.340378 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.340007    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f0a31b-29889435-65zmv"
Oct 30 13:15:22.340378 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.340010    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f0a31b-29889435-65zmv" event={"ID":"4ec36a29-9692-4a74-8297-c73fff6ab164","Type":"ContainerDied","Data":"0a4696d2c18f37d11163446ed0692d096c57aa37bf30394b32f8f939c9ac6ee2"}
Oct 30 13:15:22.341719 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.341696    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d833bd-29889435-wkpwn"
Oct 30 13:15:22.341901 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.341699    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d833bd-29889435-wkpwn" event={"ID":"0187e742-2686-400f-bb53-c74b0d65b5a8","Type":"ContainerDied","Data":"b38caacca2b0512753f113ca2c9cb0a1104c0a2685838f34be247e5aea98dd46"}
Oct 30 13:15:22.345665 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.345634    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889435-kgl8r" event={"ID":"dabc5857-b065-4205-a160-48118225ee82","Type":"ContainerDied","Data":"6f5f3107a2f675c2a9c5a9fbc410a051dfdd8a4f613ec15f92ec30af95ac9b75"}
Oct 30 13:15:22.345665 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.345654    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889435-kgl8r"
Oct 30 13:15:22.348615 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.348313    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-659d72-29889435-zl5cx"
Oct 30 13:15:22.348615 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.348332    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-659d72-29889435-zl5cx" event={"ID":"6dc41559-3131-4c6d-bef5-f5cae1150954","Type":"ContainerDied","Data":"2d7a47256dcf5c29d5fcff3096955f396ccf179f2a7ced31d7cbfb97aa463a8f"}
Oct 30 13:15:22.353174 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.353158    4364 scope.go:122] "RemoveContainer" containerID="1afec301abd694af74bdbe2251c47260bcc3b7f6e907d37df96be4b25821a23d"
Oct 30 13:15:22.353225 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.353178    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889435-km6bf" event={"ID":"cbd3c692-16b9-44ea-98a9-1aa8852f7100","Type":"ContainerDied","Data":"3c98c533eb319fb150d2bf69069295342232a5dd82043b079ddbce36936cfc63"}
Oct 30 13:15:22.353225 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.353165    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889435-km6bf"
Oct 30 13:15:22.354637 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.354617    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-8b77b0-29889435-5cb9r"
Oct 30 13:15:22.354727 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.354620    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-8b77b0-29889435-5cb9r" event={"ID":"5b7b85af-fbe9-4a34-b0d6-16068595c558","Type":"ContainerDied","Data":"4d06c9fa98f567597984fdb65f8c26915e00fd2e693f6ffcc2e967ce64f9c719"}
Oct 30 13:15:22.358353 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.358328    4364 generic.go:356] "Generic (PLEG): container finished" podID="d75b5256-b21b-47d6-8acb-45cb0afe1190" containerID="7d0f0b9b1ed07b39ea57a4da1a9c1a06c2e97be9ab84b0663f7eff148002e82d" exitCode=60
Oct 30 13:15:22.358518 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.358386    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889435-c6pcv" event={"ID":"d75b5256-b21b-47d6-8acb-45cb0afe1190","Type":"ContainerDied","Data":"7d0f0b9b1ed07b39ea57a4da1a9c1a06c2e97be9ab84b0663f7eff148002e82d"}
Oct 30 13:15:22.359331 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.359309    4364 scope.go:122] "RemoveContainer" containerID="7d0f0b9b1ed07b39ea57a4da1a9c1a06c2e97be9ab84b0663f7eff148002e82d"
Oct 30 13:15:22.359743 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:15:22.359716    4364 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889435-c6pcv_odh-ai-gateway-infra(d75b5256-b21b-47d6-8acb-45cb0afe1190)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889435-c6pcv" podUID="d75b5256-b21b-47d6-8acb-45cb0afe1190"
Oct 30 13:15:22.370366 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.370335    4364 scope.go:122] "RemoveContainer" containerID="4923bbb89b9f1e2289b77b86e672c5226d46bafbab65df1119d894d9fcddcb2b"
Oct 30 13:15:22.383292 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.383274    4364 scope.go:122] "RemoveContainer" containerID="d4139d97029671ae03c6f0c173dc2fcd6929a7778cb8857d33da731df8f8db0f"
Oct 30 13:15:22.383512 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.383488    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8af0fe-6c66bcb66d-lh75p"
Oct 30 13:15:22.398976 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.398952    4364 scope.go:122] "RemoveContainer" containerID="b940753f8f10b447075db1564ad2acaeca683deda842a4a3ff5ef24ff3f3dda3"
Oct 30 13:15:22.408713 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.408695    4364 scope.go:122] "RemoveContainer" containerID="501ec58cf77dd51b8bf0728b99f207e20aa70bc60f7fe8aaa55e407f8046f2ab"
Oct 30 13:15:22.419557 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.419535    4364 scope.go:122] "RemoveContainer" containerID="ec3647d8c74f2f2e0b017190a6e71ac936db4aae368022edf47cf69d2cacf467"
Oct 30 13:15:22.437270 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.437166    4364 scope.go:122] "RemoveContainer" containerID="4f697d15eaa899c5edc57005dc97c14248194da66e5ea37c5e8f15b6d93c5a3a"
Oct 30 13:15:22.448472 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.448443    4364 scope.go:122] "RemoveContainer" containerID="59213a626651e495ec97d13088b697667f2308db10667f55c68e157ac23e7b04"
Oct 30 13:15:22.479004 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.478951    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889435-vhbwb"]
Oct 30 13:15:22.487825 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.487752    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889435-vhbwb"]
Oct 30 13:15:22.520893 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.520867    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d833bd-29889435-wkpwn"]
Oct 30 13:15:22.527931 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.527906    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d833bd-29889435-wkpwn"]
Oct 30 13:15:22.556277 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.556252    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-659d72-29889435-zl5cx"]
Oct 30 13:15:22.564697 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.564675    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-659d72-29889435-zl5cx"]
Oct 30 13:15:22.591833 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.591813    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-8b77b0-29889435-5cb9r"]
Oct 30 13:15:22.599368 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.599344    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-8b77b0-29889435-5cb9r"]
Oct 30 13:15:22.642094 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.642069    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5a6281-29889435-9sk2f"]
Oct 30 13:15:22.648061 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.648040    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5a6281-29889435-9sk2f"]
Oct 30 13:15:22.680551 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.680527    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f0a31b-29889435-65zmv"]
Oct 30 13:15:22.691312 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.691291    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f0a31b-29889435-65zmv"]
Oct 30 13:15:22.702829 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.702805    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8af0fe-6c66bcb66d-lh75p"]
Oct 30 13:15:22.704457 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:15:22.704431    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8e76a41_e80b_4b68_b18a_eae91fc341e6.slice/crio-23a4c45aa8b7cf7397cf102787adbe365c7cbf2597c72a9a5606ad00f8a7503b WatchSource:0}: Error finding container 23a4c45aa8b7cf7397cf102787adbe365c7cbf2597c72a9a5606ad00f8a7503b: Status 404 returned error can't find the container with id 23a4c45aa8b7cf7397cf102787adbe365c7cbf2597c72a9a5606ad00f8a7503b
Oct 30 13:15:22.705965 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.705917    4364 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 13:15:22.726377 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.726355    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889435-kgl8r"]
Oct 30 13:15:22.737621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.737586    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889435-kgl8r"]
Oct 30 13:15:22.811885 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.811818    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889435-km6bf"]
Oct 30 13:15:22.822114 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.822081    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889435-km6bf"]
Oct 30 13:15:22.972876 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.972847    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0187e742-2686-400f-bb53-c74b0d65b5a8" path="/var/lib/kubelet/pods/0187e742-2686-400f-bb53-c74b0d65b5a8/volumes"
Oct 30 13:15:22.973288 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.973273    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ec36a29-9692-4a74-8297-c73fff6ab164" path="/var/lib/kubelet/pods/4ec36a29-9692-4a74-8297-c73fff6ab164/volumes"
Oct 30 13:15:22.973695 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.973680    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b7b85af-fbe9-4a34-b0d6-16068595c558" path="/var/lib/kubelet/pods/5b7b85af-fbe9-4a34-b0d6-16068595c558/volumes"
Oct 30 13:15:22.974386 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.974371    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dc41559-3131-4c6d-bef5-f5cae1150954" path="/var/lib/kubelet/pods/6dc41559-3131-4c6d-bef5-f5cae1150954/volumes"
Oct 30 13:15:22.974795 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.974781    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77ada86d-3625-4786-a01d-2c422fdf7049" path="/var/lib/kubelet/pods/77ada86d-3625-4786-a01d-2c422fdf7049/volumes"
Oct 30 13:15:22.975175 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.975163    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98232f3c-5612-48bf-b272-439425a1fe37" path="/var/lib/kubelet/pods/98232f3c-5612-48bf-b272-439425a1fe37/volumes"
Oct 30 13:15:22.975836 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.975821    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbd3c692-16b9-44ea-98a9-1aa8852f7100" path="/var/lib/kubelet/pods/cbd3c692-16b9-44ea-98a9-1aa8852f7100/volumes"
Oct 30 13:15:22.976219 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:22.976206    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dabc5857-b065-4205-a160-48118225ee82" path="/var/lib/kubelet/pods/dabc5857-b065-4205-a160-48118225ee82/volumes"
Oct 30 13:15:23.475672 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:23.467821    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8af0fe-6c66bcb66d-lh75p" event={"ID":"f8e76a41-e80b-4b68-b18a-eae91fc341e6","Type":"ContainerStarted","Data":"b7c943a82c260bf763ee4b7dfe623b3f6c3a9a6248adc8b86f2e0e7a64d36013"}
Oct 30 13:15:23.475672 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:23.467911    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8af0fe-6c66bcb66d-lh75p" event={"ID":"f8e76a41-e80b-4b68-b18a-eae91fc341e6","Type":"ContainerStarted","Data":"23a4c45aa8b7cf7397cf102787adbe365c7cbf2597c72a9a5606ad00f8a7503b"}
Oct 30 13:15:23.489645 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:23.484459    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8af0fe-6c66bcb66d-lh75p"
Oct 30 13:15:23.567651 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:23.567399    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8af0fe-6c66bcb66d-lh75p" podStartSLOduration=8.118368187 podStartE2EDuration="8.567379031s" totalImagesPullingTime="449.010844ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:23.549116729 +0000 UTC m=+1422.715481389" watchObservedRunningTime="2026-10-30 13:15:23.567379031 +0000 UTC m=+1422.733743681"
Oct 30 13:15:23.958661 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:23.953634    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889435-c6pcv"]
Oct 30 13:15:24.277694 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:24.277046    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2daeae-5b6fb8f887-22zjb"
Oct 30 13:15:24.501226 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:24.497176    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3778dacd-fqdm6"]
Oct 30 13:15:24.502068 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:24.502010    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3778dacd-fqdm6"]
Oct 30 13:15:24.778354 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:24.778326    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889435-c6pcv"
Oct 30 13:15:24.804727 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:24.804661    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d75b5256-b21b-47d6-8acb-45cb0afe1190-kube-api-access-zkpcw\" (UniqueName: \"kubernetes.io/projected/d75b5256-b21b-47d6-8acb-45cb0afe1190-kube-api-access-zkpcw\") pod \"d75b5256-b21b-47d6-8acb-45cb0afe1190\" (UID: \"d75b5256-b21b-47d6-8acb-45cb0afe1190\") "
Oct 30 13:15:24.804881 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:24.804746    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d75b5256-b21b-47d6-8acb-45cb0afe1190-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d75b5256-b21b-47d6-8acb-45cb0afe1190-maas-api-service-ca\") pod \"d75b5256-b21b-47d6-8acb-45cb0afe1190\" (UID: \"d75b5256-b21b-47d6-8acb-45cb0afe1190\") "
Oct 30 13:15:24.805128 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:24.805104    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d75b5256-b21b-47d6-8acb-45cb0afe1190-maas-api-service-ca" pod "d75b5256-b21b-47d6-8acb-45cb0afe1190" (UID: "d75b5256-b21b-47d6-8acb-45cb0afe1190"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:24.805562 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:24.805540    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d75b5256-b21b-47d6-8acb-45cb0afe1190-maas-api-service-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:15:24.807092 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:24.807060    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d75b5256-b21b-47d6-8acb-45cb0afe1190-kube-api-access-zkpcw" pod "d75b5256-b21b-47d6-8acb-45cb0afe1190" (UID: "d75b5256-b21b-47d6-8acb-45cb0afe1190"). InnerVolumeSpecName "kube-api-access-zkpcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:24.825465 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:24.825439    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45"]
Oct 30 13:15:24.825756 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:24.825730    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45" podUID="b835314b-3c09-4143-966b-928a86200e6c" containerName="istio-proxy" containerID="cri-o://e1aaa7f1ff3fb33a9df88cb64536a8fb57fb64d03cd81347e8e56bded80bf1a2" gracePeriod=30
Oct 30 13:15:24.834344 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:24.834321    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q"]
Oct 30 13:15:24.834570 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:24.834545    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q" podUID="9ab4d389-3115-40fe-aad2-137fed1ffe54" containerName="istio-proxy" containerID="cri-o://ee39c9c380a7d8958155936c2358f5d26cdad6bf1e7a658b3bc56f220deb4e12" gracePeriod=30
Oct 30 13:15:24.900370 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:24.900333    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v"]
Oct 30 13:15:24.900634 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:24.900568    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v" podUID="cc6e612c-e20e-4573-a608-edbf4b7de570" containerName="istio-proxy" containerID="cri-o://3eac5a3d4c71d8b90fd92739254e773b6047468c3b747ed5a18b998a5d26bf7a" gracePeriod=30
Oct 30 13:15:24.908141 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:24.908110    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkpcw\" (UniqueName: \"kubernetes.io/projected/d75b5256-b21b-47d6-8acb-45cb0afe1190-kube-api-access-zkpcw\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:15:24.986697 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:24.986656    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd032232-133c-4c37-b99b-88b14b88cebb" path="/var/lib/kubelet/pods/fd032232-133c-4c37-b99b-88b14b88cebb/volumes"
Oct 30 13:15:25.537990 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:25.537960    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889435-c6pcv"
Oct 30 13:15:25.537990 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:25.537972    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889435-c6pcv" event={"ID":"d75b5256-b21b-47d6-8acb-45cb0afe1190","Type":"ContainerDied","Data":"dbfcc8dd089c3e2c0ed92a8a900a96a5ccbbd4c02b6a581c42ccb25bcf18832c"}
Oct 30 13:15:25.538513 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:25.538034    4364 scope.go:122] "RemoveContainer" containerID="7d0f0b9b1ed07b39ea57a4da1a9c1a06c2e97be9ab84b0663f7eff148002e82d"
Oct 30 13:15:25.595317 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:25.595294    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889435-c6pcv"]
Oct 30 13:15:25.600744 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:25.600726    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889435-c6pcv"]
Oct 30 13:15:26.972609 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:26.972567    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d75b5256-b21b-47d6-8acb-45cb0afe1190" path="/var/lib/kubelet/pods/d75b5256-b21b-47d6-8acb-45cb0afe1190/volumes"
Oct 30 13:15:29.658441 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:29.658412    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fafb3f-cf6wb"]
Oct 30 13:15:29.659316 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:29.659299    4364 cpu_manager.go:414] "containerMap: removing container" podUID="4ec36a29-9692-4a74-8297-c73fff6ab164" containerName="cleanup"
Oct 30 13:15:29.659372 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:29.659317    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ec36a29-9692-4a74-8297-c73fff6ab164" containerName="cleanup"
Oct 30 13:15:29.659372 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:29.659332    4364 cpu_manager.go:414] "containerMap: removing container" podUID="5b7b85af-fbe9-4a34-b0d6-16068595c558" containerName="cleanup"
Oct 30 13:15:29.659372 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:29.659339    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b7b85af-fbe9-4a34-b0d6-16068595c558" containerName="cleanup"
Oct 30 13:15:29.659372 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:29.659347    4364 cpu_manager.go:414] "containerMap: removing container" podUID="d75b5256-b21b-47d6-8acb-45cb0afe1190" containerName="cleanup"
Oct 30 13:15:29.659372 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:29.659351    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d75b5256-b21b-47d6-8acb-45cb0afe1190" containerName="cleanup"
Oct 30 13:15:29.659372 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:29.659359    4364 cpu_manager.go:414] "containerMap: removing container" podUID="98232f3c-5612-48bf-b272-439425a1fe37" containerName="cleanup"
Oct 30 13:15:29.659372 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:29.659363    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98232f3c-5612-48bf-b272-439425a1fe37" containerName="cleanup"
Oct 30 13:15:29.659570 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:29.659376    4364 cpu_manager.go:414] "containerMap: removing container" podUID="cbd3c692-16b9-44ea-98a9-1aa8852f7100" containerName="cleanup"
Oct 30 13:15:29.659570 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:29.659381    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbd3c692-16b9-44ea-98a9-1aa8852f7100" containerName="cleanup"
Oct 30 13:15:29.659570 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:29.659389    4364 cpu_manager.go:414] "containerMap: removing container" podUID="0187e742-2686-400f-bb53-c74b0d65b5a8" containerName="cleanup"
Oct 30 13:15:29.659570 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:29.659393    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0187e742-2686-400f-bb53-c74b0d65b5a8" containerName="cleanup"
Oct 30 13:15:29.659570 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:29.659404    4364 cpu_manager.go:414] "containerMap: removing container" podUID="cbd3c692-16b9-44ea-98a9-1aa8852f7100" containerName="cleanup"
Oct 30 13:15:29.659570 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:29.659409    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbd3c692-16b9-44ea-98a9-1aa8852f7100" containerName="cleanup"
Oct 30 13:15:29.659570 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:29.659416    4364 cpu_manager.go:414] "containerMap: removing container" podUID="6dc41559-3131-4c6d-bef5-f5cae1150954" containerName="cleanup"
Oct 30 13:15:29.659570 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:29.659420    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dc41559-3131-4c6d-bef5-f5cae1150954" containerName="cleanup"
Oct 30 13:15:29.659570 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:29.659441    4364 cpu_manager.go:414] "containerMap: removing container" podUID="5b7b85af-fbe9-4a34-b0d6-16068595c558" containerName="cleanup"
Oct 30 13:15:29.659570 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:29.659446    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b7b85af-fbe9-4a34-b0d6-16068595c558" containerName="cleanup"
Oct 30 13:15:29.659570 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:29.659456    4364 cpu_manager.go:414] "containerMap: removing container" podUID="d75b5256-b21b-47d6-8acb-45cb0afe1190" containerName="cleanup"
Oct 30 13:15:29.659570 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:29.659460    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d75b5256-b21b-47d6-8acb-45cb0afe1190" containerName="cleanup"
Oct 30 13:15:29.659570 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:29.659468    4364 cpu_manager.go:414] "containerMap: removing container" podUID="d75b5256-b21b-47d6-8acb-45cb0afe1190" containerName="cleanup"
Oct 30 13:15:29.659570 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:29.659473    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d75b5256-b21b-47d6-8acb-45cb0afe1190" containerName="cleanup"
Oct 30 13:15:29.660007 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:29.659591    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbd3c692-16b9-44ea-98a9-1aa8852f7100" containerName="cleanup"
Oct 30 13:15:29.660007 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:29.659611    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c581513-81ee-42a4-ad45-2ae012744caf" containerName="cleanup"
Oct 30 13:15:29.660007 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:29.659620    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbd3c692-16b9-44ea-98a9-1aa8852f7100" containerName="cleanup"
Oct 30 13:15:29.660007 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:29.659626    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dc41559-3131-4c6d-bef5-f5cae1150954" containerName="cleanup"
Oct 30 13:15:29.660007 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:29.659634    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="98232f3c-5612-48bf-b272-439425a1fe37" containerName="cleanup"
Oct 30 13:15:29.660007 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:29.659641    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b7b85af-fbe9-4a34-b0d6-16068595c558" containerName="cleanup"
Oct 30 13:15:29.660007 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:29.659646    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b7b85af-fbe9-4a34-b0d6-16068595c558" containerName="cleanup"
Oct 30 13:15:29.660007 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:29.659652    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="d75b5256-b21b-47d6-8acb-45cb0afe1190" containerName="cleanup"
Oct 30 13:15:29.660007 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:29.659658    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="d75b5256-b21b-47d6-8acb-45cb0afe1190" containerName="cleanup"
Oct 30 13:15:29.660007 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:29.659663    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="0187e742-2686-400f-bb53-c74b0d65b5a8" containerName="cleanup"
Oct 30 13:15:29.660007 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:29.659672    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ec36a29-9692-4a74-8297-c73fff6ab164" containerName="cleanup"
Oct 30 13:15:29.660007 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:29.659678    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="77ada86d-3625-4786-a01d-2c422fdf7049" containerName="cleanup"
Oct 30 13:15:29.660007 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:29.659686    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="dabc5857-b065-4205-a160-48118225ee82" containerName="cleanup"
Oct 30 13:15:29.663767 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:29.663751    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fafb3f-cf6wb"
Oct 30 13:15:29.685098 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:29.685064    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fafb3f-cf6wb"]
Oct 30 13:15:29.707396 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:29.707172    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:15:29.707396 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:29.707314    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\""
Oct 30 13:15:29.809477 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:29.809430    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s84mg\" (UniqueName: \"kubernetes.io/projected/17f6dcdc-6416-44e4-8d22-d9bd1681b8a0-kube-api-access-s84mg\") pod \"maas-api-revoke-keys-e2e-api-a-fafb3f-cf6wb\" (UID: \"17f6dcdc-6416-44e4-8d22-d9bd1681b8a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fafb3f-cf6wb"
Oct 30 13:15:29.809713 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:29.809691    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17f6dcdc-6416-44e4-8d22-d9bd1681b8a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fafb3f-cf6wb\" (UID: \"17f6dcdc-6416-44e4-8d22-d9bd1681b8a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fafb3f-cf6wb"
Oct 30 13:15:29.809885 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:29.809859    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17f6dcdc-6416-44e4-8d22-d9bd1681b8a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fafb3f-cf6wb\" (UID: \"17f6dcdc-6416-44e4-8d22-d9bd1681b8a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fafb3f-cf6wb"
Oct 30 13:15:29.911428 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:29.911350    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17f6dcdc-6416-44e4-8d22-d9bd1681b8a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fafb3f-cf6wb\" (UID: \"17f6dcdc-6416-44e4-8d22-d9bd1681b8a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fafb3f-cf6wb"
Oct 30 13:15:29.911533 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:29.911478    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17f6dcdc-6416-44e4-8d22-d9bd1681b8a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fafb3f-cf6wb\" (UID: \"17f6dcdc-6416-44e4-8d22-d9bd1681b8a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fafb3f-cf6wb"
Oct 30 13:15:29.911664 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:29.911650    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s84mg\" (UniqueName: \"kubernetes.io/projected/17f6dcdc-6416-44e4-8d22-d9bd1681b8a0-kube-api-access-s84mg\") pod \"maas-api-revoke-keys-e2e-api-a-fafb3f-cf6wb\" (UID: \"17f6dcdc-6416-44e4-8d22-d9bd1681b8a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fafb3f-cf6wb"
Oct 30 13:15:29.912188 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:29.912161    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17f6dcdc-6416-44e4-8d22-d9bd1681b8a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fafb3f-cf6wb\" (UID: \"17f6dcdc-6416-44e4-8d22-d9bd1681b8a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fafb3f-cf6wb"
Oct 30 13:15:29.913590 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:29.913566    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17f6dcdc-6416-44e4-8d22-d9bd1681b8a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fafb3f-cf6wb\" (UID: \"17f6dcdc-6416-44e4-8d22-d9bd1681b8a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fafb3f-cf6wb"
Oct 30 13:15:29.937762 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:29.937730    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s84mg\" (UniqueName: \"kubernetes.io/projected/17f6dcdc-6416-44e4-8d22-d9bd1681b8a0-kube-api-access-s84mg\") pod \"maas-api-revoke-keys-e2e-api-a-fafb3f-cf6wb\" (UID: \"17f6dcdc-6416-44e4-8d22-d9bd1681b8a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fafb3f-cf6wb"
Oct 30 13:15:29.979412 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:29.979384    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fafb3f-cf6wb"
Oct 30 13:15:30.285401 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.285376    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45"
Oct 30 13:15:30.296996 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.296971    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v"
Oct 30 13:15:30.304779 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.304758    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fafb3f-cf6wb"]
Oct 30 13:15:30.306106 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.306090    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q"
Oct 30 13:15:30.306354 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:15:30.306332    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17f6dcdc_6416_44e4_8d22_d9bd1681b8a0.slice/crio-5272156ea61d52fc38333ed1be057e289da218b22de1d67aec5c4594713b72e9 WatchSource:0}: Error finding container 5272156ea61d52fc38333ed1be057e289da218b22de1d67aec5c4594713b72e9: Status 404 returned error can't find the container with id 5272156ea61d52fc38333ed1be057e289da218b22de1d67aec5c4594713b72e9
Oct 30 13:15:30.363299 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.363263    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9fcb69-6f45cd6b55-7z62k"]
Oct 30 13:15:30.364638 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.364564    4364 cpu_manager.go:414] "containerMap: removing container" podUID="5b7b85af-fbe9-4a34-b0d6-16068595c558" containerName="cleanup"
Oct 30 13:15:30.364638 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.364591    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b7b85af-fbe9-4a34-b0d6-16068595c558" containerName="cleanup"
Oct 30 13:15:30.364638 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.364630    4364 cpu_manager.go:414] "containerMap: removing container" podUID="9ab4d389-3115-40fe-aad2-137fed1ffe54" containerName="istio-proxy"
Oct 30 13:15:30.364638 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.364638    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ab4d389-3115-40fe-aad2-137fed1ffe54" containerName="istio-proxy"
Oct 30 13:15:30.364925 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.364674    4364 cpu_manager.go:414] "containerMap: removing container" podUID="cbd3c692-16b9-44ea-98a9-1aa8852f7100" containerName="cleanup"
Oct 30 13:15:30.364925 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.364683    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbd3c692-16b9-44ea-98a9-1aa8852f7100" containerName="cleanup"
Oct 30 13:15:30.364925 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.364716    4364 cpu_manager.go:414] "containerMap: removing container" podUID="b835314b-3c09-4143-966b-928a86200e6c" containerName="istio-proxy"
Oct 30 13:15:30.364925 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.364724    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b835314b-3c09-4143-966b-928a86200e6c" containerName="istio-proxy"
Oct 30 13:15:30.364925 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.364749    4364 cpu_manager.go:414] "containerMap: removing container" podUID="cc6e612c-e20e-4573-a608-edbf4b7de570" containerName="istio-proxy"
Oct 30 13:15:30.364925 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.364756    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc6e612c-e20e-4573-a608-edbf4b7de570" containerName="istio-proxy"
Oct 30 13:15:30.365199 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.364959    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="d75b5256-b21b-47d6-8acb-45cb0afe1190" containerName="cleanup"
Oct 30 13:15:30.365199 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.364986    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ab4d389-3115-40fe-aad2-137fed1ffe54" containerName="istio-proxy"
Oct 30 13:15:30.365199 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.364998    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbd3c692-16b9-44ea-98a9-1aa8852f7100" containerName="cleanup"
Oct 30 13:15:30.365199 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.365012    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="b835314b-3c09-4143-966b-928a86200e6c" containerName="istio-proxy"
Oct 30 13:15:30.365199 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.365023    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b7b85af-fbe9-4a34-b0d6-16068595c558" containerName="cleanup"
Oct 30 13:15:30.365199 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.365033    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc6e612c-e20e-4573-a608-edbf4b7de570" containerName="istio-proxy"
Oct 30 13:15:30.369301 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.369283    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9fcb69-6f45cd6b55-7z62k"
Oct 30 13:15:30.393084 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.393055    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-9fcb69-serving-cert\""
Oct 30 13:15:30.397037 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.397010    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9fcb69-6f45cd6b55-7z62k"]
Oct 30 13:15:30.421824 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.421793    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc6e612c-e20e-4573-a608-edbf4b7de570-kube-api-access-llgpx\" (UniqueName: \"kubernetes.io/projected/cc6e612c-e20e-4573-a608-edbf4b7de570-kube-api-access-llgpx\") pod \"cc6e612c-e20e-4573-a608-edbf4b7de570\" (UID: \"cc6e612c-e20e-4573-a608-edbf4b7de570\") "
Oct 30 13:15:30.421987 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.421891    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ab4d389-3115-40fe-aad2-137fed1ffe54-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ab4d389-3115-40fe-aad2-137fed1ffe54-credential-socket\") pod \"9ab4d389-3115-40fe-aad2-137fed1ffe54\" (UID: \"9ab4d389-3115-40fe-aad2-137fed1ffe54\") "
Oct 30 13:15:30.421987 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.421923    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ab4d389-3115-40fe-aad2-137fed1ffe54-kube-api-access-gkxpj\" (UniqueName: \"kubernetes.io/projected/9ab4d389-3115-40fe-aad2-137fed1ffe54-kube-api-access-gkxpj\") pod \"9ab4d389-3115-40fe-aad2-137fed1ffe54\" (UID: \"9ab4d389-3115-40fe-aad2-137fed1ffe54\") "
Oct 30 13:15:30.421987 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.421965    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc6e612c-e20e-4573-a608-edbf4b7de570-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc6e612c-e20e-4573-a608-edbf4b7de570-istio-envoy\") pod \"cc6e612c-e20e-4573-a608-edbf4b7de570\" (UID: \"cc6e612c-e20e-4573-a608-edbf4b7de570\") "
Oct 30 13:15:30.422160 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.422005    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b835314b-3c09-4143-966b-928a86200e6c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b835314b-3c09-4143-966b-928a86200e6c-workload-socket\") pod \"b835314b-3c09-4143-966b-928a86200e6c\" (UID: \"b835314b-3c09-4143-966b-928a86200e6c\") "
Oct 30 13:15:30.422160 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.422037    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc6e612c-e20e-4573-a608-edbf4b7de570-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc6e612c-e20e-4573-a608-edbf4b7de570-istio-data\") pod \"cc6e612c-e20e-4573-a608-edbf4b7de570\" (UID: \"cc6e612c-e20e-4573-a608-edbf4b7de570\") "
Oct 30 13:15:30.422160 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.422062    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b835314b-3c09-4143-966b-928a86200e6c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b835314b-3c09-4143-966b-928a86200e6c-istio-envoy\") pod \"b835314b-3c09-4143-966b-928a86200e6c\" (UID: \"b835314b-3c09-4143-966b-928a86200e6c\") "
Oct 30 13:15:30.422318 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.422184    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ab4d389-3115-40fe-aad2-137fed1ffe54-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ab4d389-3115-40fe-aad2-137fed1ffe54-istio-data\") pod \"9ab4d389-3115-40fe-aad2-137fed1ffe54\" (UID: \"9ab4d389-3115-40fe-aad2-137fed1ffe54\") "
Oct 30 13:15:30.422318 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.422220    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ab4d389-3115-40fe-aad2-137fed1ffe54-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ab4d389-3115-40fe-aad2-137fed1ffe54-workload-certs\") pod \"9ab4d389-3115-40fe-aad2-137fed1ffe54\" (UID: \"9ab4d389-3115-40fe-aad2-137fed1ffe54\") "
Oct 30 13:15:30.422318 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.422297    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b835314b-3c09-4143-966b-928a86200e6c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b835314b-3c09-4143-966b-928a86200e6c-credential-socket\") pod \"b835314b-3c09-4143-966b-928a86200e6c\" (UID: \"b835314b-3c09-4143-966b-928a86200e6c\") "
Oct 30 13:15:30.422481 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.422345    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b835314b-3c09-4143-966b-928a86200e6c-istio-token\" (UniqueName: \"kubernetes.io/projected/b835314b-3c09-4143-966b-928a86200e6c-istio-token\") pod \"b835314b-3c09-4143-966b-928a86200e6c\" (UID: \"b835314b-3c09-4143-966b-928a86200e6c\") "
Oct 30 13:15:30.422481 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.422425    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc6e612c-e20e-4573-a608-edbf4b7de570-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc6e612c-e20e-4573-a608-edbf4b7de570-istiod-ca-cert\") pod \"cc6e612c-e20e-4573-a608-edbf4b7de570\" (UID: \"cc6e612c-e20e-4573-a608-edbf4b7de570\") "
Oct 30 13:15:30.422694 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.422502    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b835314b-3c09-4143-966b-928a86200e6c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b835314b-3c09-4143-966b-928a86200e6c-workload-certs\") pod \"b835314b-3c09-4143-966b-928a86200e6c\" (UID: \"b835314b-3c09-4143-966b-928a86200e6c\") "
Oct 30 13:15:30.422694 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.422532    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b835314b-3c09-4143-966b-928a86200e6c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b835314b-3c09-4143-966b-928a86200e6c-istio-podinfo\") pod \"b835314b-3c09-4143-966b-928a86200e6c\" (UID: \"b835314b-3c09-4143-966b-928a86200e6c\") "
Oct 30 13:15:30.422694 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.422566    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9ab4d389-3115-40fe-aad2-137fed1ffe54-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ab4d389-3115-40fe-aad2-137fed1ffe54-istio-podinfo\") pod \"9ab4d389-3115-40fe-aad2-137fed1ffe54\" (UID: \"9ab4d389-3115-40fe-aad2-137fed1ffe54\") "
Oct 30 13:15:30.422694 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.422670    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ab4d389-3115-40fe-aad2-137fed1ffe54-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ab4d389-3115-40fe-aad2-137fed1ffe54-istio-envoy\") pod \"9ab4d389-3115-40fe-aad2-137fed1ffe54\" (UID: \"9ab4d389-3115-40fe-aad2-137fed1ffe54\") "
Oct 30 13:15:30.422898 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.422709    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b835314b-3c09-4143-966b-928a86200e6c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b835314b-3c09-4143-966b-928a86200e6c-istiod-ca-cert\") pod \"b835314b-3c09-4143-966b-928a86200e6c\" (UID: \"b835314b-3c09-4143-966b-928a86200e6c\") "
Oct 30 13:15:30.422898 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.422755    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b835314b-3c09-4143-966b-928a86200e6c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b835314b-3c09-4143-966b-928a86200e6c-istio-data\") pod \"b835314b-3c09-4143-966b-928a86200e6c\" (UID: \"b835314b-3c09-4143-966b-928a86200e6c\") "
Oct 30 13:15:30.422898 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.422869    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc6e612c-e20e-4573-a608-edbf4b7de570-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc6e612c-e20e-4573-a608-edbf4b7de570-istio-podinfo\") pod \"cc6e612c-e20e-4573-a608-edbf4b7de570\" (UID: \"cc6e612c-e20e-4573-a608-edbf4b7de570\") "
Oct 30 13:15:30.423047 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.422909    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ab4d389-3115-40fe-aad2-137fed1ffe54-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ab4d389-3115-40fe-aad2-137fed1ffe54-istiod-ca-cert\") pod \"9ab4d389-3115-40fe-aad2-137fed1ffe54\" (UID: \"9ab4d389-3115-40fe-aad2-137fed1ffe54\") "
Oct 30 13:15:30.423047 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.422956    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc6e612c-e20e-4573-a608-edbf4b7de570-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc6e612c-e20e-4573-a608-edbf4b7de570-credential-socket\") pod \"cc6e612c-e20e-4573-a608-edbf4b7de570\" (UID: \"cc6e612c-e20e-4573-a608-edbf4b7de570\") "
Oct 30 13:15:30.423047 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.422984    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc6e612c-e20e-4573-a608-edbf4b7de570-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc6e612c-e20e-4573-a608-edbf4b7de570-workload-certs\") pod \"cc6e612c-e20e-4573-a608-edbf4b7de570\" (UID: \"cc6e612c-e20e-4573-a608-edbf4b7de570\") "
Oct 30 13:15:30.423333 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.423309    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ab4d389-3115-40fe-aad2-137fed1ffe54-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ab4d389-3115-40fe-aad2-137fed1ffe54-workload-socket\") pod \"9ab4d389-3115-40fe-aad2-137fed1ffe54\" (UID: \"9ab4d389-3115-40fe-aad2-137fed1ffe54\") "
Oct 30 13:15:30.423397 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.423357    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ab4d389-3115-40fe-aad2-137fed1ffe54-istio-token\" (UniqueName: \"kubernetes.io/projected/9ab4d389-3115-40fe-aad2-137fed1ffe54-istio-token\") pod \"9ab4d389-3115-40fe-aad2-137fed1ffe54\" (UID: \"9ab4d389-3115-40fe-aad2-137fed1ffe54\") "
Oct 30 13:15:30.423453 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.423434    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc6e612c-e20e-4573-a608-edbf4b7de570-istio-token\" (UniqueName: \"kubernetes.io/projected/cc6e612c-e20e-4573-a608-edbf4b7de570-istio-token\") pod \"cc6e612c-e20e-4573-a608-edbf4b7de570\" (UID: \"cc6e612c-e20e-4573-a608-edbf4b7de570\") "
Oct 30 13:15:30.423569 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.422163    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ab4d389-3115-40fe-aad2-137fed1ffe54-credential-socket" pod "9ab4d389-3115-40fe-aad2-137fed1ffe54" (UID: "9ab4d389-3115-40fe-aad2-137fed1ffe54"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:15:30.423569 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.423503    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc6e612c-e20e-4573-a608-edbf4b7de570-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc6e612c-e20e-4573-a608-edbf4b7de570-workload-socket\") pod \"cc6e612c-e20e-4573-a608-edbf4b7de570\" (UID: \"cc6e612c-e20e-4573-a608-edbf4b7de570\") "
Oct 30 13:15:30.423569 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.422524    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b835314b-3c09-4143-966b-928a86200e6c-workload-socket" pod "b835314b-3c09-4143-966b-928a86200e6c" (UID: "b835314b-3c09-4143-966b-928a86200e6c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:15:30.423569 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.423475    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ab4d389-3115-40fe-aad2-137fed1ffe54-istio-data" pod "9ab4d389-3115-40fe-aad2-137fed1ffe54" (UID: "9ab4d389-3115-40fe-aad2-137fed1ffe54"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:15:30.423920 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.423584    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b835314b-3c09-4143-966b-928a86200e6c-kube-api-access-rvt6r\" (UniqueName: \"kubernetes.io/projected/b835314b-3c09-4143-966b-928a86200e6c-kube-api-access-rvt6r\") pod \"b835314b-3c09-4143-966b-928a86200e6c\" (UID: \"b835314b-3c09-4143-966b-928a86200e6c\") "
Oct 30 13:15:30.423920 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.423765    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b835314b-3c09-4143-966b-928a86200e6c-workload-certs" pod "b835314b-3c09-4143-966b-928a86200e6c" (UID: "b835314b-3c09-4143-966b-928a86200e6c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:15:30.424040 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.424014    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n7xs\" (UniqueName: \"kubernetes.io/projected/287c395c-95cf-4133-9319-f0a2c3ae60b6-kube-api-access-8n7xs\") pod \"maas-api-e2e-api-a-9fcb69-6f45cd6b55-7z62k\" (UID: \"287c395c-95cf-4133-9319-f0a2c3ae60b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9fcb69-6f45cd6b55-7z62k"
Oct 30 13:15:30.424098 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.424074    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ab4d389-3115-40fe-aad2-137fed1ffe54-workload-certs" pod "9ab4d389-3115-40fe-aad2-137fed1ffe54" (UID: "9ab4d389-3115-40fe-aad2-137fed1ffe54"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:15:30.424151 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.424095    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/287c395c-95cf-4133-9319-f0a2c3ae60b6-maas-api-tls\") pod \"maas-api-e2e-api-a-9fcb69-6f45cd6b55-7z62k\" (UID: \"287c395c-95cf-4133-9319-f0a2c3ae60b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9fcb69-6f45cd6b55-7z62k"
Oct 30 13:15:30.424151 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.424145    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/287c395c-95cf-4133-9319-f0a2c3ae60b6-openshift-service-ca\") pod \"maas-api-e2e-api-a-9fcb69-6f45cd6b55-7z62k\" (UID: \"287c395c-95cf-4133-9319-f0a2c3ae60b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9fcb69-6f45cd6b55-7z62k"
Oct 30 13:15:30.424323 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.424299    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc6e612c-e20e-4573-a608-edbf4b7de570-credential-socket" pod "cc6e612c-e20e-4573-a608-edbf4b7de570" (UID: "cc6e612c-e20e-4573-a608-edbf4b7de570"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:15:30.424384 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.424331    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/287c395c-95cf-4133-9319-f0a2c3ae60b6-metrics-certs\") pod \"maas-api-e2e-api-a-9fcb69-6f45cd6b55-7z62k\" (UID: \"287c395c-95cf-4133-9319-f0a2c3ae60b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9fcb69-6f45cd6b55-7z62k"
Oct 30 13:15:30.424862 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.424838    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc6e612c-e20e-4573-a608-edbf4b7de570-istiod-ca-cert" pod "cc6e612c-e20e-4573-a608-edbf4b7de570" (UID: "cc6e612c-e20e-4573-a608-edbf4b7de570"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:30.425087 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.425065    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b835314b-3c09-4143-966b-928a86200e6c-credential-socket" pod "b835314b-3c09-4143-966b-928a86200e6c" (UID: "b835314b-3c09-4143-966b-928a86200e6c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:15:30.425174 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.425153    4364 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b835314b-3c09-4143-966b-928a86200e6c-workload-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:15:30.425249 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.425184    4364 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc6e612c-e20e-4573-a608-edbf4b7de570-credential-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:15:30.425249 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.425199    4364 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ab4d389-3115-40fe-aad2-137fed1ffe54-credential-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:15:30.425249 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.425215    4364 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b835314b-3c09-4143-966b-928a86200e6c-workload-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:15:30.425249 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.425228    4364 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ab4d389-3115-40fe-aad2-137fed1ffe54-istio-data\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:15:30.425249 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.425241    4364 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ab4d389-3115-40fe-aad2-137fed1ffe54-workload-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:15:30.425489 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.425254    4364 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc6e612c-e20e-4573-a608-edbf4b7de570-istiod-ca-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:15:30.426137 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.425474    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc6e612c-e20e-4573-a608-edbf4b7de570-workload-certs" pod "cc6e612c-e20e-4573-a608-edbf4b7de570" (UID: "cc6e612c-e20e-4573-a608-edbf4b7de570"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:15:30.427775 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.427749    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ab4d389-3115-40fe-aad2-137fed1ffe54-workload-socket" pod "9ab4d389-3115-40fe-aad2-137fed1ffe54" (UID: "9ab4d389-3115-40fe-aad2-137fed1ffe54"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:15:30.436589 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.436567    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b835314b-3c09-4143-966b-928a86200e6c-istio-data" pod "b835314b-3c09-4143-966b-928a86200e6c" (UID: "b835314b-3c09-4143-966b-928a86200e6c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:15:30.436589 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.436585    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc6e612c-e20e-4573-a608-edbf4b7de570-istio-data" pod "cc6e612c-e20e-4573-a608-edbf4b7de570" (UID: "cc6e612c-e20e-4573-a608-edbf4b7de570"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:15:30.437095 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.436830    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ab4d389-3115-40fe-aad2-137fed1ffe54-istiod-ca-cert" pod "9ab4d389-3115-40fe-aad2-137fed1ffe54" (UID: "9ab4d389-3115-40fe-aad2-137fed1ffe54"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:30.437095 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.436893    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b835314b-3c09-4143-966b-928a86200e6c-istiod-ca-cert" pod "b835314b-3c09-4143-966b-928a86200e6c" (UID: "b835314b-3c09-4143-966b-928a86200e6c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:30.437095 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.437034    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc6e612c-e20e-4573-a608-edbf4b7de570-workload-socket" pod "cc6e612c-e20e-4573-a608-edbf4b7de570" (UID: "cc6e612c-e20e-4573-a608-edbf4b7de570"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:15:30.481016 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.480979    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc6e612c-e20e-4573-a608-edbf4b7de570-kube-api-access-llgpx" pod "cc6e612c-e20e-4573-a608-edbf4b7de570" (UID: "cc6e612c-e20e-4573-a608-edbf4b7de570"). InnerVolumeSpecName "kube-api-access-llgpx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:30.481698 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.481455    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc6e612c-e20e-4573-a608-edbf4b7de570-istio-envoy" pod "cc6e612c-e20e-4573-a608-edbf4b7de570" (UID: "cc6e612c-e20e-4573-a608-edbf4b7de570"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:15:30.481910 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.481764    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ab4d389-3115-40fe-aad2-137fed1ffe54-istio-podinfo" pod "9ab4d389-3115-40fe-aad2-137fed1ffe54" (UID: "9ab4d389-3115-40fe-aad2-137fed1ffe54"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:15:30.481910 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.481528    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ab4d389-3115-40fe-aad2-137fed1ffe54-kube-api-access-gkxpj" pod "9ab4d389-3115-40fe-aad2-137fed1ffe54" (UID: "9ab4d389-3115-40fe-aad2-137fed1ffe54"). InnerVolumeSpecName "kube-api-access-gkxpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:30.482101 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.482063    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b835314b-3c09-4143-966b-928a86200e6c-kube-api-access-rvt6r" pod "b835314b-3c09-4143-966b-928a86200e6c" (UID: "b835314b-3c09-4143-966b-928a86200e6c"). InnerVolumeSpecName "kube-api-access-rvt6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:30.482443 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.482151    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc6e612c-e20e-4573-a608-edbf4b7de570-istio-token" pod "cc6e612c-e20e-4573-a608-edbf4b7de570" (UID: "cc6e612c-e20e-4573-a608-edbf4b7de570"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:30.482581 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.482550    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b835314b-3c09-4143-966b-928a86200e6c-istio-podinfo" pod "b835314b-3c09-4143-966b-928a86200e6c" (UID: "b835314b-3c09-4143-966b-928a86200e6c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:15:30.482818 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.482797    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b835314b-3c09-4143-966b-928a86200e6c-istio-token" pod "b835314b-3c09-4143-966b-928a86200e6c" (UID: "b835314b-3c09-4143-966b-928a86200e6c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:30.483040 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.483003    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc6e612c-e20e-4573-a608-edbf4b7de570-istio-podinfo" pod "cc6e612c-e20e-4573-a608-edbf4b7de570" (UID: "cc6e612c-e20e-4573-a608-edbf4b7de570"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:15:30.483546 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.483511    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b835314b-3c09-4143-966b-928a86200e6c-istio-envoy" pod "b835314b-3c09-4143-966b-928a86200e6c" (UID: "b835314b-3c09-4143-966b-928a86200e6c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:15:30.483811 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.483788    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ab4d389-3115-40fe-aad2-137fed1ffe54-istio-token" pod "9ab4d389-3115-40fe-aad2-137fed1ffe54" (UID: "9ab4d389-3115-40fe-aad2-137fed1ffe54"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:30.484050 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.484030    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ab4d389-3115-40fe-aad2-137fed1ffe54-istio-envoy" pod "9ab4d389-3115-40fe-aad2-137fed1ffe54" (UID: "9ab4d389-3115-40fe-aad2-137fed1ffe54"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:15:30.526937 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.526839    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n7xs\" (UniqueName: \"kubernetes.io/projected/287c395c-95cf-4133-9319-f0a2c3ae60b6-kube-api-access-8n7xs\") pod \"maas-api-e2e-api-a-9fcb69-6f45cd6b55-7z62k\" (UID: \"287c395c-95cf-4133-9319-f0a2c3ae60b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9fcb69-6f45cd6b55-7z62k"
Oct 30 13:15:30.526937 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.526900    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/287c395c-95cf-4133-9319-f0a2c3ae60b6-maas-api-tls\") pod \"maas-api-e2e-api-a-9fcb69-6f45cd6b55-7z62k\" (UID: \"287c395c-95cf-4133-9319-f0a2c3ae60b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9fcb69-6f45cd6b55-7z62k"
Oct 30 13:15:30.527174 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.526938    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/287c395c-95cf-4133-9319-f0a2c3ae60b6-openshift-service-ca\") pod \"maas-api-e2e-api-a-9fcb69-6f45cd6b55-7z62k\" (UID: \"287c395c-95cf-4133-9319-f0a2c3ae60b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9fcb69-6f45cd6b55-7z62k"
Oct 30 13:15:30.527174 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.527031    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/287c395c-95cf-4133-9319-f0a2c3ae60b6-metrics-certs\") pod \"maas-api-e2e-api-a-9fcb69-6f45cd6b55-7z62k\" (UID: \"287c395c-95cf-4133-9319-f0a2c3ae60b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9fcb69-6f45cd6b55-7z62k"
Oct 30 13:15:30.527300 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.527190    4364 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc6e612c-e20e-4573-a608-edbf4b7de570-workload-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:15:30.527300 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.527203    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvt6r\" (UniqueName: \"kubernetes.io/projected/b835314b-3c09-4143-966b-928a86200e6c-kube-api-access-rvt6r\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:15:30.527300 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.527212    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llgpx\" (UniqueName: \"kubernetes.io/projected/cc6e612c-e20e-4573-a608-edbf4b7de570-kube-api-access-llgpx\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:15:30.527300 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.527221    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkxpj\" (UniqueName: \"kubernetes.io/projected/9ab4d389-3115-40fe-aad2-137fed1ffe54-kube-api-access-gkxpj\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:15:30.527300 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.527236    4364 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc6e612c-e20e-4573-a608-edbf4b7de570-istio-envoy\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:15:30.527300 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.527249    4364 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc6e612c-e20e-4573-a608-edbf4b7de570-istio-data\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:15:30.527300 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.527261    4364 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b835314b-3c09-4143-966b-928a86200e6c-istio-envoy\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:15:30.527300 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.527273    4364 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b835314b-3c09-4143-966b-928a86200e6c-credential-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:15:30.527300 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.527287    4364 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b835314b-3c09-4143-966b-928a86200e6c-istio-token\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:15:30.527300 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.527301    4364 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b835314b-3c09-4143-966b-928a86200e6c-istio-podinfo\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:15:30.527860 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.527313    4364 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ab4d389-3115-40fe-aad2-137fed1ffe54-istio-podinfo\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:15:30.527860 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.527326    4364 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ab4d389-3115-40fe-aad2-137fed1ffe54-istio-envoy\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:15:30.527860 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.527339    4364 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b835314b-3c09-4143-966b-928a86200e6c-istiod-ca-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:15:30.527860 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.527351    4364 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b835314b-3c09-4143-966b-928a86200e6c-istio-data\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:15:30.527860 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.527363    4364 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc6e612c-e20e-4573-a608-edbf4b7de570-istio-podinfo\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:15:30.527860 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.527374    4364 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ab4d389-3115-40fe-aad2-137fed1ffe54-istiod-ca-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:15:30.527860 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.527385    4364 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc6e612c-e20e-4573-a608-edbf4b7de570-workload-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:15:30.527860 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.527397    4364 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ab4d389-3115-40fe-aad2-137fed1ffe54-workload-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:15:30.527860 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.527409    4364 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ab4d389-3115-40fe-aad2-137fed1ffe54-istio-token\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:15:30.527860 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.527423    4364 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc6e612c-e20e-4573-a608-edbf4b7de570-istio-token\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:15:30.527860 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.527803    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/287c395c-95cf-4133-9319-f0a2c3ae60b6-openshift-service-ca\") pod \"maas-api-e2e-api-a-9fcb69-6f45cd6b55-7z62k\" (UID: \"287c395c-95cf-4133-9319-f0a2c3ae60b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9fcb69-6f45cd6b55-7z62k"
Oct 30 13:15:30.529516 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.529490    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/287c395c-95cf-4133-9319-f0a2c3ae60b6-maas-api-tls\") pod \"maas-api-e2e-api-a-9fcb69-6f45cd6b55-7z62k\" (UID: \"287c395c-95cf-4133-9319-f0a2c3ae60b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9fcb69-6f45cd6b55-7z62k"
Oct 30 13:15:30.529648 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.529536    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/287c395c-95cf-4133-9319-f0a2c3ae60b6-metrics-certs\") pod \"maas-api-e2e-api-a-9fcb69-6f45cd6b55-7z62k\" (UID: \"287c395c-95cf-4133-9319-f0a2c3ae60b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9fcb69-6f45cd6b55-7z62k"
Oct 30 13:15:30.543663 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.543640    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8af0fe-6c66bcb66d-lh75p"
Oct 30 13:15:30.551586 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.551562    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n7xs\" (UniqueName: \"kubernetes.io/projected/287c395c-95cf-4133-9319-f0a2c3ae60b6-kube-api-access-8n7xs\") pod \"maas-api-e2e-api-a-9fcb69-6f45cd6b55-7z62k\" (UID: \"287c395c-95cf-4133-9319-f0a2c3ae60b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9fcb69-6f45cd6b55-7z62k"
Oct 30 13:15:30.582622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.582581    4364 generic.go:356] "Generic (PLEG): container finished" podID="b835314b-3c09-4143-966b-928a86200e6c" containerID="e1aaa7f1ff3fb33a9df88cb64536a8fb57fb64d03cd81347e8e56bded80bf1a2" exitCode=0
Oct 30 13:15:30.582753 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.582642    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45" event={"ID":"b835314b-3c09-4143-966b-928a86200e6c","Type":"ContainerDied","Data":"e1aaa7f1ff3fb33a9df88cb64536a8fb57fb64d03cd81347e8e56bded80bf1a2"}
Oct 30 13:15:30.582753 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.582664    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45"
Oct 30 13:15:30.582753 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.582691    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45" event={"ID":"b835314b-3c09-4143-966b-928a86200e6c","Type":"ContainerDied","Data":"6ee2a82403a88a42543d04dce16d1d3e9458f5f59c95d4cf0391d37d3f010e2b"}
Oct 30 13:15:30.582753 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.582712    4364 scope.go:122] "RemoveContainer" containerID="e1aaa7f1ff3fb33a9df88cb64536a8fb57fb64d03cd81347e8e56bded80bf1a2"
Oct 30 13:15:30.584366 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.584338    4364 generic.go:356] "Generic (PLEG): container finished" podID="cc6e612c-e20e-4573-a608-edbf4b7de570" containerID="3eac5a3d4c71d8b90fd92739254e773b6047468c3b747ed5a18b998a5d26bf7a" exitCode=0
Oct 30 13:15:30.584480 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.584372    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v" event={"ID":"cc6e612c-e20e-4573-a608-edbf4b7de570","Type":"ContainerDied","Data":"3eac5a3d4c71d8b90fd92739254e773b6047468c3b747ed5a18b998a5d26bf7a"}
Oct 30 13:15:30.584558 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.584486    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v"
Oct 30 13:15:30.584845 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.584814    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v" event={"ID":"cc6e612c-e20e-4573-a608-edbf4b7de570","Type":"ContainerDied","Data":"34bbc56dbe8bd5b95938341aae4c077bd2bde23244088a3bc33350c38dc39e19"}
Oct 30 13:15:30.619685 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.616303    4364 generic.go:356] "Generic (PLEG): container finished" podID="9ab4d389-3115-40fe-aad2-137fed1ffe54" containerID="ee39c9c380a7d8958155936c2358f5d26cdad6bf1e7a658b3bc56f220deb4e12" exitCode=0
Oct 30 13:15:30.619685 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.616363    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q" event={"ID":"9ab4d389-3115-40fe-aad2-137fed1ffe54","Type":"ContainerDied","Data":"ee39c9c380a7d8958155936c2358f5d26cdad6bf1e7a658b3bc56f220deb4e12"}
Oct 30 13:15:30.619685 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.616433    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q" event={"ID":"9ab4d389-3115-40fe-aad2-137fed1ffe54","Type":"ContainerDied","Data":"b63321b011b3e335b150e584f84a14c12d5d8d03bb2a922d9ed742210a8cabef"}
Oct 30 13:15:30.619685 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.616378    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q"
Oct 30 13:15:30.620882 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.620852    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fafb3f-cf6wb" event={"ID":"17f6dcdc-6416-44e4-8d22-d9bd1681b8a0","Type":"ContainerStarted","Data":"5272156ea61d52fc38333ed1be057e289da218b22de1d67aec5c4594713b72e9"}
Oct 30 13:15:30.625765 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.625743    4364 scope.go:122] "RemoveContainer" containerID="e1aaa7f1ff3fb33a9df88cb64536a8fb57fb64d03cd81347e8e56bded80bf1a2"
Oct 30 13:15:30.626415 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:15:30.626046    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1aaa7f1ff3fb33a9df88cb64536a8fb57fb64d03cd81347e8e56bded80bf1a2\": container with ID starting with e1aaa7f1ff3fb33a9df88cb64536a8fb57fb64d03cd81347e8e56bded80bf1a2 not found: ID does not exist" containerID="e1aaa7f1ff3fb33a9df88cb64536a8fb57fb64d03cd81347e8e56bded80bf1a2"
Oct 30 13:15:30.626415 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.626088    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1aaa7f1ff3fb33a9df88cb64536a8fb57fb64d03cd81347e8e56bded80bf1a2"} err="failed to get container status \"e1aaa7f1ff3fb33a9df88cb64536a8fb57fb64d03cd81347e8e56bded80bf1a2\": rpc error: code = NotFound desc = could not find container \"e1aaa7f1ff3fb33a9df88cb64536a8fb57fb64d03cd81347e8e56bded80bf1a2\": container with ID starting with e1aaa7f1ff3fb33a9df88cb64536a8fb57fb64d03cd81347e8e56bded80bf1a2 not found: ID does not exist"
Oct 30 13:15:30.626415 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.626115    4364 scope.go:122] "RemoveContainer" containerID="3eac5a3d4c71d8b90fd92739254e773b6047468c3b747ed5a18b998a5d26bf7a"
Oct 30 13:15:30.642195 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.642175    4364 scope.go:122] "RemoveContainer" containerID="3eac5a3d4c71d8b90fd92739254e773b6047468c3b747ed5a18b998a5d26bf7a"
Oct 30 13:15:30.642810 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:15:30.642782    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3eac5a3d4c71d8b90fd92739254e773b6047468c3b747ed5a18b998a5d26bf7a\": container with ID starting with 3eac5a3d4c71d8b90fd92739254e773b6047468c3b747ed5a18b998a5d26bf7a not found: ID does not exist" containerID="3eac5a3d4c71d8b90fd92739254e773b6047468c3b747ed5a18b998a5d26bf7a"
Oct 30 13:15:30.642969 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.642936    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3eac5a3d4c71d8b90fd92739254e773b6047468c3b747ed5a18b998a5d26bf7a"} err="failed to get container status \"3eac5a3d4c71d8b90fd92739254e773b6047468c3b747ed5a18b998a5d26bf7a\": rpc error: code = NotFound desc = could not find container \"3eac5a3d4c71d8b90fd92739254e773b6047468c3b747ed5a18b998a5d26bf7a\": container with ID starting with 3eac5a3d4c71d8b90fd92739254e773b6047468c3b747ed5a18b998a5d26bf7a not found: ID does not exist"
Oct 30 13:15:30.642969 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.642970    4364 scope.go:122] "RemoveContainer" containerID="ee39c9c380a7d8958155936c2358f5d26cdad6bf1e7a658b3bc56f220deb4e12"
Oct 30 13:15:30.657039 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.657014    4364 scope.go:122] "RemoveContainer" containerID="ee39c9c380a7d8958155936c2358f5d26cdad6bf1e7a658b3bc56f220deb4e12"
Oct 30 13:15:30.657401 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:15:30.657369    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee39c9c380a7d8958155936c2358f5d26cdad6bf1e7a658b3bc56f220deb4e12\": container with ID starting with ee39c9c380a7d8958155936c2358f5d26cdad6bf1e7a658b3bc56f220deb4e12 not found: ID does not exist" containerID="ee39c9c380a7d8958155936c2358f5d26cdad6bf1e7a658b3bc56f220deb4e12"
Oct 30 13:15:30.657490 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.657412    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee39c9c380a7d8958155936c2358f5d26cdad6bf1e7a658b3bc56f220deb4e12"} err="failed to get container status \"ee39c9c380a7d8958155936c2358f5d26cdad6bf1e7a658b3bc56f220deb4e12\": rpc error: code = NotFound desc = could not find container \"ee39c9c380a7d8958155936c2358f5d26cdad6bf1e7a658b3bc56f220deb4e12\": container with ID starting with ee39c9c380a7d8958155936c2358f5d26cdad6bf1e7a658b3bc56f220deb4e12 not found: ID does not exist"
Oct 30 13:15:30.684483 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.684457    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q"]
Oct 30 13:15:30.689541 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.689519    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-kzg7q"]
Oct 30 13:15:30.694464 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.694444    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9fcb69-6f45cd6b55-7z62k"
Oct 30 13:15:30.722967 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.722936    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v"]
Oct 30 13:15:30.729645 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.729609    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-9tq6v"]
Oct 30 13:15:30.936450 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:30.869557    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45"]
Oct 30 13:15:31.186932 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:31.186883    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ab4d389-3115-40fe-aad2-137fed1ffe54" path="/var/lib/kubelet/pods/9ab4d389-3115-40fe-aad2-137fed1ffe54/volumes"
Oct 30 13:15:31.190053 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:31.190007    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc6e612c-e20e-4573-a608-edbf4b7de570" path="/var/lib/kubelet/pods/cc6e612c-e20e-4573-a608-edbf4b7de570/volumes"
Oct 30 13:15:31.190751 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:31.190704    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-m7g45"]
Oct 30 13:15:31.347408 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:31.347381    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9fcb69-6f45cd6b55-7z62k"]
Oct 30 13:15:31.631151 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:31.631124    4364 generic.go:356] "Generic (PLEG): container finished" podID="17f6dcdc-6416-44e4-8d22-d9bd1681b8a0" containerID="dbf83f3f7e181237d23f1661ea38da051cb0b66dfc33975236f2dee22e8de1a8" exitCode=0
Oct 30 13:15:31.631394 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:31.631213    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fafb3f-cf6wb" event={"ID":"17f6dcdc-6416-44e4-8d22-d9bd1681b8a0","Type":"ContainerDied","Data":"dbf83f3f7e181237d23f1661ea38da051cb0b66dfc33975236f2dee22e8de1a8"}
Oct 30 13:15:31.635781 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:31.635758    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9fcb69-6f45cd6b55-7z62k" event={"ID":"287c395c-95cf-4133-9319-f0a2c3ae60b6","Type":"ContainerStarted","Data":"1c338424063abb80f185f2da4d48fded152550e40982c7a6c76063712e6cfbca"}
Oct 30 13:15:32.684444 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:32.684404    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9fcb69-6f45cd6b55-7z62k" event={"ID":"287c395c-95cf-4133-9319-f0a2c3ae60b6","Type":"ContainerStarted","Data":"40cc225669b3ca2888983b08e228ec1f5f0bbc8bf8270bbe1d72b373579005ac"}
Oct 30 13:15:32.685001 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:32.684481    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9fcb69-6f45cd6b55-7z62k"
Oct 30 13:15:32.733104 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:32.733046    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9fcb69-6f45cd6b55-7z62k" podStartSLOduration=30.389484888 podStartE2EDuration="30.733029535s" totalImagesPullingTime="343.544647ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:32.725946872 +0000 UTC m=+1431.892311519" watchObservedRunningTime="2026-10-30 13:15:32.733029535 +0000 UTC m=+1431.899394183"
Oct 30 13:15:32.869114 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:32.869090    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fafb3f-cf6wb"
Oct 30 13:15:32.950719 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:32.937749    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17f6dcdc-6416-44e4-8d22-d9bd1681b8a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17f6dcdc-6416-44e4-8d22-d9bd1681b8a0-maas-api-openshift-ca\") pod \"17f6dcdc-6416-44e4-8d22-d9bd1681b8a0\" (UID: \"17f6dcdc-6416-44e4-8d22-d9bd1681b8a0\") "
Oct 30 13:15:32.950719 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:32.937886    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17f6dcdc-6416-44e4-8d22-d9bd1681b8a0-kube-api-access-s84mg\" (UniqueName: \"kubernetes.io/projected/17f6dcdc-6416-44e4-8d22-d9bd1681b8a0-kube-api-access-s84mg\") pod \"17f6dcdc-6416-44e4-8d22-d9bd1681b8a0\" (UID: \"17f6dcdc-6416-44e4-8d22-d9bd1681b8a0\") "
Oct 30 13:15:32.950719 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:32.938038    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17f6dcdc-6416-44e4-8d22-d9bd1681b8a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17f6dcdc-6416-44e4-8d22-d9bd1681b8a0-maas-api-xks-ca\") pod \"17f6dcdc-6416-44e4-8d22-d9bd1681b8a0\" (UID: \"17f6dcdc-6416-44e4-8d22-d9bd1681b8a0\") "
Oct 30 13:15:32.974642 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:32.969016    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17f6dcdc-6416-44e4-8d22-d9bd1681b8a0-maas-api-openshift-ca" pod "17f6dcdc-6416-44e4-8d22-d9bd1681b8a0" (UID: "17f6dcdc-6416-44e4-8d22-d9bd1681b8a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:33.010074 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:33.010034    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17f6dcdc-6416-44e4-8d22-d9bd1681b8a0-kube-api-access-s84mg" pod "17f6dcdc-6416-44e4-8d22-d9bd1681b8a0" (UID: "17f6dcdc-6416-44e4-8d22-d9bd1681b8a0"). InnerVolumeSpecName "kube-api-access-s84mg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:33.036190 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:33.036138    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17f6dcdc-6416-44e4-8d22-d9bd1681b8a0-maas-api-xks-ca" pod "17f6dcdc-6416-44e4-8d22-d9bd1681b8a0" (UID: "17f6dcdc-6416-44e4-8d22-d9bd1681b8a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:15:33.040757 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:33.040676    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17f6dcdc-6416-44e4-8d22-d9bd1681b8a0-maas-api-openshift-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:15:33.040757 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:33.040707    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s84mg\" (UniqueName: \"kubernetes.io/projected/17f6dcdc-6416-44e4-8d22-d9bd1681b8a0-kube-api-access-s84mg\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:15:33.040757 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:33.040722    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17f6dcdc-6416-44e4-8d22-d9bd1681b8a0-maas-api-xks-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:15:33.054711 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:33.054671    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b835314b-3c09-4143-966b-928a86200e6c" path="/var/lib/kubelet/pods/b835314b-3c09-4143-966b-928a86200e6c/volumes"
Oct 30 13:15:33.693273 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:33.693237    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fafb3f-cf6wb" event={"ID":"17f6dcdc-6416-44e4-8d22-d9bd1681b8a0","Type":"ContainerDied","Data":"5272156ea61d52fc38333ed1be057e289da218b22de1d67aec5c4594713b72e9"}
Oct 30 13:15:33.693692 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:33.693295    4364 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5272156ea61d52fc38333ed1be057e289da218b22de1d67aec5c4594713b72e9"
Oct 30 13:15:33.693692 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:33.693256    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fafb3f-cf6wb"
Oct 30 13:15:36.292231 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:36.292200    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3d3ba8b-q2sw5"]
Oct 30 13:15:36.293031 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:36.292998    4364 cpu_manager.go:414] "containerMap: removing container" podUID="17f6dcdc-6416-44e4-8d22-d9bd1681b8a0" containerName="revoke-keys"
Oct 30 13:15:36.293031 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:36.293017    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17f6dcdc-6416-44e4-8d22-d9bd1681b8a0" containerName="revoke-keys"
Oct 30 13:15:36.293166 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:36.293157    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="17f6dcdc-6416-44e4-8d22-d9bd1681b8a0" containerName="revoke-keys"
Oct 30 13:15:36.296668 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:36.296649    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d3ba8b-q2sw5"
Oct 30 13:15:36.316990 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:36.316966    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3d3ba8b-q2sw5"]
Oct 30 13:15:36.367482 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:36.367452    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:15:36.367619 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:36.367503    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\""
Oct 30 13:15:36.442890 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:36.442868    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a16d3810-dcc1-480d-b421-1f5ee1ac3da6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3d3ba8b-q2sw5\" (UID: \"a16d3810-dcc1-480d-b421-1f5ee1ac3da6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d3ba8b-q2sw5"
Oct 30 13:15:36.443006 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:36.442902    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qtnq\" (UniqueName: \"kubernetes.io/projected/a16d3810-dcc1-480d-b421-1f5ee1ac3da6-kube-api-access-4qtnq\") pod \"maas-api-revoke-sub-a3d3ba8b-q2sw5\" (UID: \"a16d3810-dcc1-480d-b421-1f5ee1ac3da6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d3ba8b-q2sw5"
Oct 30 13:15:36.443075 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:36.443056    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a16d3810-dcc1-480d-b421-1f5ee1ac3da6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3d3ba8b-q2sw5\" (UID: \"a16d3810-dcc1-480d-b421-1f5ee1ac3da6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d3ba8b-q2sw5"
Oct 30 13:15:36.544407 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:36.544350    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a16d3810-dcc1-480d-b421-1f5ee1ac3da6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3d3ba8b-q2sw5\" (UID: \"a16d3810-dcc1-480d-b421-1f5ee1ac3da6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d3ba8b-q2sw5"
Oct 30 13:15:36.544407 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:36.544382    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qtnq\" (UniqueName: \"kubernetes.io/projected/a16d3810-dcc1-480d-b421-1f5ee1ac3da6-kube-api-access-4qtnq\") pod \"maas-api-revoke-sub-a3d3ba8b-q2sw5\" (UID: \"a16d3810-dcc1-480d-b421-1f5ee1ac3da6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d3ba8b-q2sw5"
Oct 30 13:15:36.544538 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:36.544466    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a16d3810-dcc1-480d-b421-1f5ee1ac3da6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3d3ba8b-q2sw5\" (UID: \"a16d3810-dcc1-480d-b421-1f5ee1ac3da6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d3ba8b-q2sw5"
Oct 30 13:15:36.545100 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:36.545080    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a16d3810-dcc1-480d-b421-1f5ee1ac3da6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3d3ba8b-q2sw5\" (UID: \"a16d3810-dcc1-480d-b421-1f5ee1ac3da6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d3ba8b-q2sw5"
Oct 30 13:15:36.546653 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:36.546627    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a16d3810-dcc1-480d-b421-1f5ee1ac3da6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3d3ba8b-q2sw5\" (UID: \"a16d3810-dcc1-480d-b421-1f5ee1ac3da6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d3ba8b-q2sw5"
Oct 30 13:15:36.570954 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:36.570927    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qtnq\" (UniqueName: \"kubernetes.io/projected/a16d3810-dcc1-480d-b421-1f5ee1ac3da6-kube-api-access-4qtnq\") pod \"maas-api-revoke-sub-a3d3ba8b-q2sw5\" (UID: \"a16d3810-dcc1-480d-b421-1f5ee1ac3da6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d3ba8b-q2sw5"
Oct 30 13:15:36.609466 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:36.609446    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d3ba8b-q2sw5"
Oct 30 13:15:36.947174 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:36.947151    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3d3ba8b-q2sw5"]
Oct 30 13:15:36.948959 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:15:36.948929    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda16d3810_dcc1_480d_b421_1f5ee1ac3da6.slice/crio-54b2b09c8d1133fa71f75b52f2d4e0b1e78012572591c9b20cca258eabf99bb7 WatchSource:0}: Error finding container 54b2b09c8d1133fa71f75b52f2d4e0b1e78012572591c9b20cca258eabf99bb7: Status 404 returned error can't find the container with id 54b2b09c8d1133fa71f75b52f2d4e0b1e78012572591c9b20cca258eabf99bb7
Oct 30 13:15:37.733000 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:37.732917    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d3ba8b-q2sw5" event={"ID":"a16d3810-dcc1-480d-b421-1f5ee1ac3da6","Type":"ContainerStarted","Data":"e1949dc3f91430f727412b23553db768137f2138714f69fa30c6f588196dd9c8"}
Oct 30 13:15:37.733000 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:37.732967    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d3ba8b-q2sw5" event={"ID":"a16d3810-dcc1-480d-b421-1f5ee1ac3da6","Type":"ContainerStarted","Data":"54b2b09c8d1133fa71f75b52f2d4e0b1e78012572591c9b20cca258eabf99bb7"}
Oct 30 13:15:37.774424 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:37.774366    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d3ba8b-q2sw5" podStartSLOduration=1.7743495710000001 podStartE2EDuration="1.774349571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:37.773073628 +0000 UTC m=+1436.939438256" watchObservedRunningTime="2026-10-30 13:15:37.774349571 +0000 UTC m=+1436.940714214"
Oct 30 13:15:38.755527 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:38.755492    4364 generic.go:356] "Generic (PLEG): container finished" podID="a16d3810-dcc1-480d-b421-1f5ee1ac3da6" containerID="e1949dc3f91430f727412b23553db768137f2138714f69fa30c6f588196dd9c8" exitCode=0
Oct 30 13:15:38.756052 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:38.755764    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d3ba8b-q2sw5" event={"ID":"a16d3810-dcc1-480d-b421-1f5ee1ac3da6","Type":"ContainerDied","Data":"e1949dc3f91430f727412b23553db768137f2138714f69fa30c6f588196dd9c8"}
Oct 30 13:15:39.423206 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:39.423171    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:15:39.749676 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:39.746785    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9fcb69-6f45cd6b55-7z62k"
Oct 30 13:15:40.110155 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:40.110128    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d3ba8b-q2sw5"
Oct 30 13:15:40.228745 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:40.228701    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fafb3f-7674fdb99d-v9prm"]
Oct 30 13:15:40.255562 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:40.255481    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fafb3f-788c86bfc9-s6m45"]
Oct 30 13:15:40.283380 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:40.283349    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a16d3810-dcc1-480d-b421-1f5ee1ac3da6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a16d3810-dcc1-480d-b421-1f5ee1ac3da6-maas-api-xks-ca\") pod \"a16d3810-dcc1-480d-b421-1f5ee1ac3da6\" (UID: \"a16d3810-dcc1-480d-b421-1f5ee1ac3da6\") "
Oct 30 13:15:40.283744 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:40.283707    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a16d3810-dcc1-480d-b421-1f5ee1ac3da6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a16d3810-dcc1-480d-b421-1f5ee1ac3da6-maas-api-openshift-ca\") pod \"a16d3810-dcc1-480d-b421-1f5ee1ac3da6\" (UID: \"a16d3810-dcc1-480d-b421-1f5ee1ac3da6\") "
Oct 30 13:15:40.283834 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:40.283755    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a16d3810-dcc1-480d-b421-1f5ee1ac3da6-kube-api-access-4qtnq\" (UniqueName: \"kubernetes.io/projected/a16d3810-dcc1-480d-b421-1f5ee1ac3da6-kube-api-access-4qtnq\") pod \"a16d3810-dcc1-480d-b421-1f5ee1ac3da6\" (UID: \"a16d3810-dcc1-480d-b421-1f5ee1ac3da6\") "
Oct 30 13:15:40.284106 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:40.284081    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a16d3810-dcc1-480d-b421-1f5ee1ac3da6-maas-api-openshift-ca" pod "a16d3810-dcc1-480d-b421-1f5ee1ac3da6" (UID: "a16d3810-dcc1-480d-b421-1f5ee1ac3da6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:40.284678 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:40.284654    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a16d3810-dcc1-480d-b421-1f5ee1ac3da6-maas-api-openshift-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:15:40.285532 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:40.285507    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a16d3810-dcc1-480d-b421-1f5ee1ac3da6-maas-api-xks-ca" pod "a16d3810-dcc1-480d-b421-1f5ee1ac3da6" (UID: "a16d3810-dcc1-480d-b421-1f5ee1ac3da6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:15:40.286014 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:40.285996    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a16d3810-dcc1-480d-b421-1f5ee1ac3da6-kube-api-access-4qtnq" pod "a16d3810-dcc1-480d-b421-1f5ee1ac3da6" (UID: "a16d3810-dcc1-480d-b421-1f5ee1ac3da6"). InnerVolumeSpecName "kube-api-access-4qtnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:40.386187 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:40.386150    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qtnq\" (UniqueName: \"kubernetes.io/projected/a16d3810-dcc1-480d-b421-1f5ee1ac3da6-kube-api-access-4qtnq\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:15:40.386187 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:40.386181    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a16d3810-dcc1-480d-b421-1f5ee1ac3da6-maas-api-xks-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:15:40.935535 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:40.935471    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d3ba8b-q2sw5" event={"ID":"a16d3810-dcc1-480d-b421-1f5ee1ac3da6","Type":"ContainerDied","Data":"54b2b09c8d1133fa71f75b52f2d4e0b1e78012572591c9b20cca258eabf99bb7"}
Oct 30 13:15:40.935845 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:40.935812    4364 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54b2b09c8d1133fa71f75b52f2d4e0b1e78012572591c9b20cca258eabf99bb7"
Oct 30 13:15:40.936068 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:40.936047    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d3ba8b-q2sw5"
Oct 30 13:15:41.577224 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:15:41.577185    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:15:41.672034 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:41.672002    4364 scope.go:122] "RemoveContainer" containerID="9dd06988e349d811dd5a71a037240fbdb54d3abcc357d962888f58f162e973a7"
Oct 30 13:15:41.683108 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:41.683085    4364 scope.go:122] "RemoveContainer" containerID="7610e96aa9718ac6ce70b64cfcc98ed375dbac57c14cdfebf0b741a9925ef2d9"
Oct 30 13:15:42.496112 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:42.496080    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2daeae-76cf799c89-c8sgx"]
Oct 30 13:15:42.496962 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:42.496941    4364 cpu_manager.go:414] "containerMap: removing container" podUID="a16d3810-dcc1-480d-b421-1f5ee1ac3da6" containerName="revoke-keys"
Oct 30 13:15:42.496962 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:42.496958    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a16d3810-dcc1-480d-b421-1f5ee1ac3da6" containerName="revoke-keys"
Oct 30 13:15:42.497133 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:42.497099    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="a16d3810-dcc1-480d-b421-1f5ee1ac3da6" containerName="revoke-keys"
Oct 30 13:15:42.501142 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:42.501126    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2daeae-76cf799c89-c8sgx"
Oct 30 13:15:42.529520 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:42.529489    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-2daeae-serving-cert\""
Oct 30 13:15:42.539647 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:42.538615    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2daeae-76cf799c89-c8sgx"]
Oct 30 13:15:42.590088 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:42.590049    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80cbfd1b-0a69-4fdb-880b-561f4bc6cd90-maas-api-tls\") pod \"maas-api-e2e-api-b-2daeae-76cf799c89-c8sgx\" (UID: \"80cbfd1b-0a69-4fdb-880b-561f4bc6cd90\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2daeae-76cf799c89-c8sgx"
Oct 30 13:15:42.590591 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:42.590272    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80cbfd1b-0a69-4fdb-880b-561f4bc6cd90-metrics-certs\") pod \"maas-api-e2e-api-b-2daeae-76cf799c89-c8sgx\" (UID: \"80cbfd1b-0a69-4fdb-880b-561f4bc6cd90\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2daeae-76cf799c89-c8sgx"
Oct 30 13:15:42.590775 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:42.590586    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80cbfd1b-0a69-4fdb-880b-561f4bc6cd90-openshift-service-ca\") pod \"maas-api-e2e-api-b-2daeae-76cf799c89-c8sgx\" (UID: \"80cbfd1b-0a69-4fdb-880b-561f4bc6cd90\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2daeae-76cf799c89-c8sgx"
Oct 30 13:15:42.590867 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:42.590850    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flsxc\" (UniqueName: \"kubernetes.io/projected/80cbfd1b-0a69-4fdb-880b-561f4bc6cd90-kube-api-access-flsxc\") pod \"maas-api-e2e-api-b-2daeae-76cf799c89-c8sgx\" (UID: \"80cbfd1b-0a69-4fdb-880b-561f4bc6cd90\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2daeae-76cf799c89-c8sgx"
Oct 30 13:15:42.691911 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:42.691877    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80cbfd1b-0a69-4fdb-880b-561f4bc6cd90-openshift-service-ca\") pod \"maas-api-e2e-api-b-2daeae-76cf799c89-c8sgx\" (UID: \"80cbfd1b-0a69-4fdb-880b-561f4bc6cd90\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2daeae-76cf799c89-c8sgx"
Oct 30 13:15:42.692099 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:42.691948    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flsxc\" (UniqueName: \"kubernetes.io/projected/80cbfd1b-0a69-4fdb-880b-561f4bc6cd90-kube-api-access-flsxc\") pod \"maas-api-e2e-api-b-2daeae-76cf799c89-c8sgx\" (UID: \"80cbfd1b-0a69-4fdb-880b-561f4bc6cd90\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2daeae-76cf799c89-c8sgx"
Oct 30 13:15:42.692099 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:42.692008    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80cbfd1b-0a69-4fdb-880b-561f4bc6cd90-maas-api-tls\") pod \"maas-api-e2e-api-b-2daeae-76cf799c89-c8sgx\" (UID: \"80cbfd1b-0a69-4fdb-880b-561f4bc6cd90\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2daeae-76cf799c89-c8sgx"
Oct 30 13:15:42.692178 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:42.692102    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80cbfd1b-0a69-4fdb-880b-561f4bc6cd90-metrics-certs\") pod \"maas-api-e2e-api-b-2daeae-76cf799c89-c8sgx\" (UID: \"80cbfd1b-0a69-4fdb-880b-561f4bc6cd90\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2daeae-76cf799c89-c8sgx"
Oct 30 13:15:42.692731 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:42.692700    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80cbfd1b-0a69-4fdb-880b-561f4bc6cd90-openshift-service-ca\") pod \"maas-api-e2e-api-b-2daeae-76cf799c89-c8sgx\" (UID: \"80cbfd1b-0a69-4fdb-880b-561f4bc6cd90\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2daeae-76cf799c89-c8sgx"
Oct 30 13:15:42.694821 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:42.694800    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80cbfd1b-0a69-4fdb-880b-561f4bc6cd90-metrics-certs\") pod \"maas-api-e2e-api-b-2daeae-76cf799c89-c8sgx\" (UID: \"80cbfd1b-0a69-4fdb-880b-561f4bc6cd90\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2daeae-76cf799c89-c8sgx"
Oct 30 13:15:42.694821 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:42.694812    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80cbfd1b-0a69-4fdb-880b-561f4bc6cd90-maas-api-tls\") pod \"maas-api-e2e-api-b-2daeae-76cf799c89-c8sgx\" (UID: \"80cbfd1b-0a69-4fdb-880b-561f4bc6cd90\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2daeae-76cf799c89-c8sgx"
Oct 30 13:15:42.716027 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:42.716002    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flsxc\" (UniqueName: \"kubernetes.io/projected/80cbfd1b-0a69-4fdb-880b-561f4bc6cd90-kube-api-access-flsxc\") pod \"maas-api-e2e-api-b-2daeae-76cf799c89-c8sgx\" (UID: \"80cbfd1b-0a69-4fdb-880b-561f4bc6cd90\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2daeae-76cf799c89-c8sgx"
Oct 30 13:15:42.832692 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:42.832004    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2daeae-76cf799c89-c8sgx"
Oct 30 13:15:43.350780 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:43.350591    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2daeae-76cf799c89-c8sgx"]
Oct 30 13:15:43.350947 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:15:43.350926    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80cbfd1b_0a69_4fdb_880b_561f4bc6cd90.slice/crio-89ce86cb66c725d68fa3be87e1ceff28dc20e7be3083bbf297dbc4a5f9e651af WatchSource:0}: Error finding container 89ce86cb66c725d68fa3be87e1ceff28dc20e7be3083bbf297dbc4a5f9e651af: Status 404 returned error can't find the container with id 89ce86cb66c725d68fa3be87e1ceff28dc20e7be3083bbf297dbc4a5f9e651af
Oct 30 13:15:43.990806 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:43.990719    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2daeae-76cf799c89-c8sgx" event={"ID":"80cbfd1b-0a69-4fdb-880b-561f4bc6cd90","Type":"ContainerStarted","Data":"22bd6fcb36d43d58c0b38c39f53d8187b51a15136ce086d2afbdfa84f7cf71f1"}
Oct 30 13:15:43.990806 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:43.990774    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2daeae-76cf799c89-c8sgx" event={"ID":"80cbfd1b-0a69-4fdb-880b-561f4bc6cd90","Type":"ContainerStarted","Data":"89ce86cb66c725d68fa3be87e1ceff28dc20e7be3083bbf297dbc4a5f9e651af"}
Oct 30 13:15:43.990806 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:43.990808    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2daeae-76cf799c89-c8sgx"
Oct 30 13:15:44.054348 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:44.054297    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2daeae-76cf799c89-c8sgx" podStartSLOduration=1.6687252 podStartE2EDuration="2.05428201s" totalImagesPullingTime="385.55681ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:44.050723251 +0000 UTC m=+1443.217087898" watchObservedRunningTime="2026-10-30 13:15:44.05428201 +0000 UTC m=+1443.220646723"
Oct 30 13:15:44.182054 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:44.182022    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-84942a-openshift-default-58974cd687-wtjlv"]
Oct 30 13:15:44.186691 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:44.186663    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-84942a-openshift-default-58974cd687-wtjlv"
Oct 30 13:15:44.232322 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:44.232282    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-84942a-openshift-default-58974cd687-wtjlv"]
Oct 30 13:15:44.235998 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:44.235962    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/900b00e0-ab94-4293-ade2-7e89bd313fea-istiod-ca-cert\") pod \"e2e-api-b-84942a-openshift-default-58974cd687-wtjlv\" (UID: \"900b00e0-ab94-4293-ade2-7e89bd313fea\") " pod="openshift-ingress/e2e-api-b-84942a-openshift-default-58974cd687-wtjlv"
Oct 30 13:15:44.236161 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:44.236062    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdsjz\" (UniqueName: \"kubernetes.io/projected/900b00e0-ab94-4293-ade2-7e89bd313fea-kube-api-access-pdsjz\") pod \"e2e-api-b-84942a-openshift-default-58974cd687-wtjlv\" (UID: \"900b00e0-ab94-4293-ade2-7e89bd313fea\") " pod="openshift-ingress/e2e-api-b-84942a-openshift-default-58974cd687-wtjlv"
Oct 30 13:15:44.236243 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:44.236222    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/900b00e0-ab94-4293-ade2-7e89bd313fea-istio-data\") pod \"e2e-api-b-84942a-openshift-default-58974cd687-wtjlv\" (UID: \"900b00e0-ab94-4293-ade2-7e89bd313fea\") " pod="openshift-ingress/e2e-api-b-84942a-openshift-default-58974cd687-wtjlv"
Oct 30 13:15:44.236347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:44.236330    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/900b00e0-ab94-4293-ade2-7e89bd313fea-workload-socket\") pod \"e2e-api-b-84942a-openshift-default-58974cd687-wtjlv\" (UID: \"900b00e0-ab94-4293-ade2-7e89bd313fea\") " pod="openshift-ingress/e2e-api-b-84942a-openshift-default-58974cd687-wtjlv"
Oct 30 13:15:44.236536 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:44.236517    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/900b00e0-ab94-4293-ade2-7e89bd313fea-credential-socket\") pod \"e2e-api-b-84942a-openshift-default-58974cd687-wtjlv\" (UID: \"900b00e0-ab94-4293-ade2-7e89bd313fea\") " pod="openshift-ingress/e2e-api-b-84942a-openshift-default-58974cd687-wtjlv"
Oct 30 13:15:44.236625 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:44.236550    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/900b00e0-ab94-4293-ade2-7e89bd313fea-workload-certs\") pod \"e2e-api-b-84942a-openshift-default-58974cd687-wtjlv\" (UID: \"900b00e0-ab94-4293-ade2-7e89bd313fea\") " pod="openshift-ingress/e2e-api-b-84942a-openshift-default-58974cd687-wtjlv"
Oct 30 13:15:44.236920 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:44.236701    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/900b00e0-ab94-4293-ade2-7e89bd313fea-istio-token\") pod \"e2e-api-b-84942a-openshift-default-58974cd687-wtjlv\" (UID: \"900b00e0-ab94-4293-ade2-7e89bd313fea\") " pod="openshift-ingress/e2e-api-b-84942a-openshift-default-58974cd687-wtjlv"
Oct 30 13:15:44.236920 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:44.236824    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/900b00e0-ab94-4293-ade2-7e89bd313fea-istio-podinfo\") pod \"e2e-api-b-84942a-openshift-default-58974cd687-wtjlv\" (UID: \"900b00e0-ab94-4293-ade2-7e89bd313fea\") " pod="openshift-ingress/e2e-api-b-84942a-openshift-default-58974cd687-wtjlv"
Oct 30 13:15:44.236920 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:44.236860    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/900b00e0-ab94-4293-ade2-7e89bd313fea-istio-envoy\") pod \"e2e-api-b-84942a-openshift-default-58974cd687-wtjlv\" (UID: \"900b00e0-ab94-4293-ade2-7e89bd313fea\") " pod="openshift-ingress/e2e-api-b-84942a-openshift-default-58974cd687-wtjlv"
Oct 30 13:15:44.287173 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:44.287080    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-84942a-openshift-default-dockercfg-phvck\""
Oct 30 13:15:44.338115 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:44.338080    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/900b00e0-ab94-4293-ade2-7e89bd313fea-workload-socket\") pod \"e2e-api-b-84942a-openshift-default-58974cd687-wtjlv\" (UID: \"900b00e0-ab94-4293-ade2-7e89bd313fea\") " pod="openshift-ingress/e2e-api-b-84942a-openshift-default-58974cd687-wtjlv"
Oct 30 13:15:44.338289 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:44.338246    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/900b00e0-ab94-4293-ade2-7e89bd313fea-credential-socket\") pod \"e2e-api-b-84942a-openshift-default-58974cd687-wtjlv\" (UID: \"900b00e0-ab94-4293-ade2-7e89bd313fea\") " pod="openshift-ingress/e2e-api-b-84942a-openshift-default-58974cd687-wtjlv"
Oct 30 13:15:44.338289 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:44.338270    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/900b00e0-ab94-4293-ade2-7e89bd313fea-workload-certs\") pod \"e2e-api-b-84942a-openshift-default-58974cd687-wtjlv\" (UID: \"900b00e0-ab94-4293-ade2-7e89bd313fea\") " pod="openshift-ingress/e2e-api-b-84942a-openshift-default-58974cd687-wtjlv"
Oct 30 13:15:44.338372 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:44.338300    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/900b00e0-ab94-4293-ade2-7e89bd313fea-istio-token\") pod \"e2e-api-b-84942a-openshift-default-58974cd687-wtjlv\" (UID: \"900b00e0-ab94-4293-ade2-7e89bd313fea\") " pod="openshift-ingress/e2e-api-b-84942a-openshift-default-58974cd687-wtjlv"
Oct 30 13:15:44.338407 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:44.338397    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/900b00e0-ab94-4293-ade2-7e89bd313fea-istio-podinfo\") pod \"e2e-api-b-84942a-openshift-default-58974cd687-wtjlv\" (UID: \"900b00e0-ab94-4293-ade2-7e89bd313fea\") " pod="openshift-ingress/e2e-api-b-84942a-openshift-default-58974cd687-wtjlv"
Oct 30 13:15:44.338440 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:44.338429    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/900b00e0-ab94-4293-ade2-7e89bd313fea-istio-envoy\") pod \"e2e-api-b-84942a-openshift-default-58974cd687-wtjlv\" (UID: \"900b00e0-ab94-4293-ade2-7e89bd313fea\") " pod="openshift-ingress/e2e-api-b-84942a-openshift-default-58974cd687-wtjlv"
Oct 30 13:15:44.338508 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:44.338484    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/900b00e0-ab94-4293-ade2-7e89bd313fea-workload-socket\") pod \"e2e-api-b-84942a-openshift-default-58974cd687-wtjlv\" (UID: \"900b00e0-ab94-4293-ade2-7e89bd313fea\") " pod="openshift-ingress/e2e-api-b-84942a-openshift-default-58974cd687-wtjlv"
Oct 30 13:15:44.338565 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:44.338532    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/900b00e0-ab94-4293-ade2-7e89bd313fea-istiod-ca-cert\") pod \"e2e-api-b-84942a-openshift-default-58974cd687-wtjlv\" (UID: \"900b00e0-ab94-4293-ade2-7e89bd313fea\") " pod="openshift-ingress/e2e-api-b-84942a-openshift-default-58974cd687-wtjlv"
Oct 30 13:15:44.338664 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:44.338570    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdsjz\" (UniqueName: \"kubernetes.io/projected/900b00e0-ab94-4293-ade2-7e89bd313fea-kube-api-access-pdsjz\") pod \"e2e-api-b-84942a-openshift-default-58974cd687-wtjlv\" (UID: \"900b00e0-ab94-4293-ade2-7e89bd313fea\") " pod="openshift-ingress/e2e-api-b-84942a-openshift-default-58974cd687-wtjlv"
Oct 30 13:15:44.338664 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:44.338648    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/900b00e0-ab94-4293-ade2-7e89bd313fea-credential-socket\") pod \"e2e-api-b-84942a-openshift-default-58974cd687-wtjlv\" (UID: \"900b00e0-ab94-4293-ade2-7e89bd313fea\") " pod="openshift-ingress/e2e-api-b-84942a-openshift-default-58974cd687-wtjlv"
Oct 30 13:15:44.338783 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:44.338701    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/900b00e0-ab94-4293-ade2-7e89bd313fea-workload-certs\") pod \"e2e-api-b-84942a-openshift-default-58974cd687-wtjlv\" (UID: \"900b00e0-ab94-4293-ade2-7e89bd313fea\") " pod="openshift-ingress/e2e-api-b-84942a-openshift-default-58974cd687-wtjlv"
Oct 30 13:15:44.338848 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:44.338775    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/900b00e0-ab94-4293-ade2-7e89bd313fea-istio-data\") pod \"e2e-api-b-84942a-openshift-default-58974cd687-wtjlv\" (UID: \"900b00e0-ab94-4293-ade2-7e89bd313fea\") " pod="openshift-ingress/e2e-api-b-84942a-openshift-default-58974cd687-wtjlv"
Oct 30 13:15:44.339138 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:44.339117    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/900b00e0-ab94-4293-ade2-7e89bd313fea-istio-data\") pod \"e2e-api-b-84942a-openshift-default-58974cd687-wtjlv\" (UID: \"900b00e0-ab94-4293-ade2-7e89bd313fea\") " pod="openshift-ingress/e2e-api-b-84942a-openshift-default-58974cd687-wtjlv"
Oct 30 13:15:44.339276 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:44.339177    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/900b00e0-ab94-4293-ade2-7e89bd313fea-istiod-ca-cert\") pod \"e2e-api-b-84942a-openshift-default-58974cd687-wtjlv\" (UID: \"900b00e0-ab94-4293-ade2-7e89bd313fea\") " pod="openshift-ingress/e2e-api-b-84942a-openshift-default-58974cd687-wtjlv"
Oct 30 13:15:44.341000 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:44.340969    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/900b00e0-ab94-4293-ade2-7e89bd313fea-istio-envoy\") pod \"e2e-api-b-84942a-openshift-default-58974cd687-wtjlv\" (UID: \"900b00e0-ab94-4293-ade2-7e89bd313fea\") " pod="openshift-ingress/e2e-api-b-84942a-openshift-default-58974cd687-wtjlv"
Oct 30 13:15:44.341229 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:44.341211    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/900b00e0-ab94-4293-ade2-7e89bd313fea-istio-podinfo\") pod \"e2e-api-b-84942a-openshift-default-58974cd687-wtjlv\" (UID: \"900b00e0-ab94-4293-ade2-7e89bd313fea\") " pod="openshift-ingress/e2e-api-b-84942a-openshift-default-58974cd687-wtjlv"
Oct 30 13:15:44.361675 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:44.361647    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/900b00e0-ab94-4293-ade2-7e89bd313fea-istio-token\") pod \"e2e-api-b-84942a-openshift-default-58974cd687-wtjlv\" (UID: \"900b00e0-ab94-4293-ade2-7e89bd313fea\") " pod="openshift-ingress/e2e-api-b-84942a-openshift-default-58974cd687-wtjlv"
Oct 30 13:15:44.361972 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:44.361952    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdsjz\" (UniqueName: \"kubernetes.io/projected/900b00e0-ab94-4293-ade2-7e89bd313fea-kube-api-access-pdsjz\") pod \"e2e-api-b-84942a-openshift-default-58974cd687-wtjlv\" (UID: \"900b00e0-ab94-4293-ade2-7e89bd313fea\") " pod="openshift-ingress/e2e-api-b-84942a-openshift-default-58974cd687-wtjlv"
Oct 30 13:15:44.512971 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:44.512933    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-84942a-openshift-default-58974cd687-wtjlv"
Oct 30 13:15:45.198257 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:15:45.198177    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod900b00e0_ab94_4293_ade2_7e89bd313fea.slice/crio-27a334d3a761c8a7888c29a7a1c5d444fe3dec8fe120d281cc075c01d8545d3b WatchSource:0}: Error finding container 27a334d3a761c8a7888c29a7a1c5d444fe3dec8fe120d281cc075c01d8545d3b: Status 404 returned error can't find the container with id 27a334d3a761c8a7888c29a7a1c5d444fe3dec8fe120d281cc075c01d8545d3b
Oct 30 13:15:45.198727 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:45.198317    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-84942a-openshift-default-58974cd687-wtjlv"]
Oct 30 13:15:45.201420 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:45.201387    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:15:45.201522 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:45.201452    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:15:45.201522 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:45.201481    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:15:45.246817 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:45.246751    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-fafb3f-7674fdb99d-v9prm" podUID="55402878-108c-4c3d-b6c8-4362f82e82cb" containerName="payload-processing" containerID="cri-o://bf27e7a958b12cdad51267051d4ce1fe68d11bfdf869a28ee97b4c133a56f7e7" gracePeriod=25
Oct 30 13:15:45.256887 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:45.256257    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fafb3f-788c86bfc9-s6m45" podUID="f09ff8b4-d468-4c5e-9265-1f13766720f2" containerName="payload-pre-processing" containerID="cri-o://832487059ca81a31c8508ac93cec717f1c4f2a1b9953a3022f6c98d5f5dbfde8" gracePeriod=25
Oct 30 13:15:45.574729 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:45.574702    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-fafb3f-7674fdb99d-v9prm"
Oct 30 13:15:45.606471 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:45.606435    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55402878-108c-4c3d-b6c8-4362f82e82cb-kube-api-access-c6j6m\" (UniqueName: \"kubernetes.io/projected/55402878-108c-4c3d-b6c8-4362f82e82cb-kube-api-access-c6j6m\") pod \"55402878-108c-4c3d-b6c8-4362f82e82cb\" (UID: \"55402878-108c-4c3d-b6c8-4362f82e82cb\") "
Oct 30 13:15:45.606689 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:45.606512    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55402878-108c-4c3d-b6c8-4362f82e82cb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55402878-108c-4c3d-b6c8-4362f82e82cb-plugins-config-volume\") pod \"55402878-108c-4c3d-b6c8-4362f82e82cb\" (UID: \"55402878-108c-4c3d-b6c8-4362f82e82cb\") "
Oct 30 13:15:45.607030 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:45.607003    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55402878-108c-4c3d-b6c8-4362f82e82cb-plugins-config-volume" pod "55402878-108c-4c3d-b6c8-4362f82e82cb" (UID: "55402878-108c-4c3d-b6c8-4362f82e82cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:45.607518 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:45.607494    4364 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55402878-108c-4c3d-b6c8-4362f82e82cb-plugins-config-volume\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:15:45.610116 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:45.610083    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55402878-108c-4c3d-b6c8-4362f82e82cb-kube-api-access-c6j6m" pod "55402878-108c-4c3d-b6c8-4362f82e82cb" (UID: "55402878-108c-4c3d-b6c8-4362f82e82cb"). InnerVolumeSpecName "kube-api-access-c6j6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:45.631532 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:45.631506    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fafb3f-788c86bfc9-s6m45"
Oct 30 13:15:45.709738 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:45.709682    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f09ff8b4-d468-4c5e-9265-1f13766720f2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f09ff8b4-d468-4c5e-9265-1f13766720f2-plugins-config-volume\") pod \"f09ff8b4-d468-4c5e-9265-1f13766720f2\" (UID: \"f09ff8b4-d468-4c5e-9265-1f13766720f2\") "
Oct 30 13:15:45.710151 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:45.710112    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f09ff8b4-d468-4c5e-9265-1f13766720f2-kube-api-access-4t2z7\" (UniqueName: \"kubernetes.io/projected/f09ff8b4-d468-4c5e-9265-1f13766720f2-kube-api-access-4t2z7\") pod \"f09ff8b4-d468-4c5e-9265-1f13766720f2\" (UID: \"f09ff8b4-d468-4c5e-9265-1f13766720f2\") "
Oct 30 13:15:45.711347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:45.710151    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f09ff8b4-d468-4c5e-9265-1f13766720f2-plugins-config-volume" pod "f09ff8b4-d468-4c5e-9265-1f13766720f2" (UID: "f09ff8b4-d468-4c5e-9265-1f13766720f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:45.711347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:45.711296    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6j6m\" (UniqueName: \"kubernetes.io/projected/55402878-108c-4c3d-b6c8-4362f82e82cb-kube-api-access-c6j6m\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:15:45.711347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:45.711321    4364 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f09ff8b4-d468-4c5e-9265-1f13766720f2-plugins-config-volume\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:15:45.713167 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:45.713076    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f09ff8b4-d468-4c5e-9265-1f13766720f2-kube-api-access-4t2z7" pod "f09ff8b4-d468-4c5e-9265-1f13766720f2" (UID: "f09ff8b4-d468-4c5e-9265-1f13766720f2"). InnerVolumeSpecName "kube-api-access-4t2z7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:45.846150 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:45.846029    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t2z7\" (UniqueName: \"kubernetes.io/projected/f09ff8b4-d468-4c5e-9265-1f13766720f2-kube-api-access-4t2z7\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:15:46.013097 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:46.013057    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-84942a-openshift-default-58974cd687-wtjlv" event={"ID":"900b00e0-ab94-4293-ade2-7e89bd313fea","Type":"ContainerStarted","Data":"d9ce283f4848e306aa5e2b66efbbc4b69e655759246703e2c231008f4c27143f"}
Oct 30 13:15:46.013274 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:46.013123    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-84942a-openshift-default-58974cd687-wtjlv" event={"ID":"900b00e0-ab94-4293-ade2-7e89bd313fea","Type":"ContainerStarted","Data":"27a334d3a761c8a7888c29a7a1c5d444fe3dec8fe120d281cc075c01d8545d3b"}
Oct 30 13:15:46.014572 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:46.014546    4364 generic.go:356] "Generic (PLEG): container finished" podID="f09ff8b4-d468-4c5e-9265-1f13766720f2" containerID="832487059ca81a31c8508ac93cec717f1c4f2a1b9953a3022f6c98d5f5dbfde8" exitCode=0
Oct 30 13:15:46.014717 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:46.014581    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fafb3f-788c86bfc9-s6m45"
Oct 30 13:15:46.014717 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:46.014624    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fafb3f-788c86bfc9-s6m45" event={"ID":"f09ff8b4-d468-4c5e-9265-1f13766720f2","Type":"ContainerDied","Data":"832487059ca81a31c8508ac93cec717f1c4f2a1b9953a3022f6c98d5f5dbfde8"}
Oct 30 13:15:46.014717 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:46.014664    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fafb3f-788c86bfc9-s6m45" event={"ID":"f09ff8b4-d468-4c5e-9265-1f13766720f2","Type":"ContainerDied","Data":"1d30a2958754eb81c21e2af60ac8e4a17dfbfe0cf85e3a9cde13201f39f77a1a"}
Oct 30 13:15:46.014717 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:46.014686    4364 scope.go:122] "RemoveContainer" containerID="832487059ca81a31c8508ac93cec717f1c4f2a1b9953a3022f6c98d5f5dbfde8"
Oct 30 13:15:46.016125 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:46.016096    4364 generic.go:356] "Generic (PLEG): container finished" podID="55402878-108c-4c3d-b6c8-4362f82e82cb" containerID="bf27e7a958b12cdad51267051d4ce1fe68d11bfdf869a28ee97b4c133a56f7e7" exitCode=0
Oct 30 13:15:46.016234 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:46.016159    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-fafb3f-7674fdb99d-v9prm"
Oct 30 13:15:46.016302 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:46.016159    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-fafb3f-7674fdb99d-v9prm" event={"ID":"55402878-108c-4c3d-b6c8-4362f82e82cb","Type":"ContainerDied","Data":"bf27e7a958b12cdad51267051d4ce1fe68d11bfdf869a28ee97b4c133a56f7e7"}
Oct 30 13:15:46.016351 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:46.016299    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-fafb3f-7674fdb99d-v9prm" event={"ID":"55402878-108c-4c3d-b6c8-4362f82e82cb","Type":"ContainerDied","Data":"2782d5ec4f6bc631860172b9ab6a3e01409f2f8e731966253cf9c3e792147a21"}
Oct 30 13:15:46.027304 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:46.027284    4364 scope.go:122] "RemoveContainer" containerID="832487059ca81a31c8508ac93cec717f1c4f2a1b9953a3022f6c98d5f5dbfde8"
Oct 30 13:15:46.027649 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:15:46.027622    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"832487059ca81a31c8508ac93cec717f1c4f2a1b9953a3022f6c98d5f5dbfde8\": container with ID starting with 832487059ca81a31c8508ac93cec717f1c4f2a1b9953a3022f6c98d5f5dbfde8 not found: ID does not exist" containerID="832487059ca81a31c8508ac93cec717f1c4f2a1b9953a3022f6c98d5f5dbfde8"
Oct 30 13:15:46.027754 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:46.027657    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"832487059ca81a31c8508ac93cec717f1c4f2a1b9953a3022f6c98d5f5dbfde8"} err="failed to get container status \"832487059ca81a31c8508ac93cec717f1c4f2a1b9953a3022f6c98d5f5dbfde8\": rpc error: code = NotFound desc = could not find container \"832487059ca81a31c8508ac93cec717f1c4f2a1b9953a3022f6c98d5f5dbfde8\": container with ID starting with 832487059ca81a31c8508ac93cec717f1c4f2a1b9953a3022f6c98d5f5dbfde8 not found: ID does not exist"
Oct 30 13:15:46.027754 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:46.027681    4364 scope.go:122] "RemoveContainer" containerID="bf27e7a958b12cdad51267051d4ce1fe68d11bfdf869a28ee97b4c133a56f7e7"
Oct 30 13:15:46.037666 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:46.037649    4364 scope.go:122] "RemoveContainer" containerID="bf27e7a958b12cdad51267051d4ce1fe68d11bfdf869a28ee97b4c133a56f7e7"
Oct 30 13:15:46.037939 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:15:46.037919    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf27e7a958b12cdad51267051d4ce1fe68d11bfdf869a28ee97b4c133a56f7e7\": container with ID starting with bf27e7a958b12cdad51267051d4ce1fe68d11bfdf869a28ee97b4c133a56f7e7 not found: ID does not exist" containerID="bf27e7a958b12cdad51267051d4ce1fe68d11bfdf869a28ee97b4c133a56f7e7"
Oct 30 13:15:46.038006 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:46.037945    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf27e7a958b12cdad51267051d4ce1fe68d11bfdf869a28ee97b4c133a56f7e7"} err="failed to get container status \"bf27e7a958b12cdad51267051d4ce1fe68d11bfdf869a28ee97b4c133a56f7e7\": rpc error: code = NotFound desc = could not find container \"bf27e7a958b12cdad51267051d4ce1fe68d11bfdf869a28ee97b4c133a56f7e7\": container with ID starting with bf27e7a958b12cdad51267051d4ce1fe68d11bfdf869a28ee97b4c133a56f7e7 not found: ID does not exist"
Oct 30 13:15:46.061720 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:46.061668    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-84942a-openshift-default-58974cd687-wtjlv" podStartSLOduration=2.061654567 podStartE2EDuration="2.061654567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:46.058189545 +0000 UTC m=+1445.224554192" watchObservedRunningTime="2026-10-30 13:15:46.061654567 +0000 UTC m=+1445.228019215"
Oct 30 13:15:46.099906 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:46.099831    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fafb3f-7674fdb99d-v9prm"]
Oct 30 13:15:46.104759 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:46.104739    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fafb3f-7674fdb99d-v9prm"]
Oct 30 13:15:46.134874 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:46.134838    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fafb3f-788c86bfc9-s6m45"]
Oct 30 13:15:46.139498 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:46.139477    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fafb3f-788c86bfc9-s6m45"]
Oct 30 13:15:46.514399 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:46.514366    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-84942a-openshift-default-58974cd687-wtjlv"
Oct 30 13:15:46.520469 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:46.520445    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-84942a-openshift-default-58974cd687-wtjlv"
Oct 30 13:15:46.973787 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:46.973757    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55402878-108c-4c3d-b6c8-4362f82e82cb" path="/var/lib/kubelet/pods/55402878-108c-4c3d-b6c8-4362f82e82cb/volumes"
Oct 30 13:15:46.974323 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:46.974307    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f09ff8b4-d468-4c5e-9265-1f13766720f2" path="/var/lib/kubelet/pods/f09ff8b4-d468-4c5e-9265-1f13766720f2/volumes"
Oct 30 13:15:47.025955 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:47.025927    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-84942a-openshift-default-58974cd687-wtjlv"
Oct 30 13:15:47.026805 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:47.026785    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-84942a-openshift-default-58974cd687-wtjlv"
Oct 30 13:15:50.109825 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:50.109032    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2daeae-76cf799c89-c8sgx"
Oct 30 13:15:50.419668 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:50.419635    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-84942a-54bbf86548-25252"]
Oct 30 13:15:50.420442 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:50.420428    4364 cpu_manager.go:414] "containerMap: removing container" podUID="55402878-108c-4c3d-b6c8-4362f82e82cb" containerName="payload-processing"
Oct 30 13:15:50.420497 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:50.420444    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55402878-108c-4c3d-b6c8-4362f82e82cb" containerName="payload-processing"
Oct 30 13:15:50.420497 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:50.420487    4364 cpu_manager.go:414] "containerMap: removing container" podUID="f09ff8b4-d468-4c5e-9265-1f13766720f2" containerName="payload-pre-processing"
Oct 30 13:15:50.420497 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:50.420492    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f09ff8b4-d468-4c5e-9265-1f13766720f2" containerName="payload-pre-processing"
Oct 30 13:15:50.420640 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:50.420619    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="55402878-108c-4c3d-b6c8-4362f82e82cb" containerName="payload-processing"
Oct 30 13:15:50.420691 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:50.420641    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="f09ff8b4-d468-4c5e-9265-1f13766720f2" containerName="payload-pre-processing"
Oct 30 13:15:50.424257 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:50.424238    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-84942a-54bbf86548-25252"
Oct 30 13:15:50.461618 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:50.444822    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-84942a-54bbf86548-25252"]
Oct 30 13:15:50.461618 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:50.460886    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-84942a-dockercfg-2ph84\""
Oct 30 13:15:50.461618 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:50.461499    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-84942a\""
Oct 30 13:15:50.517643 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:50.517573    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kgh9\" (UniqueName: \"kubernetes.io/projected/30144d1b-99d3-41fc-9ed9-bc1d18c60a4c-kube-api-access-8kgh9\") pod \"payload-processing-e2e-api-b-84942a-54bbf86548-25252\" (UID: \"30144d1b-99d3-41fc-9ed9-bc1d18c60a4c\") " pod="openshift-ingress/payload-processing-e2e-api-b-84942a-54bbf86548-25252"
Oct 30 13:15:50.517821 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:50.517747    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30144d1b-99d3-41fc-9ed9-bc1d18c60a4c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-84942a-54bbf86548-25252\" (UID: \"30144d1b-99d3-41fc-9ed9-bc1d18c60a4c\") " pod="openshift-ingress/payload-processing-e2e-api-b-84942a-54bbf86548-25252"
Oct 30 13:15:50.637651 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:50.637613    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kgh9\" (UniqueName: \"kubernetes.io/projected/30144d1b-99d3-41fc-9ed9-bc1d18c60a4c-kube-api-access-8kgh9\") pod \"payload-processing-e2e-api-b-84942a-54bbf86548-25252\" (UID: \"30144d1b-99d3-41fc-9ed9-bc1d18c60a4c\") " pod="openshift-ingress/payload-processing-e2e-api-b-84942a-54bbf86548-25252"
Oct 30 13:15:50.637997 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:50.637973    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30144d1b-99d3-41fc-9ed9-bc1d18c60a4c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-84942a-54bbf86548-25252\" (UID: \"30144d1b-99d3-41fc-9ed9-bc1d18c60a4c\") " pod="openshift-ingress/payload-processing-e2e-api-b-84942a-54bbf86548-25252"
Oct 30 13:15:50.639302 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:50.639275    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30144d1b-99d3-41fc-9ed9-bc1d18c60a4c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-84942a-54bbf86548-25252\" (UID: \"30144d1b-99d3-41fc-9ed9-bc1d18c60a4c\") " pod="openshift-ingress/payload-processing-e2e-api-b-84942a-54bbf86548-25252"
Oct 30 13:15:50.744673 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:50.717355    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kgh9\" (UniqueName: \"kubernetes.io/projected/30144d1b-99d3-41fc-9ed9-bc1d18c60a4c-kube-api-access-8kgh9\") pod \"payload-processing-e2e-api-b-84942a-54bbf86548-25252\" (UID: \"30144d1b-99d3-41fc-9ed9-bc1d18c60a4c\") " pod="openshift-ingress/payload-processing-e2e-api-b-84942a-54bbf86548-25252"
Oct 30 13:15:50.765237 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:50.764939    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-84942a-54bbf86548-25252"
Oct 30 13:15:51.270885 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:51.270856    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-84942a-54bbf86548-25252"]
Oct 30 13:15:51.271396 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:15:51.271365    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30144d1b_99d3_41fc_9ed9_bc1d18c60a4c.slice/crio-e04b214c8ae937518926e947ed7b6a19d47b8945f067988026242c3e9b5d6b03 WatchSource:0}: Error finding container e04b214c8ae937518926e947ed7b6a19d47b8945f067988026242c3e9b5d6b03: Status 404 returned error can't find the container with id e04b214c8ae937518926e947ed7b6a19d47b8945f067988026242c3e9b5d6b03
Oct 30 13:15:52.131394 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:52.131349    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-84942a-54bbf86548-25252" event={"ID":"30144d1b-99d3-41fc-9ed9-bc1d18c60a4c","Type":"ContainerStarted","Data":"17018eec7fad9e6a04fd065e76069262f2a990b76b74b30d03b110077c1d21ff"}
Oct 30 13:15:52.131579 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:52.131417    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-84942a-54bbf86548-25252" event={"ID":"30144d1b-99d3-41fc-9ed9-bc1d18c60a4c","Type":"ContainerStarted","Data":"e04b214c8ae937518926e947ed7b6a19d47b8945f067988026242c3e9b5d6b03"}
Oct 30 13:15:52.131579 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:52.131564    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-84942a-54bbf86548-25252"
Oct 30 13:15:52.176361 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:52.176309    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-84942a-54bbf86548-25252" podStartSLOduration=2.176295081 podStartE2EDuration="2.176295081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:52.171766829 +0000 UTC m=+1451.338131478" watchObservedRunningTime="2026-10-30 13:15:52.176295081 +0000 UTC m=+1451.342659746"
Oct 30 13:15:54.984980 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:54.984942    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg"]
Oct 30 13:15:54.985409 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:54.985201    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg" podUID="0dd8443b-7879-47a7-9916-85844e072002" containerName="istio-proxy" containerID="cri-o://72dea0433e055fac820df54527178d9fab4580f403e4a6dfc155961f4c57421d" gracePeriod=30
Oct 30 13:15:54.995587 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:54.995561    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv"]
Oct 30 13:15:54.995827 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:15:54.995794    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv" podUID="4a6cf8c7-ad37-46bc-a392-90de84af768e" containerName="istio-proxy" containerID="cri-o://333bb2cccbf13ddbeef909b155668362790223696316a0995f2e56778f2d9542" gracePeriod=30
Oct 30 13:16:00.279391 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:00.279342    4364 generic.go:356] "Generic (PLEG): container finished" podID="4a6cf8c7-ad37-46bc-a392-90de84af768e" containerID="333bb2cccbf13ddbeef909b155668362790223696316a0995f2e56778f2d9542" exitCode=0
Oct 30 13:16:00.279897 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:00.279643    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv" event={"ID":"4a6cf8c7-ad37-46bc-a392-90de84af768e","Type":"ContainerDied","Data":"333bb2cccbf13ddbeef909b155668362790223696316a0995f2e56778f2d9542"}
Oct 30 13:16:00.281691 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:00.281662    4364 generic.go:356] "Generic (PLEG): container finished" podID="0dd8443b-7879-47a7-9916-85844e072002" containerID="72dea0433e055fac820df54527178d9fab4580f403e4a6dfc155961f4c57421d" exitCode=0
Oct 30 13:16:00.281840 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:00.281795    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg" event={"ID":"0dd8443b-7879-47a7-9916-85844e072002","Type":"ContainerDied","Data":"72dea0433e055fac820df54527178d9fab4580f403e4a6dfc155961f4c57421d"}
Oct 30 13:16:00.317213 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:00.317185    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg"
Oct 30 13:16:00.347882 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:00.347858    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv"
Oct 30 13:16:00.420098 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:00.420065    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0dd8443b-7879-47a7-9916-85844e072002-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dd8443b-7879-47a7-9916-85844e072002-istio-podinfo\") pod \"0dd8443b-7879-47a7-9916-85844e072002\" (UID: \"0dd8443b-7879-47a7-9916-85844e072002\") "
Oct 30 13:16:00.420267 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:00.420213    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dd8443b-7879-47a7-9916-85844e072002-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dd8443b-7879-47a7-9916-85844e072002-credential-socket\") pod \"0dd8443b-7879-47a7-9916-85844e072002\" (UID: \"0dd8443b-7879-47a7-9916-85844e072002\") "
Oct 30 13:16:00.420267 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:00.420244    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dd8443b-7879-47a7-9916-85844e072002-istio-token\" (UniqueName: \"kubernetes.io/projected/0dd8443b-7879-47a7-9916-85844e072002-istio-token\") pod \"0dd8443b-7879-47a7-9916-85844e072002\" (UID: \"0dd8443b-7879-47a7-9916-85844e072002\") "
Oct 30 13:16:00.420387 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:00.420322    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dd8443b-7879-47a7-9916-85844e072002-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dd8443b-7879-47a7-9916-85844e072002-workload-socket\") pod \"0dd8443b-7879-47a7-9916-85844e072002\" (UID: \"0dd8443b-7879-47a7-9916-85844e072002\") "
Oct 30 13:16:00.424112 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:00.420463    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dd8443b-7879-47a7-9916-85844e072002-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dd8443b-7879-47a7-9916-85844e072002-workload-certs\") pod \"0dd8443b-7879-47a7-9916-85844e072002\" (UID: \"0dd8443b-7879-47a7-9916-85844e072002\") "
Oct 30 13:16:00.424112 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:00.420538    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dd8443b-7879-47a7-9916-85844e072002-credential-socket" pod "0dd8443b-7879-47a7-9916-85844e072002" (UID: "0dd8443b-7879-47a7-9916-85844e072002"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:16:00.424112 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:00.420592    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dd8443b-7879-47a7-9916-85844e072002-kube-api-access-g6h5r\" (UniqueName: \"kubernetes.io/projected/0dd8443b-7879-47a7-9916-85844e072002-kube-api-access-g6h5r\") pod \"0dd8443b-7879-47a7-9916-85844e072002\" (UID: \"0dd8443b-7879-47a7-9916-85844e072002\") "
Oct 30 13:16:00.424112 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:00.420664    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dd8443b-7879-47a7-9916-85844e072002-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dd8443b-7879-47a7-9916-85844e072002-istio-envoy\") pod \"0dd8443b-7879-47a7-9916-85844e072002\" (UID: \"0dd8443b-7879-47a7-9916-85844e072002\") "
Oct 30 13:16:00.424112 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:00.420758    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dd8443b-7879-47a7-9916-85844e072002-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dd8443b-7879-47a7-9916-85844e072002-istio-data\") pod \"0dd8443b-7879-47a7-9916-85844e072002\" (UID: \"0dd8443b-7879-47a7-9916-85844e072002\") "
Oct 30 13:16:00.424112 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:00.420790    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dd8443b-7879-47a7-9916-85844e072002-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dd8443b-7879-47a7-9916-85844e072002-istiod-ca-cert\") pod \"0dd8443b-7879-47a7-9916-85844e072002\" (UID: \"0dd8443b-7879-47a7-9916-85844e072002\") "
Oct 30 13:16:00.424112 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:00.423501    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0dd8443b-7879-47a7-9916-85844e072002-istio-podinfo" pod "0dd8443b-7879-47a7-9916-85844e072002" (UID: "0dd8443b-7879-47a7-9916-85844e072002"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:16:00.424112 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:00.423878    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dd8443b-7879-47a7-9916-85844e072002-istiod-ca-cert" pod "0dd8443b-7879-47a7-9916-85844e072002" (UID: "0dd8443b-7879-47a7-9916-85844e072002"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:16:00.424112 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:00.424037    4364 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dd8443b-7879-47a7-9916-85844e072002-istio-podinfo\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:16:00.424112 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:00.424059    4364 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dd8443b-7879-47a7-9916-85844e072002-credential-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:16:00.424815 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:00.424186    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dd8443b-7879-47a7-9916-85844e072002-istio-data" pod "0dd8443b-7879-47a7-9916-85844e072002" (UID: "0dd8443b-7879-47a7-9916-85844e072002"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:16:00.424815 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:00.424232    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dd8443b-7879-47a7-9916-85844e072002-workload-certs" pod "0dd8443b-7879-47a7-9916-85844e072002" (UID: "0dd8443b-7879-47a7-9916-85844e072002"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:16:00.424815 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:00.424290    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dd8443b-7879-47a7-9916-85844e072002-workload-socket" pod "0dd8443b-7879-47a7-9916-85844e072002" (UID: "0dd8443b-7879-47a7-9916-85844e072002"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:16:00.424815 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:00.424692    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dd8443b-7879-47a7-9916-85844e072002-kube-api-access-g6h5r" pod "0dd8443b-7879-47a7-9916-85844e072002" (UID: "0dd8443b-7879-47a7-9916-85844e072002"). InnerVolumeSpecName "kube-api-access-g6h5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:16:00.426034 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:00.426004    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dd8443b-7879-47a7-9916-85844e072002-istio-envoy" pod "0dd8443b-7879-47a7-9916-85844e072002" (UID: "0dd8443b-7879-47a7-9916-85844e072002"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:16:00.426146 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:00.426096    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dd8443b-7879-47a7-9916-85844e072002-istio-token" pod "0dd8443b-7879-47a7-9916-85844e072002" (UID: "0dd8443b-7879-47a7-9916-85844e072002"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:16:00.525527 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:00.525453    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4a6cf8c7-ad37-46bc-a392-90de84af768e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a6cf8c7-ad37-46bc-a392-90de84af768e-istio-podinfo\") pod \"4a6cf8c7-ad37-46bc-a392-90de84af768e\" (UID: \"4a6cf8c7-ad37-46bc-a392-90de84af768e\") "
Oct 30 13:16:00.525691 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:00.525538    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a6cf8c7-ad37-46bc-a392-90de84af768e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a6cf8c7-ad37-46bc-a392-90de84af768e-workload-certs\") pod \"4a6cf8c7-ad37-46bc-a392-90de84af768e\" (UID: \"4a6cf8c7-ad37-46bc-a392-90de84af768e\") "
Oct 30 13:16:00.525691 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:00.525565    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a6cf8c7-ad37-46bc-a392-90de84af768e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a6cf8c7-ad37-46bc-a392-90de84af768e-credential-socket\") pod \"4a6cf8c7-ad37-46bc-a392-90de84af768e\" (UID: \"4a6cf8c7-ad37-46bc-a392-90de84af768e\") "
Oct 30 13:16:00.525691 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:00.525679    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a6cf8c7-ad37-46bc-a392-90de84af768e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a6cf8c7-ad37-46bc-a392-90de84af768e-istio-envoy\") pod \"4a6cf8c7-ad37-46bc-a392-90de84af768e\" (UID: \"4a6cf8c7-ad37-46bc-a392-90de84af768e\") "
Oct 30 13:16:00.525852 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:00.525814    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a6cf8c7-ad37-46bc-a392-90de84af768e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a6cf8c7-ad37-46bc-a392-90de84af768e-istiod-ca-cert\") pod \"4a6cf8c7-ad37-46bc-a392-90de84af768e\" (UID: \"4a6cf8c7-ad37-46bc-a392-90de84af768e\") "
Oct 30 13:16:00.525913 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:00.525887    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a6cf8c7-ad37-46bc-a392-90de84af768e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a6cf8c7-ad37-46bc-a392-90de84af768e-workload-socket\") pod \"4a6cf8c7-ad37-46bc-a392-90de84af768e\" (UID: \"4a6cf8c7-ad37-46bc-a392-90de84af768e\") "
Oct 30 13:16:00.525980 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:00.525905    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a6cf8c7-ad37-46bc-a392-90de84af768e-workload-certs" pod "4a6cf8c7-ad37-46bc-a392-90de84af768e" (UID: "4a6cf8c7-ad37-46bc-a392-90de84af768e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:16:00.525980 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:00.525928    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a6cf8c7-ad37-46bc-a392-90de84af768e-credential-socket" pod "4a6cf8c7-ad37-46bc-a392-90de84af768e" (UID: "4a6cf8c7-ad37-46bc-a392-90de84af768e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:16:00.525980 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:00.525935    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a6cf8c7-ad37-46bc-a392-90de84af768e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a6cf8c7-ad37-46bc-a392-90de84af768e-istio-data\") pod \"4a6cf8c7-ad37-46bc-a392-90de84af768e\" (UID: \"4a6cf8c7-ad37-46bc-a392-90de84af768e\") "
Oct 30 13:16:00.526140 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:00.526118    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a6cf8c7-ad37-46bc-a392-90de84af768e-istio-data" pod "4a6cf8c7-ad37-46bc-a392-90de84af768e" (UID: "4a6cf8c7-ad37-46bc-a392-90de84af768e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:16:00.526140 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:00.526130    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a6cf8c7-ad37-46bc-a392-90de84af768e-workload-socket" pod "4a6cf8c7-ad37-46bc-a392-90de84af768e" (UID: "4a6cf8c7-ad37-46bc-a392-90de84af768e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:16:00.526247 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:00.526147    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a6cf8c7-ad37-46bc-a392-90de84af768e-istio-token\" (UniqueName: \"kubernetes.io/projected/4a6cf8c7-ad37-46bc-a392-90de84af768e-istio-token\") pod \"4a6cf8c7-ad37-46bc-a392-90de84af768e\" (UID: \"4a6cf8c7-ad37-46bc-a392-90de84af768e\") "
Oct 30 13:16:00.526247 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:00.526182    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a6cf8c7-ad37-46bc-a392-90de84af768e-kube-api-access-tmg87\" (UniqueName: \"kubernetes.io/projected/4a6cf8c7-ad37-46bc-a392-90de84af768e-kube-api-access-tmg87\") pod \"4a6cf8c7-ad37-46bc-a392-90de84af768e\" (UID: \"4a6cf8c7-ad37-46bc-a392-90de84af768e\") "
Oct 30 13:16:00.526357 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:00.526230    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a6cf8c7-ad37-46bc-a392-90de84af768e-istiod-ca-cert" pod "4a6cf8c7-ad37-46bc-a392-90de84af768e" (UID: "4a6cf8c7-ad37-46bc-a392-90de84af768e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:16:00.526911 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:00.526887    4364 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dd8443b-7879-47a7-9916-85844e072002-workload-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:16:00.526911 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:00.526909    4364 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a6cf8c7-ad37-46bc-a392-90de84af768e-workload-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:16:00.527080 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:00.526918    4364 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a6cf8c7-ad37-46bc-a392-90de84af768e-credential-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:16:00.527080 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:00.526927    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6h5r\" (UniqueName: \"kubernetes.io/projected/0dd8443b-7879-47a7-9916-85844e072002-kube-api-access-g6h5r\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:16:00.527080 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:00.526937    4364 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dd8443b-7879-47a7-9916-85844e072002-istio-envoy\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:16:00.527080 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:00.526946    4364 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dd8443b-7879-47a7-9916-85844e072002-istio-data\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:16:00.527080 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:00.526953    4364 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dd8443b-7879-47a7-9916-85844e072002-istiod-ca-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:16:00.527080 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:00.526961    4364 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a6cf8c7-ad37-46bc-a392-90de84af768e-istiod-ca-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:16:00.527080 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:00.526972    4364 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a6cf8c7-ad37-46bc-a392-90de84af768e-workload-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:16:00.527080 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:00.526982    4364 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a6cf8c7-ad37-46bc-a392-90de84af768e-istio-data\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:16:00.527080 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:00.526992    4364 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dd8443b-7879-47a7-9916-85844e072002-istio-token\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:16:00.527080 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:00.526999    4364 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dd8443b-7879-47a7-9916-85844e072002-workload-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:16:00.528156 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:00.528132    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a6cf8c7-ad37-46bc-a392-90de84af768e-istio-token" pod "4a6cf8c7-ad37-46bc-a392-90de84af768e" (UID: "4a6cf8c7-ad37-46bc-a392-90de84af768e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:16:00.528265 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:00.528167    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4a6cf8c7-ad37-46bc-a392-90de84af768e-istio-podinfo" pod "4a6cf8c7-ad37-46bc-a392-90de84af768e" (UID: "4a6cf8c7-ad37-46bc-a392-90de84af768e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:16:00.528312 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:00.528268    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a6cf8c7-ad37-46bc-a392-90de84af768e-istio-envoy" pod "4a6cf8c7-ad37-46bc-a392-90de84af768e" (UID: "4a6cf8c7-ad37-46bc-a392-90de84af768e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:16:00.528449 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:00.528430    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a6cf8c7-ad37-46bc-a392-90de84af768e-kube-api-access-tmg87" pod "4a6cf8c7-ad37-46bc-a392-90de84af768e" (UID: "4a6cf8c7-ad37-46bc-a392-90de84af768e"). InnerVolumeSpecName "kube-api-access-tmg87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:16:00.628423 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:00.628392    4364 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a6cf8c7-ad37-46bc-a392-90de84af768e-istio-token\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:16:00.628423 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:00.628420    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmg87\" (UniqueName: \"kubernetes.io/projected/4a6cf8c7-ad37-46bc-a392-90de84af768e-kube-api-access-tmg87\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:16:00.628640 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:00.628437    4364 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a6cf8c7-ad37-46bc-a392-90de84af768e-istio-podinfo\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:16:00.628640 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:00.628448    4364 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a6cf8c7-ad37-46bc-a392-90de84af768e-istio-envoy\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:16:01.224303 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:01.224270    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj"]
Oct 30 13:16:01.225450 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:01.225244    4364 cpu_manager.go:414] "containerMap: removing container" podUID="0dd8443b-7879-47a7-9916-85844e072002" containerName="istio-proxy"
Oct 30 13:16:01.225450 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:01.225267    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dd8443b-7879-47a7-9916-85844e072002" containerName="istio-proxy"
Oct 30 13:16:01.225450 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:01.225331    4364 cpu_manager.go:414] "containerMap: removing container" podUID="4a6cf8c7-ad37-46bc-a392-90de84af768e" containerName="istio-proxy"
Oct 30 13:16:01.225450 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:01.225341    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a6cf8c7-ad37-46bc-a392-90de84af768e" containerName="istio-proxy"
Oct 30 13:16:01.225877 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:01.225562    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dd8443b-7879-47a7-9916-85844e072002" containerName="istio-proxy"
Oct 30 13:16:01.225877 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:01.225614    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a6cf8c7-ad37-46bc-a392-90de84af768e" containerName="istio-proxy"
Oct 30 13:16:01.229701 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:01.229679    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj"
Oct 30 13:16:01.257690 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:01.257662    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj"]
Oct 30 13:16:01.259681 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:01.259656    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-4644a1-openshift-default-dockercfg-ds2kw\""
Oct 30 13:16:01.296816 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:01.296791    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg"
Oct 30 13:16:01.297305 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:01.296813    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg" event={"ID":"0dd8443b-7879-47a7-9916-85844e072002","Type":"ContainerDied","Data":"ac24cd773670edfb8408ba7d0a5470bef6a340262a1489810a3c70518d546998"}
Oct 30 13:16:01.297305 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:01.296875    4364 scope.go:122] "RemoveContainer" containerID="72dea0433e055fac820df54527178d9fab4580f403e4a6dfc155961f4c57421d"
Oct 30 13:16:01.298987 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:01.298966    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv"
Oct 30 13:16:01.299081 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:01.298981    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv" event={"ID":"4a6cf8c7-ad37-46bc-a392-90de84af768e","Type":"ContainerDied","Data":"fb5b7ebba6e8caa16fe61ecf2939128a1ef7fb73661838e6c5ce947a2f61c093"}
Oct 30 13:16:01.309116 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:01.309093    4364 scope.go:122] "RemoveContainer" containerID="333bb2cccbf13ddbeef909b155668362790223696316a0995f2e56778f2d9542"
Oct 30 13:16:01.361617 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:01.355317    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc3dc010-a693-406c-ba1e-55593789e70a-workload-certs\") pod \"e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj\" (UID: \"dc3dc010-a693-406c-ba1e-55593789e70a\") " pod="openshift-ingress/e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj"
Oct 30 13:16:01.361617 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:01.355430    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc3dc010-a693-406c-ba1e-55593789e70a-istio-podinfo\") pod \"e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj\" (UID: \"dc3dc010-a693-406c-ba1e-55593789e70a\") " pod="openshift-ingress/e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj"
Oct 30 13:16:01.361617 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:01.355545    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc3dc010-a693-406c-ba1e-55593789e70a-istio-envoy\") pod \"e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj\" (UID: \"dc3dc010-a693-406c-ba1e-55593789e70a\") " pod="openshift-ingress/e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj"
Oct 30 13:16:01.361617 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:01.355630    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc3dc010-a693-406c-ba1e-55593789e70a-istiod-ca-cert\") pod \"e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj\" (UID: \"dc3dc010-a693-406c-ba1e-55593789e70a\") " pod="openshift-ingress/e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj"
Oct 30 13:16:01.361617 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:01.355714    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc3dc010-a693-406c-ba1e-55593789e70a-istio-token\") pod \"e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj\" (UID: \"dc3dc010-a693-406c-ba1e-55593789e70a\") " pod="openshift-ingress/e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj"
Oct 30 13:16:01.361617 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:01.355812    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc3dc010-a693-406c-ba1e-55593789e70a-credential-socket\") pod \"e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj\" (UID: \"dc3dc010-a693-406c-ba1e-55593789e70a\") " pod="openshift-ingress/e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj"
Oct 30 13:16:01.361617 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:01.355941    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62bl6\" (UniqueName: \"kubernetes.io/projected/dc3dc010-a693-406c-ba1e-55593789e70a-kube-api-access-62bl6\") pod \"e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj\" (UID: \"dc3dc010-a693-406c-ba1e-55593789e70a\") " pod="openshift-ingress/e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj"
Oct 30 13:16:01.361617 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:01.355976    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc3dc010-a693-406c-ba1e-55593789e70a-istio-data\") pod \"e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj\" (UID: \"dc3dc010-a693-406c-ba1e-55593789e70a\") " pod="openshift-ingress/e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj"
Oct 30 13:16:01.361617 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:01.356017    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc3dc010-a693-406c-ba1e-55593789e70a-workload-socket\") pod \"e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj\" (UID: \"dc3dc010-a693-406c-ba1e-55593789e70a\") " pod="openshift-ingress/e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj"
Oct 30 13:16:01.523702 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:01.499259    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62bl6\" (UniqueName: \"kubernetes.io/projected/dc3dc010-a693-406c-ba1e-55593789e70a-kube-api-access-62bl6\") pod \"e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj\" (UID: \"dc3dc010-a693-406c-ba1e-55593789e70a\") " pod="openshift-ingress/e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj"
Oct 30 13:16:01.523702 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:01.499317    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc3dc010-a693-406c-ba1e-55593789e70a-istio-data\") pod \"e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj\" (UID: \"dc3dc010-a693-406c-ba1e-55593789e70a\") " pod="openshift-ingress/e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj"
Oct 30 13:16:01.523702 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:01.499360    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc3dc010-a693-406c-ba1e-55593789e70a-workload-socket\") pod \"e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj\" (UID: \"dc3dc010-a693-406c-ba1e-55593789e70a\") " pod="openshift-ingress/e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj"
Oct 30 13:16:01.523702 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:01.499559    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc3dc010-a693-406c-ba1e-55593789e70a-workload-certs\") pod \"e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj\" (UID: \"dc3dc010-a693-406c-ba1e-55593789e70a\") " pod="openshift-ingress/e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj"
Oct 30 13:16:01.523702 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:01.499615    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc3dc010-a693-406c-ba1e-55593789e70a-istio-podinfo\") pod \"e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj\" (UID: \"dc3dc010-a693-406c-ba1e-55593789e70a\") " pod="openshift-ingress/e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj"
Oct 30 13:16:01.523702 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:01.499728    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc3dc010-a693-406c-ba1e-55593789e70a-istio-envoy\") pod \"e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj\" (UID: \"dc3dc010-a693-406c-ba1e-55593789e70a\") " pod="openshift-ingress/e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj"
Oct 30 13:16:01.523702 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:01.499794    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc3dc010-a693-406c-ba1e-55593789e70a-istiod-ca-cert\") pod \"e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj\" (UID: \"dc3dc010-a693-406c-ba1e-55593789e70a\") " pod="openshift-ingress/e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj"
Oct 30 13:16:01.523702 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:01.499835    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc3dc010-a693-406c-ba1e-55593789e70a-istio-token\") pod \"e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj\" (UID: \"dc3dc010-a693-406c-ba1e-55593789e70a\") " pod="openshift-ingress/e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj"
Oct 30 13:16:01.523702 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:01.499902    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc3dc010-a693-406c-ba1e-55593789e70a-credential-socket\") pod \"e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj\" (UID: \"dc3dc010-a693-406c-ba1e-55593789e70a\") " pod="openshift-ingress/e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj"
Oct 30 13:16:01.523702 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:01.500463    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc3dc010-a693-406c-ba1e-55593789e70a-credential-socket\") pod \"e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj\" (UID: \"dc3dc010-a693-406c-ba1e-55593789e70a\") " pod="openshift-ingress/e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj"
Oct 30 13:16:01.523702 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:01.501042    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc3dc010-a693-406c-ba1e-55593789e70a-workload-certs\") pod \"e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj\" (UID: \"dc3dc010-a693-406c-ba1e-55593789e70a\") " pod="openshift-ingress/e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj"
Oct 30 13:16:01.523702 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:01.501534    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc3dc010-a693-406c-ba1e-55593789e70a-istio-data\") pod \"e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj\" (UID: \"dc3dc010-a693-406c-ba1e-55593789e70a\") " pod="openshift-ingress/e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj"
Oct 30 13:16:01.523702 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:01.501812    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc3dc010-a693-406c-ba1e-55593789e70a-workload-socket\") pod \"e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj\" (UID: \"dc3dc010-a693-406c-ba1e-55593789e70a\") " pod="openshift-ingress/e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj"
Oct 30 13:16:01.523702 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:01.502441    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc3dc010-a693-406c-ba1e-55593789e70a-istiod-ca-cert\") pod \"e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj\" (UID: \"dc3dc010-a693-406c-ba1e-55593789e70a\") " pod="openshift-ingress/e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj"
Oct 30 13:16:01.550369 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:01.541667    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv"]
Oct 30 13:16:01.583621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:01.562750    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc3dc010-a693-406c-ba1e-55593789e70a-istio-podinfo\") pod \"e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj\" (UID: \"dc3dc010-a693-406c-ba1e-55593789e70a\") " pod="openshift-ingress/e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj"
Oct 30 13:16:01.810096 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:01.810013    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gh9kv"]
Oct 30 13:16:01.832617 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:01.826898    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg"]
Oct 30 13:16:01.851646 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:01.851590    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc3dc010-a693-406c-ba1e-55593789e70a-istio-envoy\") pod \"e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj\" (UID: \"dc3dc010-a693-406c-ba1e-55593789e70a\") " pod="openshift-ingress/e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj"
Oct 30 13:16:01.852380 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:01.852351    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62bl6\" (UniqueName: \"kubernetes.io/projected/dc3dc010-a693-406c-ba1e-55593789e70a-kube-api-access-62bl6\") pod \"e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj\" (UID: \"dc3dc010-a693-406c-ba1e-55593789e70a\") " pod="openshift-ingress/e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj"
Oct 30 13:16:01.916634 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:01.916582    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-gkmdg"]
Oct 30 13:16:01.932548 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:01.930739    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc3dc010-a693-406c-ba1e-55593789e70a-istio-token\") pod \"e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj\" (UID: \"dc3dc010-a693-406c-ba1e-55593789e70a\") " pod="openshift-ingress/e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj"
Oct 30 13:16:02.523649 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:02.519765    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz" podUID="bc5caf69-a649-47e5-8910-9b8f331dfd4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:16:02.677682 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:02.674211    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj"
Oct 30 13:16:03.015168 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:03.015118    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dd8443b-7879-47a7-9916-85844e072002" path="/var/lib/kubelet/pods/0dd8443b-7879-47a7-9916-85844e072002/volumes"
Oct 30 13:16:03.016145 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:03.016120    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a6cf8c7-ad37-46bc-a392-90de84af768e" path="/var/lib/kubelet/pods/4a6cf8c7-ad37-46bc-a392-90de84af768e/volumes"
Oct 30 13:16:03.144241 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:03.144210    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-84942a-54bbf86548-25252"
Oct 30 13:16:03.325646 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:03.325569    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df6c3661-5tfnb"]
Oct 30 13:16:03.334405 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:03.334350    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df6c3661-5tfnb"]
Oct 30 13:16:03.383895 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:03.383860    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj"]
Oct 30 13:16:03.513166 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:03.508118    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:16:03.513166 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:03.508231    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:16:03.513166 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:03.508281    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:16:04.001220 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:04.001180    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj" event={"ID":"dc3dc010-a693-406c-ba1e-55593789e70a","Type":"ContainerStarted","Data":"f905015273e255c0722a45299627b4b2fb5bcd0811c8d44578957967701da44c"}
Oct 30 13:16:04.001677 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:04.001238    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj" event={"ID":"dc3dc010-a693-406c-ba1e-55593789e70a","Type":"ContainerStarted","Data":"3da49413b3f36822e9987a7d65e7e88c824c69a69d7d595e7e9ad5e9c288c1ad"}
Oct 30 13:16:04.050984 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:04.050931    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj" podStartSLOduration=3.050916475 podStartE2EDuration="3.050916475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:16:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:16:04.04693616 +0000 UTC m=+1463.213300807" watchObservedRunningTime="2026-10-30 13:16:04.050916475 +0000 UTC m=+1463.217281123"
Oct 30 13:16:04.200855 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:04.200825    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-cda0d9-74764bd4cf-l64mp"]
Oct 30 13:16:04.205734 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:04.205710    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-cda0d9-74764bd4cf-l64mp"
Oct 30 13:16:04.221782 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:04.221744    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-cda0d9\""
Oct 30 13:16:04.222175 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:04.222157    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-cda0d9-dockercfg-t4lmz\""
Oct 30 13:16:04.225021 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:04.225000    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-cda0d9-74764bd4cf-l64mp"]
Oct 30 13:16:04.598174 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:04.598116    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd1b7e78-a663-4d45-a742-5994a0259a24-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-cda0d9-74764bd4cf-l64mp\" (UID: \"bd1b7e78-a663-4d45-a742-5994a0259a24\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-cda0d9-74764bd4cf-l64mp"
Oct 30 13:16:04.600767 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:04.600738    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjxrf\" (UniqueName: \"kubernetes.io/projected/bd1b7e78-a663-4d45-a742-5994a0259a24-kube-api-access-kjxrf\") pod \"payload-pre-processing-e2e-shared-b-w6-cda0d9-74764bd4cf-l64mp\" (UID: \"bd1b7e78-a663-4d45-a742-5994a0259a24\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-cda0d9-74764bd4cf-l64mp"
Oct 30 13:16:04.678135 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:04.678102    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj"
Oct 30 13:16:04.685992 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:04.685942    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj"
Oct 30 13:16:04.711148 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:04.711104    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjxrf\" (UniqueName: \"kubernetes.io/projected/bd1b7e78-a663-4d45-a742-5994a0259a24-kube-api-access-kjxrf\") pod \"payload-pre-processing-e2e-shared-b-w6-cda0d9-74764bd4cf-l64mp\" (UID: \"bd1b7e78-a663-4d45-a742-5994a0259a24\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-cda0d9-74764bd4cf-l64mp"
Oct 30 13:16:04.711347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:04.711323    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd1b7e78-a663-4d45-a742-5994a0259a24-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-cda0d9-74764bd4cf-l64mp\" (UID: \"bd1b7e78-a663-4d45-a742-5994a0259a24\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-cda0d9-74764bd4cf-l64mp"
Oct 30 13:16:04.712723 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:04.712693    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd1b7e78-a663-4d45-a742-5994a0259a24-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-cda0d9-74764bd4cf-l64mp\" (UID: \"bd1b7e78-a663-4d45-a742-5994a0259a24\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-cda0d9-74764bd4cf-l64mp"
Oct 30 13:16:04.745850 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:04.745818    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjxrf\" (UniqueName: \"kubernetes.io/projected/bd1b7e78-a663-4d45-a742-5994a0259a24-kube-api-access-kjxrf\") pod \"payload-pre-processing-e2e-shared-b-w6-cda0d9-74764bd4cf-l64mp\" (UID: \"bd1b7e78-a663-4d45-a742-5994a0259a24\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-cda0d9-74764bd4cf-l64mp"
Oct 30 13:16:04.824424 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:04.824393    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-cda0d9-74764bd4cf-l64mp"
Oct 30 13:16:04.975904 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:04.975871    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e82ae840-d302-42ed-aa7c-daea8b5e1c23" path="/var/lib/kubelet/pods/e82ae840-d302-42ed-aa7c-daea8b5e1c23/volumes"
Oct 30 13:16:05.009554 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:05.009509    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj"
Oct 30 13:16:05.010543 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:05.010519    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj"
Oct 30 13:16:05.136206 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:16:05.136173    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd1b7e78_a663_4d45_a742_5994a0259a24.slice/crio-3b761fc92258613301f1ff16dbe170f92a72a7a5da819268e44c87bccc7a7b8d WatchSource:0}: Error finding container 3b761fc92258613301f1ff16dbe170f92a72a7a5da819268e44c87bccc7a7b8d: Status 404 returned error can't find the container with id 3b761fc92258613301f1ff16dbe170f92a72a7a5da819268e44c87bccc7a7b8d
Oct 30 13:16:05.136934 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:05.136909    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-cda0d9-74764bd4cf-l64mp"]
Oct 30 13:16:06.019776 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:06.019741    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-cda0d9-74764bd4cf-l64mp" event={"ID":"bd1b7e78-a663-4d45-a742-5994a0259a24","Type":"ContainerStarted","Data":"e7b248d8c1f66460e433ea637fdb1a9e085e5d7c8ed4532d5cf5c6b9603a9ce7"}
Oct 30 13:16:06.020220 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:06.019796    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-cda0d9-74764bd4cf-l64mp" event={"ID":"bd1b7e78-a663-4d45-a742-5994a0259a24","Type":"ContainerStarted","Data":"3b761fc92258613301f1ff16dbe170f92a72a7a5da819268e44c87bccc7a7b8d"}
Oct 30 13:16:06.020220 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:06.019983    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-cda0d9-74764bd4cf-l64mp"
Oct 30 13:16:06.104237 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:06.104126    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-cda0d9-74764bd4cf-l64mp" podStartSLOduration=2.104102583 podStartE2EDuration="2.104102583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:16:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:16:06.071243939 +0000 UTC m=+1465.237608580" watchObservedRunningTime="2026-10-30 13:16:06.104102583 +0000 UTC m=+1465.270467233"
Oct 30 13:16:08.656269 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:08.656228    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4644a1-74c746f55f-rkfmk"]
Oct 30 13:16:08.661919 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:08.661895    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4644a1-74c746f55f-rkfmk"
Oct 30 13:16:08.684126 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:08.684072    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4644a1-74c746f55f-rkfmk"]
Oct 30 13:16:08.688995 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:08.688961    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-4644a1-dockercfg-z4cq9\""
Oct 30 13:16:08.732702 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:08.732657    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-4644a1\""
Oct 30 13:16:08.924913 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:08.900452    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn92k\" (UniqueName: \"kubernetes.io/projected/ad81d931-2563-49a1-aebe-a7f0f29cda94-kube-api-access-kn92k\") pod \"payload-pre-processing-e2e-api-b-4644a1-74c746f55f-rkfmk\" (UID: \"ad81d931-2563-49a1-aebe-a7f0f29cda94\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4644a1-74c746f55f-rkfmk"
Oct 30 13:16:08.924913 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:08.900693    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad81d931-2563-49a1-aebe-a7f0f29cda94-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4644a1-74c746f55f-rkfmk\" (UID: \"ad81d931-2563-49a1-aebe-a7f0f29cda94\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4644a1-74c746f55f-rkfmk"
Oct 30 13:16:09.059452 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:09.023730    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn92k\" (UniqueName: \"kubernetes.io/projected/ad81d931-2563-49a1-aebe-a7f0f29cda94-kube-api-access-kn92k\") pod \"payload-pre-processing-e2e-api-b-4644a1-74c746f55f-rkfmk\" (UID: \"ad81d931-2563-49a1-aebe-a7f0f29cda94\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4644a1-74c746f55f-rkfmk"
Oct 30 13:16:09.059452 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:09.023944    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad81d931-2563-49a1-aebe-a7f0f29cda94-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4644a1-74c746f55f-rkfmk\" (UID: \"ad81d931-2563-49a1-aebe-a7f0f29cda94\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4644a1-74c746f55f-rkfmk"
Oct 30 13:16:09.059452 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:09.032060    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad81d931-2563-49a1-aebe-a7f0f29cda94-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4644a1-74c746f55f-rkfmk\" (UID: \"ad81d931-2563-49a1-aebe-a7f0f29cda94\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4644a1-74c746f55f-rkfmk"
Oct 30 13:16:09.145658 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:09.131592    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn92k\" (UniqueName: \"kubernetes.io/projected/ad81d931-2563-49a1-aebe-a7f0f29cda94-kube-api-access-kn92k\") pod \"payload-pre-processing-e2e-api-b-4644a1-74c746f55f-rkfmk\" (UID: \"ad81d931-2563-49a1-aebe-a7f0f29cda94\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4644a1-74c746f55f-rkfmk"
Oct 30 13:16:09.278527 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:09.278442    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4644a1-74c746f55f-rkfmk"
Oct 30 13:16:09.593227 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:09.593201    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4644a1-74c746f55f-rkfmk"]
Oct 30 13:16:09.595649 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:16:09.595617    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad81d931_2563_49a1_aebe_a7f0f29cda94.slice/crio-86f3a087716d3e234010f495a2feba469e5af0841ae58c536ddd699eed7dcc1b WatchSource:0}: Error finding container 86f3a087716d3e234010f495a2feba469e5af0841ae58c536ddd699eed7dcc1b: Status 404 returned error can't find the container with id 86f3a087716d3e234010f495a2feba469e5af0841ae58c536ddd699eed7dcc1b
Oct 30 13:16:09.959589 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:09.959545    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr"]
Oct 30 13:16:09.960012 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:09.959790    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr" podUID="747bdafd-1a41-4b0e-9547-f14fe568d3f1" containerName="istio-proxy" containerID="cri-o://c6b047c72d2a3839a86bbecb3a07abaaed62689d2473a04811aa1cc928573f11" gracePeriod=30
Oct 30 13:16:09.971970 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:09.971940    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh"]
Oct 30 13:16:09.972154 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:09.972130    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh" podUID="42d369e1-ee25-47fe-a808-cb0470669962" containerName="istio-proxy" containerID="cri-o://835dfbab0f094930fb9b622678c7abd6d9ac1a0d33b57275fd0ec8599ad319d5" gracePeriod=30
Oct 30 13:16:10.154035 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:10.153995    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4644a1-74c746f55f-rkfmk" event={"ID":"ad81d931-2563-49a1-aebe-a7f0f29cda94","Type":"ContainerStarted","Data":"e9b0fba95ed493f6476f97398d85c7c3f4b5375db66321e1857e9f4c0f75626e"}
Oct 30 13:16:10.154217 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:10.154096    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4644a1-74c746f55f-rkfmk" event={"ID":"ad81d931-2563-49a1-aebe-a7f0f29cda94","Type":"ContainerStarted","Data":"86f3a087716d3e234010f495a2feba469e5af0841ae58c536ddd699eed7dcc1b"}
Oct 30 13:16:10.154217 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:10.154128    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4644a1-74c746f55f-rkfmk"
Oct 30 13:16:10.293469 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:10.293365    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4644a1-74c746f55f-rkfmk" podStartSLOduration=2.293349591 podStartE2EDuration="2.293349591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:16:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:16:10.285491541 +0000 UTC m=+1469.451856191" watchObservedRunningTime="2026-10-30 13:16:10.293349591 +0000 UTC m=+1469.459714238"
Oct 30 13:16:11.506225 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:16:11.506187    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:16:12.209298 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:12.209262    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-df6c3661-7f764d9794-cq64m"]
Oct 30 13:16:12.214225 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:12.214205    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-df6c3661-7f764d9794-cq64m"
Oct 30 13:16:12.230897 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:12.230874    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-df6c3661\""
Oct 30 13:16:12.251417 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:12.251378    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-df6c3661-7f764d9794-cq64m"]
Oct 30 13:16:12.276454 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:12.276431    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-df6c3661-5866c5b5c5-qfrtz"]
Oct 30 13:16:12.281826 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:12.281804    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-df6c3661-5866c5b5c5-qfrtz"
Oct 30 13:16:12.293758 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:12.293734    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5298fc7-1369-4e44-8ee3-f058d1e20c37-plugins-config-volume\") pod \"payload-processing-e2e-mt-df6c3661-7f764d9794-cq64m\" (UID: \"d5298fc7-1369-4e44-8ee3-f058d1e20c37\") " pod="openshift-ingress/payload-processing-e2e-mt-df6c3661-7f764d9794-cq64m"
Oct 30 13:16:12.293888 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:12.293782    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9jgk\" (UniqueName: \"kubernetes.io/projected/d5298fc7-1369-4e44-8ee3-f058d1e20c37-kube-api-access-z9jgk\") pod \"payload-processing-e2e-mt-df6c3661-7f764d9794-cq64m\" (UID: \"d5298fc7-1369-4e44-8ee3-f058d1e20c37\") " pod="openshift-ingress/payload-processing-e2e-mt-df6c3661-7f764d9794-cq64m"
Oct 30 13:16:12.303885 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:12.303860    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-df6c3661-5866c5b5c5-qfrtz"]
Oct 30 13:16:12.432625 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:12.411529    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-df6c3661-dockercfg-5sgs8\""
Oct 30 13:16:12.461731 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:12.442632    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w5ld\" (UniqueName: \"kubernetes.io/projected/42c43460-b498-432d-a8e5-a68d5046aba6-kube-api-access-5w5ld\") pod \"payload-pre-processing-e2e-mt-df6c3661-5866c5b5c5-qfrtz\" (UID: \"42c43460-b498-432d-a8e5-a68d5046aba6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-df6c3661-5866c5b5c5-qfrtz"
Oct 30 13:16:12.461731 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:12.442807    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5298fc7-1369-4e44-8ee3-f058d1e20c37-plugins-config-volume\") pod \"payload-processing-e2e-mt-df6c3661-7f764d9794-cq64m\" (UID: \"d5298fc7-1369-4e44-8ee3-f058d1e20c37\") " pod="openshift-ingress/payload-processing-e2e-mt-df6c3661-7f764d9794-cq64m"
Oct 30 13:16:12.461731 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:12.442844    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42c43460-b498-432d-a8e5-a68d5046aba6-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-df6c3661-5866c5b5c5-qfrtz\" (UID: \"42c43460-b498-432d-a8e5-a68d5046aba6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-df6c3661-5866c5b5c5-qfrtz"
Oct 30 13:16:12.461731 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:12.442880    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9jgk\" (UniqueName: \"kubernetes.io/projected/d5298fc7-1369-4e44-8ee3-f058d1e20c37-kube-api-access-z9jgk\") pod \"payload-processing-e2e-mt-df6c3661-7f764d9794-cq64m\" (UID: \"d5298fc7-1369-4e44-8ee3-f058d1e20c37\") " pod="openshift-ingress/payload-processing-e2e-mt-df6c3661-7f764d9794-cq64m"
Oct 30 13:16:12.461731 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:12.444665    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5298fc7-1369-4e44-8ee3-f058d1e20c37-plugins-config-volume\") pod \"payload-processing-e2e-mt-df6c3661-7f764d9794-cq64m\" (UID: \"d5298fc7-1369-4e44-8ee3-f058d1e20c37\") " pod="openshift-ingress/payload-processing-e2e-mt-df6c3661-7f764d9794-cq64m"
Oct 30 13:16:12.520629 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:12.519344    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9jgk\" (UniqueName: \"kubernetes.io/projected/d5298fc7-1369-4e44-8ee3-f058d1e20c37-kube-api-access-z9jgk\") pod \"payload-processing-e2e-mt-df6c3661-7f764d9794-cq64m\" (UID: \"d5298fc7-1369-4e44-8ee3-f058d1e20c37\") " pod="openshift-ingress/payload-processing-e2e-mt-df6c3661-7f764d9794-cq64m"
Oct 30 13:16:12.541323 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:12.539937    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-df6c3661-7f764d9794-cq64m"
Oct 30 13:16:12.563759 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:12.563277    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42c43460-b498-432d-a8e5-a68d5046aba6-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-df6c3661-5866c5b5c5-qfrtz\" (UID: \"42c43460-b498-432d-a8e5-a68d5046aba6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-df6c3661-5866c5b5c5-qfrtz"
Oct 30 13:16:12.609789 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:12.579144    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42c43460-b498-432d-a8e5-a68d5046aba6-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-df6c3661-5866c5b5c5-qfrtz\" (UID: \"42c43460-b498-432d-a8e5-a68d5046aba6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-df6c3661-5866c5b5c5-qfrtz"
Oct 30 13:16:12.609789 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:12.579260    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w5ld\" (UniqueName: \"kubernetes.io/projected/42c43460-b498-432d-a8e5-a68d5046aba6-kube-api-access-5w5ld\") pod \"payload-pre-processing-e2e-mt-df6c3661-5866c5b5c5-qfrtz\" (UID: \"42c43460-b498-432d-a8e5-a68d5046aba6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-df6c3661-5866c5b5c5-qfrtz"
Oct 30 13:16:12.934914 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:12.934862    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w5ld\" (UniqueName: \"kubernetes.io/projected/42c43460-b498-432d-a8e5-a68d5046aba6-kube-api-access-5w5ld\") pod \"payload-pre-processing-e2e-mt-df6c3661-5866c5b5c5-qfrtz\" (UID: \"42c43460-b498-432d-a8e5-a68d5046aba6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-df6c3661-5866c5b5c5-qfrtz"
Oct 30 13:16:13.204530 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:13.204444    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-df6c3661-5866c5b5c5-qfrtz"
Oct 30 13:16:13.256708 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:16:13.256680    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5298fc7_1369_4e44_8ee3_f058d1e20c37.slice/crio-9acf3b8a4609ee32265b50181b36dd682185c88a8e2fe0bd0c51ec18b9e4cb47 WatchSource:0}: Error finding container 9acf3b8a4609ee32265b50181b36dd682185c88a8e2fe0bd0c51ec18b9e4cb47: Status 404 returned error can't find the container with id 9acf3b8a4609ee32265b50181b36dd682185c88a8e2fe0bd0c51ec18b9e4cb47
Oct 30 13:16:13.258041 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:13.258014    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-df6c3661-7f764d9794-cq64m"]
Oct 30 13:16:13.518047 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:16:13.518021    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42c43460_b498_432d_a8e5_a68d5046aba6.slice/crio-28c1f443ead0042c785f8ff93665b7f118507c4e6caeab675e252e417dd34798 WatchSource:0}: Error finding container 28c1f443ead0042c785f8ff93665b7f118507c4e6caeab675e252e417dd34798: Status 404 returned error can't find the container with id 28c1f443ead0042c785f8ff93665b7f118507c4e6caeab675e252e417dd34798
Oct 30 13:16:13.518372 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:13.518349    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-df6c3661-5866c5b5c5-qfrtz"]
Oct 30 13:16:14.192353 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:14.192320    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-df6c3661-5866c5b5c5-qfrtz" event={"ID":"42c43460-b498-432d-a8e5-a68d5046aba6","Type":"ContainerStarted","Data":"86ea448ef0edc8f9d9e4529d7ee60513aa4b4acf04b74f17fa97c9f02304372d"}
Oct 30 13:16:14.192803 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:14.192367    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-df6c3661-5866c5b5c5-qfrtz" event={"ID":"42c43460-b498-432d-a8e5-a68d5046aba6","Type":"ContainerStarted","Data":"28c1f443ead0042c785f8ff93665b7f118507c4e6caeab675e252e417dd34798"}
Oct 30 13:16:14.192803 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:14.192543    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-df6c3661-5866c5b5c5-qfrtz"
Oct 30 13:16:14.194214 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:14.194188    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-df6c3661-7f764d9794-cq64m" event={"ID":"d5298fc7-1369-4e44-8ee3-f058d1e20c37","Type":"ContainerStarted","Data":"f247962a8f3f262f84177b9495a9bb7231e51ff3c47580a505605210a1c52783"}
Oct 30 13:16:14.194333 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:14.194225    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-df6c3661-7f764d9794-cq64m" event={"ID":"d5298fc7-1369-4e44-8ee3-f058d1e20c37","Type":"ContainerStarted","Data":"9acf3b8a4609ee32265b50181b36dd682185c88a8e2fe0bd0c51ec18b9e4cb47"}
Oct 30 13:16:14.194333 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:14.194253    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-df6c3661-7f764d9794-cq64m"
Oct 30 13:16:14.237588 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:14.237543    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-df6c3661-5866c5b5c5-qfrtz" podStartSLOduration=2.237530201 podStartE2EDuration="2.237530201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:16:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:16:14.232253745 +0000 UTC m=+1473.398618414" watchObservedRunningTime="2026-10-30 13:16:14.237530201 +0000 UTC m=+1473.403894848"
Oct 30 13:16:14.272792 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:14.272745    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-df6c3661-7f764d9794-cq64m" podStartSLOduration=2.272731034 podStartE2EDuration="2.272731034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:16:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:16:14.272053077 +0000 UTC m=+1473.438417725" watchObservedRunningTime="2026-10-30 13:16:14.272731034 +0000 UTC m=+1473.439095685"
Oct 30 13:16:15.211506 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:15.211467    4364 generic.go:356] "Generic (PLEG): container finished" podID="42d369e1-ee25-47fe-a808-cb0470669962" containerID="835dfbab0f094930fb9b622678c7abd6d9ac1a0d33b57275fd0ec8599ad319d5" exitCode=0
Oct 30 13:16:15.212074 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:15.211581    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh" event={"ID":"42d369e1-ee25-47fe-a808-cb0470669962","Type":"ContainerDied","Data":"835dfbab0f094930fb9b622678c7abd6d9ac1a0d33b57275fd0ec8599ad319d5"}
Oct 30 13:16:15.214983 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:15.214956    4364 generic.go:356] "Generic (PLEG): container finished" podID="747bdafd-1a41-4b0e-9547-f14fe568d3f1" containerID="c6b047c72d2a3839a86bbecb3a07abaaed62689d2473a04811aa1cc928573f11" exitCode=0
Oct 30 13:16:15.216170 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:15.216143    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr" event={"ID":"747bdafd-1a41-4b0e-9547-f14fe568d3f1","Type":"ContainerDied","Data":"c6b047c72d2a3839a86bbecb3a07abaaed62689d2473a04811aa1cc928573f11"}
Oct 30 13:16:15.295451 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:15.295427    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr"
Oct 30 13:16:15.323921 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:15.323896    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh"
Oct 30 13:16:15.390994 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:15.390963    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/747bdafd-1a41-4b0e-9547-f14fe568d3f1-kube-api-access-sr82m\" (UniqueName: \"kubernetes.io/projected/747bdafd-1a41-4b0e-9547-f14fe568d3f1-kube-api-access-sr82m\") pod \"747bdafd-1a41-4b0e-9547-f14fe568d3f1\" (UID: \"747bdafd-1a41-4b0e-9547-f14fe568d3f1\") "
Oct 30 13:16:15.391169 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:15.391024    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42d369e1-ee25-47fe-a808-cb0470669962-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d369e1-ee25-47fe-a808-cb0470669962-workload-socket\") pod \"42d369e1-ee25-47fe-a808-cb0470669962\" (UID: \"42d369e1-ee25-47fe-a808-cb0470669962\") "
Oct 30 13:16:15.391169 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:15.391075    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/42d369e1-ee25-47fe-a808-cb0470669962-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42d369e1-ee25-47fe-a808-cb0470669962-istio-podinfo\") pod \"42d369e1-ee25-47fe-a808-cb0470669962\" (UID: \"42d369e1-ee25-47fe-a808-cb0470669962\") "
Oct 30 13:16:15.391169 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:15.391098    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/747bdafd-1a41-4b0e-9547-f14fe568d3f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/747bdafd-1a41-4b0e-9547-f14fe568d3f1-istio-envoy\") pod \"747bdafd-1a41-4b0e-9547-f14fe568d3f1\" (UID: \"747bdafd-1a41-4b0e-9547-f14fe568d3f1\") "
Oct 30 13:16:15.391361 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:15.391178    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/747bdafd-1a41-4b0e-9547-f14fe568d3f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/747bdafd-1a41-4b0e-9547-f14fe568d3f1-workload-socket\") pod \"747bdafd-1a41-4b0e-9547-f14fe568d3f1\" (UID: \"747bdafd-1a41-4b0e-9547-f14fe568d3f1\") "
Oct 30 13:16:15.391361 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:15.391216    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/747bdafd-1a41-4b0e-9547-f14fe568d3f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/747bdafd-1a41-4b0e-9547-f14fe568d3f1-istiod-ca-cert\") pod \"747bdafd-1a41-4b0e-9547-f14fe568d3f1\" (UID: \"747bdafd-1a41-4b0e-9547-f14fe568d3f1\") "
Oct 30 13:16:15.391361 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:15.391341    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42d369e1-ee25-47fe-a808-cb0470669962-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42d369e1-ee25-47fe-a808-cb0470669962-istio-data\") pod \"42d369e1-ee25-47fe-a808-cb0470669962\" (UID: \"42d369e1-ee25-47fe-a808-cb0470669962\") "
Oct 30 13:16:15.391528 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:15.391379    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42d369e1-ee25-47fe-a808-cb0470669962-workload-socket" pod "42d369e1-ee25-47fe-a808-cb0470669962" (UID: "42d369e1-ee25-47fe-a808-cb0470669962"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:16:15.391528 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:15.391398    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/747bdafd-1a41-4b0e-9547-f14fe568d3f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/747bdafd-1a41-4b0e-9547-f14fe568d3f1-istio-podinfo\") pod \"747bdafd-1a41-4b0e-9547-f14fe568d3f1\" (UID: \"747bdafd-1a41-4b0e-9547-f14fe568d3f1\") "
Oct 30 13:16:15.391528 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:15.391477    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/747bdafd-1a41-4b0e-9547-f14fe568d3f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/747bdafd-1a41-4b0e-9547-f14fe568d3f1-workload-certs\") pod \"747bdafd-1a41-4b0e-9547-f14fe568d3f1\" (UID: \"747bdafd-1a41-4b0e-9547-f14fe568d3f1\") "
Oct 30 13:16:15.391528 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:15.391508    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/747bdafd-1a41-4b0e-9547-f14fe568d3f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/747bdafd-1a41-4b0e-9547-f14fe568d3f1-credential-socket\") pod \"747bdafd-1a41-4b0e-9547-f14fe568d3f1\" (UID: \"747bdafd-1a41-4b0e-9547-f14fe568d3f1\") "
Oct 30 13:16:15.391822 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:15.391548    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42d369e1-ee25-47fe-a808-cb0470669962-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42d369e1-ee25-47fe-a808-cb0470669962-istiod-ca-cert\") pod \"42d369e1-ee25-47fe-a808-cb0470669962\" (UID: \"42d369e1-ee25-47fe-a808-cb0470669962\") "
Oct 30 13:16:15.391822 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:15.391636    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42d369e1-ee25-47fe-a808-cb0470669962-kube-api-access-p8zz9\" (UniqueName: \"kubernetes.io/projected/42d369e1-ee25-47fe-a808-cb0470669962-kube-api-access-p8zz9\") pod \"42d369e1-ee25-47fe-a808-cb0470669962\" (UID: \"42d369e1-ee25-47fe-a808-cb0470669962\") "
Oct 30 13:16:15.391822 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:15.391657    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/747bdafd-1a41-4b0e-9547-f14fe568d3f1-workload-socket" pod "747bdafd-1a41-4b0e-9547-f14fe568d3f1" (UID: "747bdafd-1a41-4b0e-9547-f14fe568d3f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:16:15.391822 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:15.391713    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42d369e1-ee25-47fe-a808-cb0470669962-istio-token\" (UniqueName: \"kubernetes.io/projected/42d369e1-ee25-47fe-a808-cb0470669962-istio-token\") pod \"42d369e1-ee25-47fe-a808-cb0470669962\" (UID: \"42d369e1-ee25-47fe-a808-cb0470669962\") "
Oct 30 13:16:15.391822 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:15.391800    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/747bdafd-1a41-4b0e-9547-f14fe568d3f1-istio-token\" (UniqueName: \"kubernetes.io/projected/747bdafd-1a41-4b0e-9547-f14fe568d3f1-istio-token\") pod \"747bdafd-1a41-4b0e-9547-f14fe568d3f1\" (UID: \"747bdafd-1a41-4b0e-9547-f14fe568d3f1\") "
Oct 30 13:16:15.392091 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:15.391884    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42d369e1-ee25-47fe-a808-cb0470669962-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42d369e1-ee25-47fe-a808-cb0470669962-istio-envoy\") pod \"42d369e1-ee25-47fe-a808-cb0470669962\" (UID: \"42d369e1-ee25-47fe-a808-cb0470669962\") "
Oct 30 13:16:15.392091 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:15.391962    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42d369e1-ee25-47fe-a808-cb0470669962-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d369e1-ee25-47fe-a808-cb0470669962-credential-socket\") pod \"42d369e1-ee25-47fe-a808-cb0470669962\" (UID: \"42d369e1-ee25-47fe-a808-cb0470669962\") "
Oct 30 13:16:15.392091 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:15.392000    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/747bdafd-1a41-4b0e-9547-f14fe568d3f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/747bdafd-1a41-4b0e-9547-f14fe568d3f1-istio-data\") pod \"747bdafd-1a41-4b0e-9547-f14fe568d3f1\" (UID: \"747bdafd-1a41-4b0e-9547-f14fe568d3f1\") "
Oct 30 13:16:15.392091 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:15.392031    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42d369e1-ee25-47fe-a808-cb0470669962-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42d369e1-ee25-47fe-a808-cb0470669962-workload-certs\") pod \"42d369e1-ee25-47fe-a808-cb0470669962\" (UID: \"42d369e1-ee25-47fe-a808-cb0470669962\") "
Oct 30 13:16:15.392308 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:15.392128    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/747bdafd-1a41-4b0e-9547-f14fe568d3f1-credential-socket" pod "747bdafd-1a41-4b0e-9547-f14fe568d3f1" (UID: "747bdafd-1a41-4b0e-9547-f14fe568d3f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:16:15.392993 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:15.392542    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/747bdafd-1a41-4b0e-9547-f14fe568d3f1-istiod-ca-cert" pod "747bdafd-1a41-4b0e-9547-f14fe568d3f1" (UID: "747bdafd-1a41-4b0e-9547-f14fe568d3f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:16:15.392993 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:15.392616    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42d369e1-ee25-47fe-a808-cb0470669962-istiod-ca-cert" pod "42d369e1-ee25-47fe-a808-cb0470669962" (UID: "42d369e1-ee25-47fe-a808-cb0470669962"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:16:15.392993 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:15.392781    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42d369e1-ee25-47fe-a808-cb0470669962-workload-certs" pod "42d369e1-ee25-47fe-a808-cb0470669962" (UID: "42d369e1-ee25-47fe-a808-cb0470669962"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:16:15.392993 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:15.392855    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42d369e1-ee25-47fe-a808-cb0470669962-istio-data" pod "42d369e1-ee25-47fe-a808-cb0470669962" (UID: "42d369e1-ee25-47fe-a808-cb0470669962"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:16:15.392993 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:15.392966    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42d369e1-ee25-47fe-a808-cb0470669962-credential-socket" pod "42d369e1-ee25-47fe-a808-cb0470669962" (UID: "42d369e1-ee25-47fe-a808-cb0470669962"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:16:15.393350 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:15.393012    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/747bdafd-1a41-4b0e-9547-f14fe568d3f1-istio-data" pod "747bdafd-1a41-4b0e-9547-f14fe568d3f1" (UID: "747bdafd-1a41-4b0e-9547-f14fe568d3f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:16:15.393350 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:15.393209    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/747bdafd-1a41-4b0e-9547-f14fe568d3f1-workload-certs" pod "747bdafd-1a41-4b0e-9547-f14fe568d3f1" (UID: "747bdafd-1a41-4b0e-9547-f14fe568d3f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:16:15.393558 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:15.393533    4364 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d369e1-ee25-47fe-a808-cb0470669962-credential-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:16:15.393701 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:15.393564    4364 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/747bdafd-1a41-4b0e-9547-f14fe568d3f1-istio-data\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:16:15.393701 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:15.393578    4364 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42d369e1-ee25-47fe-a808-cb0470669962-workload-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:16:15.393701 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:15.393593    4364 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d369e1-ee25-47fe-a808-cb0470669962-workload-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:16:15.393701 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:15.393623    4364 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/747bdafd-1a41-4b0e-9547-f14fe568d3f1-workload-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:16:15.393701 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:15.393637    4364 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/747bdafd-1a41-4b0e-9547-f14fe568d3f1-istiod-ca-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:16:15.393701 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:15.393649    4364 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42d369e1-ee25-47fe-a808-cb0470669962-istio-data\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:16:15.393701 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:15.393661    4364 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/747bdafd-1a41-4b0e-9547-f14fe568d3f1-workload-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:16:15.393701 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:15.393675    4364 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/747bdafd-1a41-4b0e-9547-f14fe568d3f1-credential-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:16:15.393701 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:15.393688    4364 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42d369e1-ee25-47fe-a808-cb0470669962-istiod-ca-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:16:15.394182 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:15.394101    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/42d369e1-ee25-47fe-a808-cb0470669962-istio-podinfo" pod "42d369e1-ee25-47fe-a808-cb0470669962" (UID: "42d369e1-ee25-47fe-a808-cb0470669962"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:16:15.394318 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:15.394295    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/747bdafd-1a41-4b0e-9547-f14fe568d3f1-kube-api-access-sr82m" pod "747bdafd-1a41-4b0e-9547-f14fe568d3f1" (UID: "747bdafd-1a41-4b0e-9547-f14fe568d3f1"). InnerVolumeSpecName "kube-api-access-sr82m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:16:15.395402 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:15.395372    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/747bdafd-1a41-4b0e-9547-f14fe568d3f1-istio-envoy" pod "747bdafd-1a41-4b0e-9547-f14fe568d3f1" (UID: "747bdafd-1a41-4b0e-9547-f14fe568d3f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:16:15.395505 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:15.395406    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42d369e1-ee25-47fe-a808-cb0470669962-istio-token" pod "42d369e1-ee25-47fe-a808-cb0470669962" (UID: "42d369e1-ee25-47fe-a808-cb0470669962"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:16:15.396612 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:15.396575    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42d369e1-ee25-47fe-a808-cb0470669962-istio-envoy" pod "42d369e1-ee25-47fe-a808-cb0470669962" (UID: "42d369e1-ee25-47fe-a808-cb0470669962"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:16:15.396714 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:15.396593    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42d369e1-ee25-47fe-a808-cb0470669962-kube-api-access-p8zz9" pod "42d369e1-ee25-47fe-a808-cb0470669962" (UID: "42d369e1-ee25-47fe-a808-cb0470669962"). InnerVolumeSpecName "kube-api-access-p8zz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:16:15.397200 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:15.397177    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/747bdafd-1a41-4b0e-9547-f14fe568d3f1-istio-podinfo" pod "747bdafd-1a41-4b0e-9547-f14fe568d3f1" (UID: "747bdafd-1a41-4b0e-9547-f14fe568d3f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:16:15.397200 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:15.397185    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/747bdafd-1a41-4b0e-9547-f14fe568d3f1-istio-token" pod "747bdafd-1a41-4b0e-9547-f14fe568d3f1" (UID: "747bdafd-1a41-4b0e-9547-f14fe568d3f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:16:15.495562 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:15.495478    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8zz9\" (UniqueName: \"kubernetes.io/projected/42d369e1-ee25-47fe-a808-cb0470669962-kube-api-access-p8zz9\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:16:15.495562 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:15.495515    4364 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42d369e1-ee25-47fe-a808-cb0470669962-istio-token\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:16:15.495562 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:15.495529    4364 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/747bdafd-1a41-4b0e-9547-f14fe568d3f1-istio-token\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:16:15.495562 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:15.495542    4364 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42d369e1-ee25-47fe-a808-cb0470669962-istio-envoy\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:16:15.495562 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:15.495554    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sr82m\" (UniqueName: \"kubernetes.io/projected/747bdafd-1a41-4b0e-9547-f14fe568d3f1-kube-api-access-sr82m\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:16:15.495562 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:15.495566    4364 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42d369e1-ee25-47fe-a808-cb0470669962-istio-podinfo\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:16:15.496051 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:15.495579    4364 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/747bdafd-1a41-4b0e-9547-f14fe568d3f1-istio-envoy\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:16:15.496051 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:15.495592    4364 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/747bdafd-1a41-4b0e-9547-f14fe568d3f1-istio-podinfo\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:16:16.226178 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:16.226148    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr"
Oct 30 13:16:16.226708 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:16.226145    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr" event={"ID":"747bdafd-1a41-4b0e-9547-f14fe568d3f1","Type":"ContainerDied","Data":"2c20e055bc19ee5e57034cb9ef4538b8648212618f5b3febd4b6394be57239f7"}
Oct 30 13:16:16.226708 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:16.226320    4364 scope.go:122] "RemoveContainer" containerID="c6b047c72d2a3839a86bbecb3a07abaaed62689d2473a04811aa1cc928573f11"
Oct 30 13:16:16.228546 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:16.228522    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh" event={"ID":"42d369e1-ee25-47fe-a808-cb0470669962","Type":"ContainerDied","Data":"8b49bab2db54d09ac276b584cb1b2521217e73cfefbfc23054772227b6f55661"}
Oct 30 13:16:16.228887 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:16.228628    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh"
Oct 30 13:16:16.245575 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:16.245547    4364 scope.go:122] "RemoveContainer" containerID="835dfbab0f094930fb9b622678c7abd6d9ac1a0d33b57275fd0ec8599ad319d5"
Oct 30 13:16:16.298993 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:16.298961    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr"]
Oct 30 13:16:16.309676 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:16.309649    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-fw4fr"]
Oct 30 13:16:16.335796 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:16.335767    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh"]
Oct 30 13:16:16.341388 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:16.341364    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-x6zgh"]
Oct 30 13:16:16.973184 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:16.973148    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42d369e1-ee25-47fe-a808-cb0470669962" path="/var/lib/kubelet/pods/42d369e1-ee25-47fe-a808-cb0470669962/volumes"
Oct 30 13:16:16.973704 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:16.973690    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="747bdafd-1a41-4b0e-9547-f14fe568d3f1" path="/var/lib/kubelet/pods/747bdafd-1a41-4b0e-9547-f14fe568d3f1/volumes"
Oct 30 13:16:17.030302 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:17.030274    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-cda0d9-74764bd4cf-l64mp"
Oct 30 13:16:21.165376 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:21.165344    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4644a1-74c746f55f-rkfmk"
Oct 30 13:16:22.865622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:22.863052    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:16:25.259906 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:25.259695    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-df6c3661-7f764d9794-cq64m"
Oct 30 13:16:25.268906 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:25.267827    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-df6c3661-5866c5b5c5-qfrtz"
Oct 30 13:16:28.230754 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:28.230671    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74" podUID="4f2b06ac-9bf8-4b57-bd22-85532576f9c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded"
Oct 30 13:16:28.237721 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:28.237672    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh" podUID="d41ac9e5-eeed-4e07-a7fa-05d3260630b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 30 13:16:28.244694 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:28.244648    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc" podUID="8911b790-331d-490c-876b-647e6d2fcd14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:16:28.284621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:28.280668    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz" podUID="a568f1ce-602d-4069-a41c-9fe1a7635c5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 30 13:16:28.376622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:28.372665    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4" podUID="0cbe6af7-d884-4613-90b8-192e5ffdb027" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:16:28.376622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:28.373020    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr" podUID="446c12df-de12-4c7c-8c71-dfed93cc33f1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:16:28.385762 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:28.385107    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz" podUID="bc5caf69-a649-47e5-8910-9b8f331dfd4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:16:28.856237 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:28.856194    4364 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz" podUID="bc5caf69-a649-47e5-8910-9b8f331dfd4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:16:28.869905 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:28.869871    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-84942a-7d58989cc9-j2l2j"]
Oct 30 13:16:28.871441 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:28.871414    4364 cpu_manager.go:414] "containerMap: removing container" podUID="42d369e1-ee25-47fe-a808-cb0470669962" containerName="istio-proxy"
Oct 30 13:16:28.871617 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:28.871584    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42d369e1-ee25-47fe-a808-cb0470669962" containerName="istio-proxy"
Oct 30 13:16:28.871745 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:28.871732    4364 cpu_manager.go:414] "containerMap: removing container" podUID="747bdafd-1a41-4b0e-9547-f14fe568d3f1" containerName="istio-proxy"
Oct 30 13:16:28.871840 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:28.871827    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="747bdafd-1a41-4b0e-9547-f14fe568d3f1" containerName="istio-proxy"
Oct 30 13:16:28.872167 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:28.872148    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="42d369e1-ee25-47fe-a808-cb0470669962" containerName="istio-proxy"
Oct 30 13:16:28.872279 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:28.872268    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="747bdafd-1a41-4b0e-9547-f14fe568d3f1" containerName="istio-proxy"
Oct 30 13:16:28.900952 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:28.896686    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6" podUID="b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:16:28.938521 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:28.938488    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-84942a-7d58989cc9-j2l2j"]
Oct 30 13:16:28.938889 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:28.938816    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84942a-7d58989cc9-j2l2j"
Oct 30 13:16:28.970351 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:28.970096    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-84942a-serving-cert\""
Oct 30 13:16:29.044620 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:29.042592    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/644cd57e-9920-4049-a7f6-d4a41a5069e4-metrics-certs\") pod \"maas-api-e2e-api-b-84942a-7d58989cc9-j2l2j\" (UID: \"644cd57e-9920-4049-a7f6-d4a41a5069e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84942a-7d58989cc9-j2l2j"
Oct 30 13:16:29.044620 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:29.042683    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/644cd57e-9920-4049-a7f6-d4a41a5069e4-openshift-service-ca\") pod \"maas-api-e2e-api-b-84942a-7d58989cc9-j2l2j\" (UID: \"644cd57e-9920-4049-a7f6-d4a41a5069e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84942a-7d58989cc9-j2l2j"
Oct 30 13:16:29.044620 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:29.042946    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/644cd57e-9920-4049-a7f6-d4a41a5069e4-maas-api-tls\") pod \"maas-api-e2e-api-b-84942a-7d58989cc9-j2l2j\" (UID: \"644cd57e-9920-4049-a7f6-d4a41a5069e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84942a-7d58989cc9-j2l2j"
Oct 30 13:16:29.044620 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:29.043196    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlqs5\" (UniqueName: \"kubernetes.io/projected/644cd57e-9920-4049-a7f6-d4a41a5069e4-kube-api-access-mlqs5\") pod \"maas-api-e2e-api-b-84942a-7d58989cc9-j2l2j\" (UID: \"644cd57e-9920-4049-a7f6-d4a41a5069e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84942a-7d58989cc9-j2l2j"
Oct 30 13:16:29.144410 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:29.144380    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlqs5\" (UniqueName: \"kubernetes.io/projected/644cd57e-9920-4049-a7f6-d4a41a5069e4-kube-api-access-mlqs5\") pod \"maas-api-e2e-api-b-84942a-7d58989cc9-j2l2j\" (UID: \"644cd57e-9920-4049-a7f6-d4a41a5069e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84942a-7d58989cc9-j2l2j"
Oct 30 13:16:29.144581 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:29.144445    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/644cd57e-9920-4049-a7f6-d4a41a5069e4-metrics-certs\") pod \"maas-api-e2e-api-b-84942a-7d58989cc9-j2l2j\" (UID: \"644cd57e-9920-4049-a7f6-d4a41a5069e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84942a-7d58989cc9-j2l2j"
Oct 30 13:16:29.144581 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:29.144479    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/644cd57e-9920-4049-a7f6-d4a41a5069e4-openshift-service-ca\") pod \"maas-api-e2e-api-b-84942a-7d58989cc9-j2l2j\" (UID: \"644cd57e-9920-4049-a7f6-d4a41a5069e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84942a-7d58989cc9-j2l2j"
Oct 30 13:16:29.144677 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:29.144593    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/644cd57e-9920-4049-a7f6-d4a41a5069e4-maas-api-tls\") pod \"maas-api-e2e-api-b-84942a-7d58989cc9-j2l2j\" (UID: \"644cd57e-9920-4049-a7f6-d4a41a5069e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84942a-7d58989cc9-j2l2j"
Oct 30 13:16:29.145248 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:29.145224    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/644cd57e-9920-4049-a7f6-d4a41a5069e4-openshift-service-ca\") pod \"maas-api-e2e-api-b-84942a-7d58989cc9-j2l2j\" (UID: \"644cd57e-9920-4049-a7f6-d4a41a5069e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84942a-7d58989cc9-j2l2j"
Oct 30 13:16:29.146997 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:29.146969    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/644cd57e-9920-4049-a7f6-d4a41a5069e4-metrics-certs\") pod \"maas-api-e2e-api-b-84942a-7d58989cc9-j2l2j\" (UID: \"644cd57e-9920-4049-a7f6-d4a41a5069e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84942a-7d58989cc9-j2l2j"
Oct 30 13:16:29.147121 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:29.147063    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/644cd57e-9920-4049-a7f6-d4a41a5069e4-maas-api-tls\") pod \"maas-api-e2e-api-b-84942a-7d58989cc9-j2l2j\" (UID: \"644cd57e-9920-4049-a7f6-d4a41a5069e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84942a-7d58989cc9-j2l2j"
Oct 30 13:16:29.163317 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:29.163294    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-659d72-8glwc"]
Oct 30 13:16:29.169697 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:29.169680    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-659d72-8glwc"
Oct 30 13:16:29.173512 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:29.173489    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlqs5\" (UniqueName: \"kubernetes.io/projected/644cd57e-9920-4049-a7f6-d4a41a5069e4-kube-api-access-mlqs5\") pod \"maas-api-e2e-api-b-84942a-7d58989cc9-j2l2j\" (UID: \"644cd57e-9920-4049-a7f6-d4a41a5069e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84942a-7d58989cc9-j2l2j"
Oct 30 13:16:29.210622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:29.205396    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-659d72-8glwc"]
Oct 30 13:16:29.227627 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:29.217039    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:16:29.227627 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:29.217806    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\""
Oct 30 13:16:29.297618 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:29.297339    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djspv\" (UniqueName: \"kubernetes.io/projected/f10e7b19-6e46-4e00-b631-6bc0a94b98a7-kube-api-access-djspv\") pod \"maas-api-revoke-keys-e2e-api-a-659d72-8glwc\" (UID: \"f10e7b19-6e46-4e00-b631-6bc0a94b98a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-659d72-8glwc"
Oct 30 13:16:29.298182 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:29.297765    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f10e7b19-6e46-4e00-b631-6bc0a94b98a7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-659d72-8glwc\" (UID: \"f10e7b19-6e46-4e00-b631-6bc0a94b98a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-659d72-8glwc"
Oct 30 13:16:29.298182 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:29.297918    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f10e7b19-6e46-4e00-b631-6bc0a94b98a7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-659d72-8glwc\" (UID: \"f10e7b19-6e46-4e00-b631-6bc0a94b98a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-659d72-8glwc"
Oct 30 13:16:29.389683 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:29.370041    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84942a-7d58989cc9-j2l2j"
Oct 30 13:16:29.447280 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:29.436149    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f10e7b19-6e46-4e00-b631-6bc0a94b98a7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-659d72-8glwc\" (UID: \"f10e7b19-6e46-4e00-b631-6bc0a94b98a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-659d72-8glwc"
Oct 30 13:16:29.447280 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:29.436320    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f10e7b19-6e46-4e00-b631-6bc0a94b98a7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-659d72-8glwc\" (UID: \"f10e7b19-6e46-4e00-b631-6bc0a94b98a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-659d72-8glwc"
Oct 30 13:16:29.447280 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:29.436487    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djspv\" (UniqueName: \"kubernetes.io/projected/f10e7b19-6e46-4e00-b631-6bc0a94b98a7-kube-api-access-djspv\") pod \"maas-api-revoke-keys-e2e-api-a-659d72-8glwc\" (UID: \"f10e7b19-6e46-4e00-b631-6bc0a94b98a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-659d72-8glwc"
Oct 30 13:16:29.447280 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:29.437902    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f10e7b19-6e46-4e00-b631-6bc0a94b98a7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-659d72-8glwc\" (UID: \"f10e7b19-6e46-4e00-b631-6bc0a94b98a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-659d72-8glwc"
Oct 30 13:16:29.470581 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:29.470551    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djspv\" (UniqueName: \"kubernetes.io/projected/f10e7b19-6e46-4e00-b631-6bc0a94b98a7-kube-api-access-djspv\") pod \"maas-api-revoke-keys-e2e-api-a-659d72-8glwc\" (UID: \"f10e7b19-6e46-4e00-b631-6bc0a94b98a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-659d72-8glwc"
Oct 30 13:16:29.471167 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:29.471141    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f10e7b19-6e46-4e00-b631-6bc0a94b98a7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-659d72-8glwc\" (UID: \"f10e7b19-6e46-4e00-b631-6bc0a94b98a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-659d72-8glwc"
Oct 30 13:16:29.765651 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:29.765566    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-659d72-8glwc"
Oct 30 13:16:29.940670 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:29.940614    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-84942a-7d58989cc9-j2l2j"]
Oct 30 13:16:30.472623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:30.449976    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-659d72-8glwc"]
Oct 30 13:16:30.553162 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:16:30.553125    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf10e7b19_6e46_4e00_b631_6bc0a94b98a7.slice/crio-653f35edb344bbe2f32b0a19e98251835e66573eb1880ad4a502a5f7e44e61ec WatchSource:0}: Error finding container 653f35edb344bbe2f32b0a19e98251835e66573eb1880ad4a502a5f7e44e61ec: Status 404 returned error can't find the container with id 653f35edb344bbe2f32b0a19e98251835e66573eb1880ad4a502a5f7e44e61ec
Oct 30 13:16:30.734576 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:30.732655    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84942a-7d58989cc9-j2l2j" event={"ID":"644cd57e-9920-4049-a7f6-d4a41a5069e4","Type":"ContainerStarted","Data":"f7e4ae8a0f3253dd1bb2d5bb0331193bc9f09c5c55ff8494db9248c46a343796"}
Oct 30 13:16:30.734576 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:30.734188    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-659d72-8glwc" event={"ID":"f10e7b19-6e46-4e00-b631-6bc0a94b98a7","Type":"ContainerStarted","Data":"653f35edb344bbe2f32b0a19e98251835e66573eb1880ad4a502a5f7e44e61ec"}
Oct 30 13:16:31.745693 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:31.745647    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-659d72-8glwc" event={"ID":"f10e7b19-6e46-4e00-b631-6bc0a94b98a7","Type":"ContainerStarted","Data":"4715374acb1486834f3cc751a2079c3ff0a97bc45ac5aeae4394a5aa7e67b9d5"}
Oct 30 13:16:31.747901 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:31.747867    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84942a-7d58989cc9-j2l2j" event={"ID":"644cd57e-9920-4049-a7f6-d4a41a5069e4","Type":"ContainerStarted","Data":"037ee5a283f37e7bd69f9b4da00e753a3288dabcc4f160139c97900077649b30"}
Oct 30 13:16:31.748445 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:31.748424    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84942a-7d58989cc9-j2l2j"
Oct 30 13:16:31.791407 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:31.791363    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-659d72-8glwc" podStartSLOduration=2.791350079 podStartE2EDuration="2.791350079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:16:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:16:31.785215831 +0000 UTC m=+1490.951580480" watchObservedRunningTime="2026-10-30 13:16:31.791350079 +0000 UTC m=+1490.957714727"
Oct 30 13:16:31.830944 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:31.830901    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84942a-7d58989cc9-j2l2j" podStartSLOduration=3.172550877 podStartE2EDuration="3.830885018s" totalImagesPullingTime="658.334141ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:16:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:16:31.829754739 +0000 UTC m=+1490.996119387" watchObservedRunningTime="2026-10-30 13:16:31.830885018 +0000 UTC m=+1490.997249668"
Oct 30 13:16:32.760661 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:32.760624    4364 generic.go:356] "Generic (PLEG): container finished" podID="f10e7b19-6e46-4e00-b631-6bc0a94b98a7" containerID="4715374acb1486834f3cc751a2079c3ff0a97bc45ac5aeae4394a5aa7e67b9d5" exitCode=0
Oct 30 13:16:32.761040 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:32.760698    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-659d72-8glwc" event={"ID":"f10e7b19-6e46-4e00-b631-6bc0a94b98a7","Type":"ContainerDied","Data":"4715374acb1486834f3cc751a2079c3ff0a97bc45ac5aeae4394a5aa7e67b9d5"}
Oct 30 13:16:33.961774 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:33.961749    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-659d72-8glwc"
Oct 30 13:16:34.021894 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:34.021820    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f10e7b19-6e46-4e00-b631-6bc0a94b98a7-kube-api-access-djspv\" (UniqueName: \"kubernetes.io/projected/f10e7b19-6e46-4e00-b631-6bc0a94b98a7-kube-api-access-djspv\") pod \"f10e7b19-6e46-4e00-b631-6bc0a94b98a7\" (UID: \"f10e7b19-6e46-4e00-b631-6bc0a94b98a7\") "
Oct 30 13:16:34.021894 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:34.021895    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f10e7b19-6e46-4e00-b631-6bc0a94b98a7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f10e7b19-6e46-4e00-b631-6bc0a94b98a7-maas-api-openshift-ca\") pod \"f10e7b19-6e46-4e00-b631-6bc0a94b98a7\" (UID: \"f10e7b19-6e46-4e00-b631-6bc0a94b98a7\") "
Oct 30 13:16:34.022121 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:34.021953    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f10e7b19-6e46-4e00-b631-6bc0a94b98a7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f10e7b19-6e46-4e00-b631-6bc0a94b98a7-maas-api-xks-ca\") pod \"f10e7b19-6e46-4e00-b631-6bc0a94b98a7\" (UID: \"f10e7b19-6e46-4e00-b631-6bc0a94b98a7\") "
Oct 30 13:16:34.024341 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:34.024301    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f10e7b19-6e46-4e00-b631-6bc0a94b98a7-maas-api-openshift-ca" pod "f10e7b19-6e46-4e00-b631-6bc0a94b98a7" (UID: "f10e7b19-6e46-4e00-b631-6bc0a94b98a7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:16:34.026298 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:34.026261    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f10e7b19-6e46-4e00-b631-6bc0a94b98a7-kube-api-access-djspv" pod "f10e7b19-6e46-4e00-b631-6bc0a94b98a7" (UID: "f10e7b19-6e46-4e00-b631-6bc0a94b98a7"). InnerVolumeSpecName "kube-api-access-djspv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:16:34.033861 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:34.033837    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f10e7b19-6e46-4e00-b631-6bc0a94b98a7-maas-api-xks-ca" pod "f10e7b19-6e46-4e00-b631-6bc0a94b98a7" (UID: "f10e7b19-6e46-4e00-b631-6bc0a94b98a7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:16:34.123845 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:34.123810    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djspv\" (UniqueName: \"kubernetes.io/projected/f10e7b19-6e46-4e00-b631-6bc0a94b98a7-kube-api-access-djspv\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:16:34.123845 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:34.123847    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f10e7b19-6e46-4e00-b631-6bc0a94b98a7-maas-api-openshift-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:16:34.124053 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:34.123862    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f10e7b19-6e46-4e00-b631-6bc0a94b98a7-maas-api-xks-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:16:34.783726 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:34.783694    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-659d72-8glwc"
Oct 30 13:16:34.783888 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:34.783723    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-659d72-8glwc" event={"ID":"f10e7b19-6e46-4e00-b631-6bc0a94b98a7","Type":"ContainerDied","Data":"653f35edb344bbe2f32b0a19e98251835e66573eb1880ad4a502a5f7e44e61ec"}
Oct 30 13:16:34.783888 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:34.783782    4364 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="653f35edb344bbe2f32b0a19e98251835e66573eb1880ad4a502a5f7e44e61ec"
Oct 30 13:16:35.700123 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:35.698762    4364 patch_prober.go:29] interesting pod/e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": read tcp 10.133.0.2:59714->10.133.0.135:15021: read: connection reset by peer" start-of-body=
Oct 30 13:16:35.700123 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:35.698829    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm" podUID="9ef21594-2f1f-4b43-9c5f-529c9f9763f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": read tcp 10.133.0.2:59714->10.133.0.135:15021: read: connection reset by peer"
Oct 30 13:16:35.793475 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:35.793447    4364 generic.go:356] "Generic (PLEG): container finished" podID="9ef21594-2f1f-4b43-9c5f-529c9f9763f7" containerID="e07d871ed6035bbaa3a9a1cf1365580f2ebcbb398ca33e4cebc9e511aa7a3527" exitCode=0
Oct 30 13:16:35.793578 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:35.793497    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm" event={"ID":"9ef21594-2f1f-4b43-9c5f-529c9f9763f7","Type":"ContainerDied","Data":"e07d871ed6035bbaa3a9a1cf1365580f2ebcbb398ca33e4cebc9e511aa7a3527"}
Oct 30 13:16:35.793891 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:35.793876    4364 scope.go:122] "RemoveContainer" containerID="e07d871ed6035bbaa3a9a1cf1365580f2ebcbb398ca33e4cebc9e511aa7a3527"
Oct 30 13:16:35.795560 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:35.795534    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:16:35.795640 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:35.795626    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:16:35.795680 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:35.795661    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:16:36.817247 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:36.817212    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm" event={"ID":"9ef21594-2f1f-4b43-9c5f-529c9f9763f7","Type":"ContainerStarted","Data":"800d907b4ccaf08e8c7e840e40094698c1c300512259106b4b43ee082b51f297"}
Oct 30 13:16:37.497909 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:37.495699    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm"
Oct 30 13:16:37.513625 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:37.507473    4364 patch_prober.go:29] interesting pod/e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Oct 30 13:16:37.513625 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:37.507544    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm" podUID="9ef21594-2f1f-4b43-9c5f-529c9f9763f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Oct 30 13:16:37.724776 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:37.724743    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fafb3f-cf6wb"]
Oct 30 13:16:37.734235 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:37.734205    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fafb3f-cf6wb"]
Oct 30 13:16:38.462566 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:38.462535    4364 patch_prober.go:29] interesting pod/e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Oct 30 13:16:38.462991 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:38.462591    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm" podUID="9ef21594-2f1f-4b43-9c5f-529c9f9763f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Oct 30 13:16:38.777282 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:38.777214    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84942a-7d58989cc9-j2l2j"
Oct 30 13:16:39.003638 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:39.003031    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17f6dcdc-6416-44e4-8d22-d9bd1681b8a0" path="/var/lib/kubelet/pods/17f6dcdc-6416-44e4-8d22-d9bd1681b8a0/volumes"
Oct 30 13:16:39.462795 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:39.462761    4364 patch_prober.go:29] interesting pod/e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Oct 30 13:16:39.463176 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:39.462822    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm" podUID="9ef21594-2f1f-4b43-9c5f-529c9f9763f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Oct 30 13:16:40.019006 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:40.018971    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-659d72-c87c7d4f5-cn8jk"]
Oct 30 13:16:40.478434 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:40.471782    4364 patch_prober.go:29] interesting pod/e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Oct 30 13:16:40.478434 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:40.471841    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm" podUID="9ef21594-2f1f-4b43-9c5f-529c9f9763f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Oct 30 13:16:41.469526 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:41.468909    4364 patch_prober.go:29] interesting pod/e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Oct 30 13:16:41.469526 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:41.468982    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm" podUID="9ef21594-2f1f-4b43-9c5f-529c9f9763f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Oct 30 13:16:41.517172 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:16:41.517127    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:16:42.463071 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:42.463038    4364 patch_prober.go:29] interesting pod/e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Oct 30 13:16:42.463263 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:42.463100    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm" podUID="9ef21594-2f1f-4b43-9c5f-529c9f9763f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Oct 30 13:16:43.467191 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:43.467158    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm"
Oct 30 13:16:43.467661 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:43.467642    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm"
Oct 30 13:16:43.468356 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:43.468338    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm"
Oct 30 13:16:45.020111 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:45.020041    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-659d72-c87c7d4f5-cn8jk" podUID="b564990b-f890-4e82-a667-c24363ec69d2" containerName="payload-pre-processing" containerID="cri-o://d2cb9b717380728e4b1654b82ef84cc7d757f777c62a49d209332c25b01800f3" gracePeriod=25
Oct 30 13:16:45.493999 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:45.493970    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-659d72-c87c7d4f5-cn8jk"
Oct 30 13:16:45.616133 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:45.616102    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b564990b-f890-4e82-a667-c24363ec69d2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b564990b-f890-4e82-a667-c24363ec69d2-plugins-config-volume\") pod \"b564990b-f890-4e82-a667-c24363ec69d2\" (UID: \"b564990b-f890-4e82-a667-c24363ec69d2\") "
Oct 30 13:16:45.616405 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:45.616382    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b564990b-f890-4e82-a667-c24363ec69d2-kube-api-access-dwdtk\" (UniqueName: \"kubernetes.io/projected/b564990b-f890-4e82-a667-c24363ec69d2-kube-api-access-dwdtk\") pod \"b564990b-f890-4e82-a667-c24363ec69d2\" (UID: \"b564990b-f890-4e82-a667-c24363ec69d2\") "
Oct 30 13:16:45.616470 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:45.616450    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b564990b-f890-4e82-a667-c24363ec69d2-plugins-config-volume" pod "b564990b-f890-4e82-a667-c24363ec69d2" (UID: "b564990b-f890-4e82-a667-c24363ec69d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:16:45.617055 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:45.617037    4364 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b564990b-f890-4e82-a667-c24363ec69d2-plugins-config-volume\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:16:45.618453 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:45.618429    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b564990b-f890-4e82-a667-c24363ec69d2-kube-api-access-dwdtk" pod "b564990b-f890-4e82-a667-c24363ec69d2" (UID: "b564990b-f890-4e82-a667-c24363ec69d2"). InnerVolumeSpecName "kube-api-access-dwdtk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:16:45.718930 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:45.718897    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwdtk\" (UniqueName: \"kubernetes.io/projected/b564990b-f890-4e82-a667-c24363ec69d2-kube-api-access-dwdtk\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:16:46.013820 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:46.013723    4364 generic.go:356] "Generic (PLEG): container finished" podID="b564990b-f890-4e82-a667-c24363ec69d2" containerID="d2cb9b717380728e4b1654b82ef84cc7d757f777c62a49d209332c25b01800f3" exitCode=0
Oct 30 13:16:46.013820 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:46.013782    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-659d72-c87c7d4f5-cn8jk"
Oct 30 13:16:46.014298 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:46.013809    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-659d72-c87c7d4f5-cn8jk" event={"ID":"b564990b-f890-4e82-a667-c24363ec69d2","Type":"ContainerDied","Data":"d2cb9b717380728e4b1654b82ef84cc7d757f777c62a49d209332c25b01800f3"}
Oct 30 13:16:46.014298 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:46.013892    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-659d72-c87c7d4f5-cn8jk" event={"ID":"b564990b-f890-4e82-a667-c24363ec69d2","Type":"ContainerDied","Data":"dcf614d8035c004f60776016ce173e9b7aeecb0048ca758d00e42d63b68422f1"}
Oct 30 13:16:46.014298 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:46.013941    4364 scope.go:122] "RemoveContainer" containerID="d2cb9b717380728e4b1654b82ef84cc7d757f777c62a49d209332c25b01800f3"
Oct 30 13:16:46.043692 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:46.038839    4364 scope.go:122] "RemoveContainer" containerID="d2cb9b717380728e4b1654b82ef84cc7d757f777c62a49d209332c25b01800f3"
Oct 30 13:16:46.044019 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:16:46.043735    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2cb9b717380728e4b1654b82ef84cc7d757f777c62a49d209332c25b01800f3\": container with ID starting with d2cb9b717380728e4b1654b82ef84cc7d757f777c62a49d209332c25b01800f3 not found: ID does not exist" containerID="d2cb9b717380728e4b1654b82ef84cc7d757f777c62a49d209332c25b01800f3"
Oct 30 13:16:46.044019 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:46.043778    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2cb9b717380728e4b1654b82ef84cc7d757f777c62a49d209332c25b01800f3"} err="failed to get container status \"d2cb9b717380728e4b1654b82ef84cc7d757f777c62a49d209332c25b01800f3\": rpc error: code = NotFound desc = could not find container \"d2cb9b717380728e4b1654b82ef84cc7d757f777c62a49d209332c25b01800f3\": container with ID starting with d2cb9b717380728e4b1654b82ef84cc7d757f777c62a49d209332c25b01800f3 not found: ID does not exist"
Oct 30 13:16:46.072895 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:46.072870    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-659d72-c87c7d4f5-cn8jk"]
Oct 30 13:16:46.086975 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:46.086951    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-659d72-c87c7d4f5-cn8jk"]
Oct 30 13:16:46.972561 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:16:46.972529    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b564990b-f890-4e82-a667-c24363ec69d2" path="/var/lib/kubelet/pods/b564990b-f890-4e82-a667-c24363ec69d2/volumes"
Oct 30 13:17:00.517162 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:00.517129    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6b0881-56688d7bdb-jfqwc"]
Oct 30 13:17:00.598937 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:00.598911    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6b0881-cf9d79bc-8vds2"]
Oct 30 13:17:03.314765 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:03.314736    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g6bnr"]
Oct 30 13:17:03.315173 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:03.314921    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g6bnr" podUID="8ea01099-fc92-469c-99cf-875275e18a04" containerName="discovery" containerID="cri-o://89e81246db246657fb88dff1f180d8d242e7f7aac9dd433248e808036c7e0e90" gracePeriod=30
Oct 30 13:17:03.636995 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:03.636973    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g6bnr"
Oct 30 13:17:03.768473 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:03.768438    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ea01099-fc92-469c-99cf-875275e18a04-istio-token\" (UniqueName: \"kubernetes.io/projected/8ea01099-fc92-469c-99cf-875275e18a04-istio-token\") pod \"8ea01099-fc92-469c-99cf-875275e18a04\" (UID: \"8ea01099-fc92-469c-99cf-875275e18a04\") "
Oct 30 13:17:03.768670 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:03.768591    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ea01099-fc92-469c-99cf-875275e18a04-kube-api-access-k7kc4\" (UniqueName: \"kubernetes.io/projected/8ea01099-fc92-469c-99cf-875275e18a04-kube-api-access-k7kc4\") pod \"8ea01099-fc92-469c-99cf-875275e18a04\" (UID: \"8ea01099-fc92-469c-99cf-875275e18a04\") "
Oct 30 13:17:03.768814 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:03.768794    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ea01099-fc92-469c-99cf-875275e18a04-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ea01099-fc92-469c-99cf-875275e18a04-local-certs\") pod \"8ea01099-fc92-469c-99cf-875275e18a04\" (UID: \"8ea01099-fc92-469c-99cf-875275e18a04\") "
Oct 30 13:17:03.768892 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:03.768832    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ea01099-fc92-469c-99cf-875275e18a04-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8ea01099-fc92-469c-99cf-875275e18a04-istio-kubeconfig\") pod \"8ea01099-fc92-469c-99cf-875275e18a04\" (UID: \"8ea01099-fc92-469c-99cf-875275e18a04\") "
Oct 30 13:17:03.768995 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:03.768974    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ea01099-fc92-469c-99cf-875275e18a04-cacerts\" (UniqueName: \"kubernetes.io/secret/8ea01099-fc92-469c-99cf-875275e18a04-cacerts\") pod \"8ea01099-fc92-469c-99cf-875275e18a04\" (UID: \"8ea01099-fc92-469c-99cf-875275e18a04\") "
Oct 30 13:17:03.769218 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:03.769201    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ea01099-fc92-469c-99cf-875275e18a04-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8ea01099-fc92-469c-99cf-875275e18a04-istio-csr-ca-configmap\") pod \"8ea01099-fc92-469c-99cf-875275e18a04\" (UID: \"8ea01099-fc92-469c-99cf-875275e18a04\") "
Oct 30 13:17:03.769798 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:03.769688    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ea01099-fc92-469c-99cf-875275e18a04-istio-csr-ca-configmap" pod "8ea01099-fc92-469c-99cf-875275e18a04" (UID: "8ea01099-fc92-469c-99cf-875275e18a04"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:17:03.769992 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:03.769974    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ea01099-fc92-469c-99cf-875275e18a04-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8ea01099-fc92-469c-99cf-875275e18a04-istio-csr-dns-cert\") pod \"8ea01099-fc92-469c-99cf-875275e18a04\" (UID: \"8ea01099-fc92-469c-99cf-875275e18a04\") "
Oct 30 13:17:03.772128 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:03.771314    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ea01099-fc92-469c-99cf-875275e18a04-istio-kubeconfig" pod "8ea01099-fc92-469c-99cf-875275e18a04" (UID: "8ea01099-fc92-469c-99cf-875275e18a04"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:17:03.772128 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:03.771447    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ea01099-fc92-469c-99cf-875275e18a04-local-certs" pod "8ea01099-fc92-469c-99cf-875275e18a04" (UID: "8ea01099-fc92-469c-99cf-875275e18a04"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:17:03.772128 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:03.771491    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ea01099-fc92-469c-99cf-875275e18a04-kube-api-access-k7kc4" pod "8ea01099-fc92-469c-99cf-875275e18a04" (UID: "8ea01099-fc92-469c-99cf-875275e18a04"). InnerVolumeSpecName "kube-api-access-k7kc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:17:03.772128 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:03.771718    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ea01099-fc92-469c-99cf-875275e18a04-istio-token" pod "8ea01099-fc92-469c-99cf-875275e18a04" (UID: "8ea01099-fc92-469c-99cf-875275e18a04"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:17:03.772128 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:03.771799    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ea01099-fc92-469c-99cf-875275e18a04-cacerts" pod "8ea01099-fc92-469c-99cf-875275e18a04" (UID: "8ea01099-fc92-469c-99cf-875275e18a04"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:17:03.772559 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:03.772539    4364 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8ea01099-fc92-469c-99cf-875275e18a04-istio-csr-ca-configmap\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:17:03.774437 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:03.774413    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ea01099-fc92-469c-99cf-875275e18a04-istio-csr-dns-cert" pod "8ea01099-fc92-469c-99cf-875275e18a04" (UID: "8ea01099-fc92-469c-99cf-875275e18a04"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:17:03.894626 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:03.893764    4364 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ea01099-fc92-469c-99cf-875275e18a04-local-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:17:03.894626 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:03.893801    4364 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8ea01099-fc92-469c-99cf-875275e18a04-istio-kubeconfig\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:17:03.894626 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:03.893814    4364 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8ea01099-fc92-469c-99cf-875275e18a04-cacerts\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:17:03.894626 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:03.893827    4364 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8ea01099-fc92-469c-99cf-875275e18a04-istio-csr-dns-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:17:03.894626 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:03.893840    4364 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ea01099-fc92-469c-99cf-875275e18a04-istio-token\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:17:03.894626 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:03.893856    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7kc4\" (UniqueName: \"kubernetes.io/projected/8ea01099-fc92-469c-99cf-875275e18a04-kube-api-access-k7kc4\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:17:04.285335 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:04.285251    4364 generic.go:356] "Generic (PLEG): container finished" podID="8ea01099-fc92-469c-99cf-875275e18a04" containerID="89e81246db246657fb88dff1f180d8d242e7f7aac9dd433248e808036c7e0e90" exitCode=0
Oct 30 13:17:04.285335 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:04.285295    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g6bnr"
Oct 30 13:17:04.285576 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:04.285340    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g6bnr" event={"ID":"8ea01099-fc92-469c-99cf-875275e18a04","Type":"ContainerDied","Data":"89e81246db246657fb88dff1f180d8d242e7f7aac9dd433248e808036c7e0e90"}
Oct 30 13:17:04.285576 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:04.285402    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g6bnr" event={"ID":"8ea01099-fc92-469c-99cf-875275e18a04","Type":"ContainerDied","Data":"6178e41a64c9015caea6a7836ae453146c1fdad2993bec42707e14fcb425aa62"}
Oct 30 13:17:04.285576 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:04.285429    4364 scope.go:122] "RemoveContainer" containerID="89e81246db246657fb88dff1f180d8d242e7f7aac9dd433248e808036c7e0e90"
Oct 30 13:17:04.298194 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:04.298173    4364 scope.go:122] "RemoveContainer" containerID="89e81246db246657fb88dff1f180d8d242e7f7aac9dd433248e808036c7e0e90"
Oct 30 13:17:04.298483 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:17:04.298465    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89e81246db246657fb88dff1f180d8d242e7f7aac9dd433248e808036c7e0e90\": container with ID starting with 89e81246db246657fb88dff1f180d8d242e7f7aac9dd433248e808036c7e0e90 not found: ID does not exist" containerID="89e81246db246657fb88dff1f180d8d242e7f7aac9dd433248e808036c7e0e90"
Oct 30 13:17:04.298530 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:04.298492    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89e81246db246657fb88dff1f180d8d242e7f7aac9dd433248e808036c7e0e90"} err="failed to get container status \"89e81246db246657fb88dff1f180d8d242e7f7aac9dd433248e808036c7e0e90\": rpc error: code = NotFound desc = could not find container \"89e81246db246657fb88dff1f180d8d242e7f7aac9dd433248e808036c7e0e90\": container with ID starting with 89e81246db246657fb88dff1f180d8d242e7f7aac9dd433248e808036c7e0e90 not found: ID does not exist"
Oct 30 13:17:04.390649 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:04.390628    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g6bnr"]
Oct 30 13:17:04.407669 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:04.402781    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g6bnr"]
Oct 30 13:17:04.973873 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:04.973843    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ea01099-fc92-469c-99cf-875275e18a04" path="/var/lib/kubelet/pods/8ea01099-fc92-469c-99cf-875275e18a04/volumes"
Oct 30 13:17:05.518072 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:05.518036    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-6b0881-56688d7bdb-jfqwc" podUID="afeac9f9-fa22-47b0-8446-9683377383e2" containerName="payload-processing" containerID="cri-o://0198b8b567b08ac6aa51fbf71bf019ef7a3302f5d980165813e01bf721cfb72a" gracePeriod=25
Oct 30 13:17:05.599306 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:05.599271    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6b0881-cf9d79bc-8vds2" podUID="bbbe037e-4f22-4ecd-9d73-d8d32419647f" containerName="payload-pre-processing" containerID="cri-o://15c4f44d30b26687802ef40c690aa7ba191cb8f8a7da687340e610adbc24434c" gracePeriod=25
Oct 30 13:17:05.895671 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:05.895644    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6b0881-56688d7bdb-jfqwc"
Oct 30 13:17:06.013882 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:06.013852    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6b0881-cf9d79bc-8vds2"
Oct 30 13:17:06.026691 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:06.021460    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afeac9f9-fa22-47b0-8446-9683377383e2-kube-api-access-r9w4f\" (UniqueName: \"kubernetes.io/projected/afeac9f9-fa22-47b0-8446-9683377383e2-kube-api-access-r9w4f\") pod \"afeac9f9-fa22-47b0-8446-9683377383e2\" (UID: \"afeac9f9-fa22-47b0-8446-9683377383e2\") "
Oct 30 13:17:06.026691 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:06.021586    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afeac9f9-fa22-47b0-8446-9683377383e2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afeac9f9-fa22-47b0-8446-9683377383e2-plugins-config-volume\") pod \"afeac9f9-fa22-47b0-8446-9683377383e2\" (UID: \"afeac9f9-fa22-47b0-8446-9683377383e2\") "
Oct 30 13:17:06.026691 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:06.025944    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afeac9f9-fa22-47b0-8446-9683377383e2-plugins-config-volume" pod "afeac9f9-fa22-47b0-8446-9683377383e2" (UID: "afeac9f9-fa22-47b0-8446-9683377383e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:17:06.035141 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:06.035111    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afeac9f9-fa22-47b0-8446-9683377383e2-kube-api-access-r9w4f" pod "afeac9f9-fa22-47b0-8446-9683377383e2" (UID: "afeac9f9-fa22-47b0-8446-9683377383e2"). InnerVolumeSpecName "kube-api-access-r9w4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:17:06.051360 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:06.051061    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9w4f\" (UniqueName: \"kubernetes.io/projected/afeac9f9-fa22-47b0-8446-9683377383e2-kube-api-access-r9w4f\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:17:06.051360 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:06.051095    4364 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afeac9f9-fa22-47b0-8446-9683377383e2-plugins-config-volume\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:17:06.171739 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:06.168164    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbbe037e-4f22-4ecd-9d73-d8d32419647f-kube-api-access-fm6gt\" (UniqueName: \"kubernetes.io/projected/bbbe037e-4f22-4ecd-9d73-d8d32419647f-kube-api-access-fm6gt\") pod \"bbbe037e-4f22-4ecd-9d73-d8d32419647f\" (UID: \"bbbe037e-4f22-4ecd-9d73-d8d32419647f\") "
Oct 30 13:17:06.171739 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:06.168299    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbbe037e-4f22-4ecd-9d73-d8d32419647f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbbe037e-4f22-4ecd-9d73-d8d32419647f-plugins-config-volume\") pod \"bbbe037e-4f22-4ecd-9d73-d8d32419647f\" (UID: \"bbbe037e-4f22-4ecd-9d73-d8d32419647f\") "
Oct 30 13:17:06.196620 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:06.192038    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbbe037e-4f22-4ecd-9d73-d8d32419647f-kube-api-access-fm6gt" pod "bbbe037e-4f22-4ecd-9d73-d8d32419647f" (UID: "bbbe037e-4f22-4ecd-9d73-d8d32419647f"). InnerVolumeSpecName "kube-api-access-fm6gt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:17:06.199175 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:06.198990    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbbe037e-4f22-4ecd-9d73-d8d32419647f-plugins-config-volume" pod "bbbe037e-4f22-4ecd-9d73-d8d32419647f" (UID: "bbbe037e-4f22-4ecd-9d73-d8d32419647f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:17:06.277230 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:06.277184    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fm6gt\" (UniqueName: \"kubernetes.io/projected/bbbe037e-4f22-4ecd-9d73-d8d32419647f-kube-api-access-fm6gt\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:17:06.277230 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:06.277216    4364 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbbe037e-4f22-4ecd-9d73-d8d32419647f-plugins-config-volume\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:17:06.310593 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:06.310557    4364 generic.go:356] "Generic (PLEG): container finished" podID="bbbe037e-4f22-4ecd-9d73-d8d32419647f" containerID="15c4f44d30b26687802ef40c690aa7ba191cb8f8a7da687340e610adbc24434c" exitCode=0
Oct 30 13:17:06.310793 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:06.310687    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6b0881-cf9d79bc-8vds2"
Oct 30 13:17:06.310793 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:06.310714    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6b0881-cf9d79bc-8vds2" event={"ID":"bbbe037e-4f22-4ecd-9d73-d8d32419647f","Type":"ContainerDied","Data":"15c4f44d30b26687802ef40c690aa7ba191cb8f8a7da687340e610adbc24434c"}
Oct 30 13:17:06.310793 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:06.310768    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6b0881-cf9d79bc-8vds2" event={"ID":"bbbe037e-4f22-4ecd-9d73-d8d32419647f","Type":"ContainerDied","Data":"bc26edc017eaf98e2fba62715e250cbbd1d5bd18d1c885d4c8c56acc767d9649"}
Oct 30 13:17:06.310793 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:06.310786    4364 scope.go:122] "RemoveContainer" containerID="15c4f44d30b26687802ef40c690aa7ba191cb8f8a7da687340e610adbc24434c"
Oct 30 13:17:06.312314 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:06.312293    4364 generic.go:356] "Generic (PLEG): container finished" podID="afeac9f9-fa22-47b0-8446-9683377383e2" containerID="0198b8b567b08ac6aa51fbf71bf019ef7a3302f5d980165813e01bf721cfb72a" exitCode=0
Oct 30 13:17:06.312434 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:06.312418    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6b0881-56688d7bdb-jfqwc" event={"ID":"afeac9f9-fa22-47b0-8446-9683377383e2","Type":"ContainerDied","Data":"0198b8b567b08ac6aa51fbf71bf019ef7a3302f5d980165813e01bf721cfb72a"}
Oct 30 13:17:06.312509 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:06.312447    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6b0881-56688d7bdb-jfqwc" event={"ID":"afeac9f9-fa22-47b0-8446-9683377383e2","Type":"ContainerDied","Data":"ba14d550484e8b97665c073ab3665a2b7a2e66852afa0fd38c33a99823183bcf"}
Oct 30 13:17:06.312509 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:06.312498    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6b0881-56688d7bdb-jfqwc"
Oct 30 13:17:06.323219 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:06.323198    4364 scope.go:122] "RemoveContainer" containerID="15c4f44d30b26687802ef40c690aa7ba191cb8f8a7da687340e610adbc24434c"
Oct 30 13:17:06.323510 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:17:06.323488    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15c4f44d30b26687802ef40c690aa7ba191cb8f8a7da687340e610adbc24434c\": container with ID starting with 15c4f44d30b26687802ef40c690aa7ba191cb8f8a7da687340e610adbc24434c not found: ID does not exist" containerID="15c4f44d30b26687802ef40c690aa7ba191cb8f8a7da687340e610adbc24434c"
Oct 30 13:17:06.323593 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:06.323520    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15c4f44d30b26687802ef40c690aa7ba191cb8f8a7da687340e610adbc24434c"} err="failed to get container status \"15c4f44d30b26687802ef40c690aa7ba191cb8f8a7da687340e610adbc24434c\": rpc error: code = NotFound desc = could not find container \"15c4f44d30b26687802ef40c690aa7ba191cb8f8a7da687340e610adbc24434c\": container with ID starting with 15c4f44d30b26687802ef40c690aa7ba191cb8f8a7da687340e610adbc24434c not found: ID does not exist"
Oct 30 13:17:06.323593 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:06.323543    4364 scope.go:122] "RemoveContainer" containerID="0198b8b567b08ac6aa51fbf71bf019ef7a3302f5d980165813e01bf721cfb72a"
Oct 30 13:17:06.337853 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:06.337833    4364 scope.go:122] "RemoveContainer" containerID="0198b8b567b08ac6aa51fbf71bf019ef7a3302f5d980165813e01bf721cfb72a"
Oct 30 13:17:06.338129 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:17:06.338107    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0198b8b567b08ac6aa51fbf71bf019ef7a3302f5d980165813e01bf721cfb72a\": container with ID starting with 0198b8b567b08ac6aa51fbf71bf019ef7a3302f5d980165813e01bf721cfb72a not found: ID does not exist" containerID="0198b8b567b08ac6aa51fbf71bf019ef7a3302f5d980165813e01bf721cfb72a"
Oct 30 13:17:06.338224 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:06.338133    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0198b8b567b08ac6aa51fbf71bf019ef7a3302f5d980165813e01bf721cfb72a"} err="failed to get container status \"0198b8b567b08ac6aa51fbf71bf019ef7a3302f5d980165813e01bf721cfb72a\": rpc error: code = NotFound desc = could not find container \"0198b8b567b08ac6aa51fbf71bf019ef7a3302f5d980165813e01bf721cfb72a\": container with ID starting with 0198b8b567b08ac6aa51fbf71bf019ef7a3302f5d980165813e01bf721cfb72a not found: ID does not exist"
Oct 30 13:17:06.378030 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:06.377992    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6b0881-cf9d79bc-8vds2"]
Oct 30 13:17:06.383006 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:06.382985    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6b0881-cf9d79bc-8vds2"]
Oct 30 13:17:06.433715 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:06.433666    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6b0881-56688d7bdb-jfqwc"]
Oct 30 13:17:06.459507 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:06.459484    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6b0881-56688d7bdb-jfqwc"]
Oct 30 13:17:06.972098 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:06.972070    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afeac9f9-fa22-47b0-8446-9683377383e2" path="/var/lib/kubelet/pods/afeac9f9-fa22-47b0-8446-9683377383e2/volumes"
Oct 30 13:17:06.972526 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:06.972510    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbbe037e-4f22-4ecd-9d73-d8d32419647f" path="/var/lib/kubelet/pods/bbbe037e-4f22-4ecd-9d73-d8d32419647f/volumes"
Oct 30 13:17:11.529632 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:17:11.529568    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:17:18.067761 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:18.067715    4364 patch_prober.go:29] interesting pod/e2e-api-b-84942a-openshift-default-58974cd687-wtjlv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:17:18.068239 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:18.067807    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-84942a-openshift-default-58974cd687-wtjlv" podUID="900b00e0-ab94-4293-ade2-7e89bd313fea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:17:18.654137 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:18.654023    4364 generic.go:356] "Generic (PLEG): container finished" podID="900b00e0-ab94-4293-ade2-7e89bd313fea" containerID="d9ce283f4848e306aa5e2b66efbbc4b69e655759246703e2c231008f4c27143f" exitCode=0
Oct 30 13:17:18.654873 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:18.654814    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-84942a-openshift-default-58974cd687-wtjlv" event={"ID":"900b00e0-ab94-4293-ade2-7e89bd313fea","Type":"ContainerDied","Data":"d9ce283f4848e306aa5e2b66efbbc4b69e655759246703e2c231008f4c27143f"}
Oct 30 13:17:18.661656 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:18.655802    4364 scope.go:122] "RemoveContainer" containerID="d9ce283f4848e306aa5e2b66efbbc4b69e655759246703e2c231008f4c27143f"
Oct 30 13:17:18.664841 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:18.664789    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:17:18.664980 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:18.664902    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:17:18.664980 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:18.664959    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:17:19.665104 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:19.665068    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-84942a-openshift-default-58974cd687-wtjlv" event={"ID":"900b00e0-ab94-4293-ade2-7e89bd313fea","Type":"ContainerStarted","Data":"7f8d4538497a7934ffdd59df1c4840ac99336ca49c873227643a5e716acd7885"}
Oct 30 13:17:20.513937 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:20.513904    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-84942a-openshift-default-58974cd687-wtjlv"
Oct 30 13:17:20.514092 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:20.514054    4364 patch_prober.go:29] interesting pod/e2e-api-b-84942a-openshift-default-58974cd687-wtjlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Oct 30 13:17:20.514155 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:20.514104    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-84942a-openshift-default-58974cd687-wtjlv" podUID="900b00e0-ab94-4293-ade2-7e89bd313fea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Oct 30 13:17:21.514318 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:21.514284    4364 patch_prober.go:29] interesting pod/e2e-api-b-84942a-openshift-default-58974cd687-wtjlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Oct 30 13:17:21.514749 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:21.514338    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-84942a-openshift-default-58974cd687-wtjlv" podUID="900b00e0-ab94-4293-ade2-7e89bd313fea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Oct 30 13:17:22.514544 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:22.514512    4364 patch_prober.go:29] interesting pod/e2e-api-b-84942a-openshift-default-58974cd687-wtjlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Oct 30 13:17:22.514959 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:22.514562    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-84942a-openshift-default-58974cd687-wtjlv" podUID="900b00e0-ab94-4293-ade2-7e89bd313fea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Oct 30 13:17:23.514542 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:23.514511    4364 patch_prober.go:29] interesting pod/e2e-api-b-84942a-openshift-default-58974cd687-wtjlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Oct 30 13:17:23.514730 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:23.514560    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-84942a-openshift-default-58974cd687-wtjlv" podUID="900b00e0-ab94-4293-ade2-7e89bd313fea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Oct 30 13:17:24.514192 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:24.514160    4364 patch_prober.go:29] interesting pod/e2e-api-b-84942a-openshift-default-58974cd687-wtjlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Oct 30 13:17:24.514367 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:24.514216    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-84942a-openshift-default-58974cd687-wtjlv" podUID="900b00e0-ab94-4293-ade2-7e89bd313fea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Oct 30 13:17:25.518569 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:25.518541    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-84942a-openshift-default-58974cd687-wtjlv"
Oct 30 13:17:25.519003 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:25.518986    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-84942a-openshift-default-58974cd687-wtjlv"
Oct 30 13:17:25.519695 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:25.519675    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-84942a-openshift-default-58974cd687-wtjlv"
Oct 30 13:17:33.680040 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:33.680010    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9fcb69-59fd4c8c56-w96vh"]
Oct 30 13:17:38.680704 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:38.680666    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-9fcb69-59fd4c8c56-w96vh" podUID="79f77504-1067-40e5-9ea6-1f7b752d0f8d" containerName="payload-processing" containerID="cri-o://7308e7164d8e5aba0eb93f581c5c1036dee1df1c8a3fe89418923ada8e068674" gracePeriod=25
Oct 30 13:17:39.055024 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:39.054996    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9fcb69-59fd4c8c56-w96vh"
Oct 30 13:17:39.127755 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:39.127725    4364 generic.go:356] "Generic (PLEG): container finished" podID="79f77504-1067-40e5-9ea6-1f7b752d0f8d" containerID="7308e7164d8e5aba0eb93f581c5c1036dee1df1c8a3fe89418923ada8e068674" exitCode=0
Oct 30 13:17:39.127928 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:39.127782    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9fcb69-59fd4c8c56-w96vh"
Oct 30 13:17:39.127928 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:39.127819    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9fcb69-59fd4c8c56-w96vh" event={"ID":"79f77504-1067-40e5-9ea6-1f7b752d0f8d","Type":"ContainerDied","Data":"7308e7164d8e5aba0eb93f581c5c1036dee1df1c8a3fe89418923ada8e068674"}
Oct 30 13:17:39.127928 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:39.127895    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9fcb69-59fd4c8c56-w96vh" event={"ID":"79f77504-1067-40e5-9ea6-1f7b752d0f8d","Type":"ContainerDied","Data":"03fc8df8ac6da358ba0fe7da32afe79ef5ba67a15b7e63a789afd1e9337a29ee"}
Oct 30 13:17:39.127928 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:39.127929    4364 scope.go:122] "RemoveContainer" containerID="7308e7164d8e5aba0eb93f581c5c1036dee1df1c8a3fe89418923ada8e068674"
Oct 30 13:17:39.142866 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:39.142849    4364 scope.go:122] "RemoveContainer" containerID="7308e7164d8e5aba0eb93f581c5c1036dee1df1c8a3fe89418923ada8e068674"
Oct 30 13:17:39.143125 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:17:39.143107    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7308e7164d8e5aba0eb93f581c5c1036dee1df1c8a3fe89418923ada8e068674\": container with ID starting with 7308e7164d8e5aba0eb93f581c5c1036dee1df1c8a3fe89418923ada8e068674 not found: ID does not exist" containerID="7308e7164d8e5aba0eb93f581c5c1036dee1df1c8a3fe89418923ada8e068674"
Oct 30 13:17:39.143193 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:39.143132    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7308e7164d8e5aba0eb93f581c5c1036dee1df1c8a3fe89418923ada8e068674"} err="failed to get container status \"7308e7164d8e5aba0eb93f581c5c1036dee1df1c8a3fe89418923ada8e068674\": rpc error: code = NotFound desc = could not find container \"7308e7164d8e5aba0eb93f581c5c1036dee1df1c8a3fe89418923ada8e068674\": container with ID starting with 7308e7164d8e5aba0eb93f581c5c1036dee1df1c8a3fe89418923ada8e068674 not found: ID does not exist"
Oct 30 13:17:39.145826 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:39.145808    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79f77504-1067-40e5-9ea6-1f7b752d0f8d-kube-api-access-f6zx7\" (UniqueName: \"kubernetes.io/projected/79f77504-1067-40e5-9ea6-1f7b752d0f8d-kube-api-access-f6zx7\") pod \"79f77504-1067-40e5-9ea6-1f7b752d0f8d\" (UID: \"79f77504-1067-40e5-9ea6-1f7b752d0f8d\") "
Oct 30 13:17:39.145934 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:39.145922    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79f77504-1067-40e5-9ea6-1f7b752d0f8d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79f77504-1067-40e5-9ea6-1f7b752d0f8d-plugins-config-volume\") pod \"79f77504-1067-40e5-9ea6-1f7b752d0f8d\" (UID: \"79f77504-1067-40e5-9ea6-1f7b752d0f8d\") "
Oct 30 13:17:39.146266 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:39.146232    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79f77504-1067-40e5-9ea6-1f7b752d0f8d-plugins-config-volume" pod "79f77504-1067-40e5-9ea6-1f7b752d0f8d" (UID: "79f77504-1067-40e5-9ea6-1f7b752d0f8d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:17:39.146537 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:39.146519    4364 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79f77504-1067-40e5-9ea6-1f7b752d0f8d-plugins-config-volume\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:17:39.147764 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:39.147744    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79f77504-1067-40e5-9ea6-1f7b752d0f8d-kube-api-access-f6zx7" pod "79f77504-1067-40e5-9ea6-1f7b752d0f8d" (UID: "79f77504-1067-40e5-9ea6-1f7b752d0f8d"). InnerVolumeSpecName "kube-api-access-f6zx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:17:39.247991 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:39.247933    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6zx7\" (UniqueName: \"kubernetes.io/projected/79f77504-1067-40e5-9ea6-1f7b752d0f8d-kube-api-access-f6zx7\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:17:39.490416 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:39.490381    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9fcb69-59fd4c8c56-w96vh"]
Oct 30 13:17:39.495277 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:39.495251    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9fcb69-59fd4c8c56-w96vh"]
Oct 30 13:17:40.973806 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:40.973776    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79f77504-1067-40e5-9ea6-1f7b752d0f8d" path="/var/lib/kubelet/pods/79f77504-1067-40e5-9ea6-1f7b752d0f8d/volumes"
Oct 30 13:17:41.511862 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:17:41.511817    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:17:45.675626 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:45.655793    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:17:50.050084 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:50.050049    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8af0fe-5cf9cd699-gzxlc"]
Oct 30 13:17:51.808166 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:51.808128    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-659d72-8glwc"]
Oct 30 13:17:51.819362 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:51.819335    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-659d72-8glwc"]
Oct 30 13:17:52.972381 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:52.972350    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f10e7b19-6e46-4e00-b631-6bc0a94b98a7" path="/var/lib/kubelet/pods/f10e7b19-6e46-4e00-b631-6bc0a94b98a7/volumes"
Oct 30 13:17:53.735448 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:53.735410    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-df6c3661-7f764d9794-cq64m"]
Oct 30 13:17:53.795030 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:53.794997    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-df6c3661-5866c5b5c5-qfrtz"]
Oct 30 13:17:55.050820 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:55.050763    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8af0fe-5cf9cd699-gzxlc" podUID="3fb63d98-c5f7-49fd-9ba0-b58f7329bd63" containerName="payload-pre-processing" containerID="cri-o://3d3ef208745d7571d97084e026c4249d2c86001cc59fbb52329887a707d59ab0" gracePeriod=25
Oct 30 13:17:55.349642 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:55.349577    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8af0fe-5cf9cd699-gzxlc"
Oct 30 13:17:55.430005 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:55.429966    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fb63d98-c5f7-49fd-9ba0-b58f7329bd63-kube-api-access-wvcn5\" (UniqueName: \"kubernetes.io/projected/3fb63d98-c5f7-49fd-9ba0-b58f7329bd63-kube-api-access-wvcn5\") pod \"3fb63d98-c5f7-49fd-9ba0-b58f7329bd63\" (UID: \"3fb63d98-c5f7-49fd-9ba0-b58f7329bd63\") "
Oct 30 13:17:55.430185 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:55.430056    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fb63d98-c5f7-49fd-9ba0-b58f7329bd63-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fb63d98-c5f7-49fd-9ba0-b58f7329bd63-plugins-config-volume\") pod \"3fb63d98-c5f7-49fd-9ba0-b58f7329bd63\" (UID: \"3fb63d98-c5f7-49fd-9ba0-b58f7329bd63\") "
Oct 30 13:17:55.430493 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:55.430467    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fb63d98-c5f7-49fd-9ba0-b58f7329bd63-plugins-config-volume" pod "3fb63d98-c5f7-49fd-9ba0-b58f7329bd63" (UID: "3fb63d98-c5f7-49fd-9ba0-b58f7329bd63"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:17:55.430739 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:55.430721    4364 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fb63d98-c5f7-49fd-9ba0-b58f7329bd63-plugins-config-volume\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:17:55.432109 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:55.432084    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fb63d98-c5f7-49fd-9ba0-b58f7329bd63-kube-api-access-wvcn5" pod "3fb63d98-c5f7-49fd-9ba0-b58f7329bd63" (UID: "3fb63d98-c5f7-49fd-9ba0-b58f7329bd63"). InnerVolumeSpecName "kube-api-access-wvcn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:17:55.450862 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:55.450761    4364 generic.go:356] "Generic (PLEG): container finished" podID="3fb63d98-c5f7-49fd-9ba0-b58f7329bd63" containerID="3d3ef208745d7571d97084e026c4249d2c86001cc59fbb52329887a707d59ab0" exitCode=0
Oct 30 13:17:55.450973 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:55.450945    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8af0fe-5cf9cd699-gzxlc"
Oct 30 13:17:55.452166 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:55.451302    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8af0fe-5cf9cd699-gzxlc" event={"ID":"3fb63d98-c5f7-49fd-9ba0-b58f7329bd63","Type":"ContainerDied","Data":"3d3ef208745d7571d97084e026c4249d2c86001cc59fbb52329887a707d59ab0"}
Oct 30 13:17:55.452166 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:55.451365    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8af0fe-5cf9cd699-gzxlc" event={"ID":"3fb63d98-c5f7-49fd-9ba0-b58f7329bd63","Type":"ContainerDied","Data":"114ba6e8da54a06506b441a923381b0019cafa19f0991b452c549ab99aabfb49"}
Oct 30 13:17:55.452166 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:55.451395    4364 scope.go:122] "RemoveContainer" containerID="3d3ef208745d7571d97084e026c4249d2c86001cc59fbb52329887a707d59ab0"
Oct 30 13:17:55.465460 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:55.465436    4364 scope.go:122] "RemoveContainer" containerID="3d3ef208745d7571d97084e026c4249d2c86001cc59fbb52329887a707d59ab0"
Oct 30 13:17:55.465776 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:17:55.465754    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d3ef208745d7571d97084e026c4249d2c86001cc59fbb52329887a707d59ab0\": container with ID starting with 3d3ef208745d7571d97084e026c4249d2c86001cc59fbb52329887a707d59ab0 not found: ID does not exist" containerID="3d3ef208745d7571d97084e026c4249d2c86001cc59fbb52329887a707d59ab0"
Oct 30 13:17:55.465878 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:55.465786    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d3ef208745d7571d97084e026c4249d2c86001cc59fbb52329887a707d59ab0"} err="failed to get container status \"3d3ef208745d7571d97084e026c4249d2c86001cc59fbb52329887a707d59ab0\": rpc error: code = NotFound desc = could not find container \"3d3ef208745d7571d97084e026c4249d2c86001cc59fbb52329887a707d59ab0\": container with ID starting with 3d3ef208745d7571d97084e026c4249d2c86001cc59fbb52329887a707d59ab0 not found: ID does not exist"
Oct 30 13:17:55.511171 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:55.511147    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8af0fe-5cf9cd699-gzxlc"]
Oct 30 13:17:55.516009 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:55.515989    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8af0fe-5cf9cd699-gzxlc"]
Oct 30 13:17:55.532708 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:55.532682    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvcn5\" (UniqueName: \"kubernetes.io/projected/3fb63d98-c5f7-49fd-9ba0-b58f7329bd63-kube-api-access-wvcn5\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:17:55.678909 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:55.678879    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq"]
Oct 30 13:17:55.679100 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:55.679077    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq" podUID="d4826806-75a8-47d6-866b-415116519d9d" containerName="istio-proxy" containerID="cri-o://1b14ca834bdf6d9785e98377ab7a04e200530a0de03e2c2a2622fc357ba80a6d" gracePeriod=30
Oct 30 13:17:56.972844 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:56.972810    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fb63d98-c5f7-49fd-9ba0-b58f7329bd63" path="/var/lib/kubelet/pods/3fb63d98-c5f7-49fd-9ba0-b58f7329bd63/volumes"
Oct 30 13:17:58.736014 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:58.735975    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-df6c3661-7f764d9794-cq64m" podUID="d5298fc7-1369-4e44-8ee3-f058d1e20c37" containerName="payload-processing" containerID="cri-o://f247962a8f3f262f84177b9495a9bb7231e51ff3c47580a505605210a1c52783" gracePeriod=25
Oct 30 13:17:58.796057 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:58.796026    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-df6c3661-5866c5b5c5-qfrtz" podUID="42c43460-b498-432d-a8e5-a68d5046aba6" containerName="payload-pre-processing" containerID="cri-o://86ea448ef0edc8f9d9e4529d7ee60513aa4b4acf04b74f17fa97c9f02304372d" gracePeriod=25
Oct 30 13:17:59.180900 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:59.180877    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-df6c3661-7f764d9794-cq64m"
Oct 30 13:17:59.184810 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:59.184788    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-df6c3661-5866c5b5c5-qfrtz"
Oct 30 13:17:59.327495 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:59.327409    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5298fc7-1369-4e44-8ee3-f058d1e20c37-kube-api-access-z9jgk\" (UniqueName: \"kubernetes.io/projected/d5298fc7-1369-4e44-8ee3-f058d1e20c37-kube-api-access-z9jgk\") pod \"d5298fc7-1369-4e44-8ee3-f058d1e20c37\" (UID: \"d5298fc7-1369-4e44-8ee3-f058d1e20c37\") "
Oct 30 13:17:59.327495 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:59.327458    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5298fc7-1369-4e44-8ee3-f058d1e20c37-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5298fc7-1369-4e44-8ee3-f058d1e20c37-plugins-config-volume\") pod \"d5298fc7-1369-4e44-8ee3-f058d1e20c37\" (UID: \"d5298fc7-1369-4e44-8ee3-f058d1e20c37\") "
Oct 30 13:17:59.327821 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:59.327497    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42c43460-b498-432d-a8e5-a68d5046aba6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42c43460-b498-432d-a8e5-a68d5046aba6-plugins-config-volume\") pod \"42c43460-b498-432d-a8e5-a68d5046aba6\" (UID: \"42c43460-b498-432d-a8e5-a68d5046aba6\") "
Oct 30 13:17:59.327821 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:59.327627    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42c43460-b498-432d-a8e5-a68d5046aba6-kube-api-access-5w5ld\" (UniqueName: \"kubernetes.io/projected/42c43460-b498-432d-a8e5-a68d5046aba6-kube-api-access-5w5ld\") pod \"42c43460-b498-432d-a8e5-a68d5046aba6\" (UID: \"42c43460-b498-432d-a8e5-a68d5046aba6\") "
Oct 30 13:17:59.327976 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:59.327947    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42c43460-b498-432d-a8e5-a68d5046aba6-plugins-config-volume" pod "42c43460-b498-432d-a8e5-a68d5046aba6" (UID: "42c43460-b498-432d-a8e5-a68d5046aba6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:17:59.328441 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:59.328414    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5298fc7-1369-4e44-8ee3-f058d1e20c37-plugins-config-volume" pod "d5298fc7-1369-4e44-8ee3-f058d1e20c37" (UID: "d5298fc7-1369-4e44-8ee3-f058d1e20c37"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:17:59.328441 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:59.328435    4364 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42c43460-b498-432d-a8e5-a68d5046aba6-plugins-config-volume\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:17:59.329859 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:59.329823    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5298fc7-1369-4e44-8ee3-f058d1e20c37-kube-api-access-z9jgk" pod "d5298fc7-1369-4e44-8ee3-f058d1e20c37" (UID: "d5298fc7-1369-4e44-8ee3-f058d1e20c37"). InnerVolumeSpecName "kube-api-access-z9jgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:17:59.330781 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:59.330759    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42c43460-b498-432d-a8e5-a68d5046aba6-kube-api-access-5w5ld" pod "42c43460-b498-432d-a8e5-a68d5046aba6" (UID: "42c43460-b498-432d-a8e5-a68d5046aba6"). InnerVolumeSpecName "kube-api-access-5w5ld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:17:59.430451 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:59.430421    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9jgk\" (UniqueName: \"kubernetes.io/projected/d5298fc7-1369-4e44-8ee3-f058d1e20c37-kube-api-access-z9jgk\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:17:59.430451 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:59.430447    4364 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5298fc7-1369-4e44-8ee3-f058d1e20c37-plugins-config-volume\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:17:59.430451 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:59.430456    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w5ld\" (UniqueName: \"kubernetes.io/projected/42c43460-b498-432d-a8e5-a68d5046aba6-kube-api-access-5w5ld\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:17:59.489815 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:59.489785    4364 generic.go:356] "Generic (PLEG): container finished" podID="42c43460-b498-432d-a8e5-a68d5046aba6" containerID="86ea448ef0edc8f9d9e4529d7ee60513aa4b4acf04b74f17fa97c9f02304372d" exitCode=0
Oct 30 13:17:59.489972 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:59.489829    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-df6c3661-5866c5b5c5-qfrtz"
Oct 30 13:17:59.489972 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:59.489862    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-df6c3661-5866c5b5c5-qfrtz" event={"ID":"42c43460-b498-432d-a8e5-a68d5046aba6","Type":"ContainerDied","Data":"86ea448ef0edc8f9d9e4529d7ee60513aa4b4acf04b74f17fa97c9f02304372d"}
Oct 30 13:17:59.489972 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:59.489962    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-df6c3661-5866c5b5c5-qfrtz" event={"ID":"42c43460-b498-432d-a8e5-a68d5046aba6","Type":"ContainerDied","Data":"28c1f443ead0042c785f8ff93665b7f118507c4e6caeab675e252e417dd34798"}
Oct 30 13:17:59.490138 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:59.489992    4364 scope.go:122] "RemoveContainer" containerID="86ea448ef0edc8f9d9e4529d7ee60513aa4b4acf04b74f17fa97c9f02304372d"
Oct 30 13:17:59.491575 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:59.491545    4364 generic.go:356] "Generic (PLEG): container finished" podID="d5298fc7-1369-4e44-8ee3-f058d1e20c37" containerID="f247962a8f3f262f84177b9495a9bb7231e51ff3c47580a505605210a1c52783" exitCode=0
Oct 30 13:17:59.491740 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:59.491584    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-df6c3661-7f764d9794-cq64m" event={"ID":"d5298fc7-1369-4e44-8ee3-f058d1e20c37","Type":"ContainerDied","Data":"f247962a8f3f262f84177b9495a9bb7231e51ff3c47580a505605210a1c52783"}
Oct 30 13:17:59.491740 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:59.491675    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-df6c3661-7f764d9794-cq64m" event={"ID":"d5298fc7-1369-4e44-8ee3-f058d1e20c37","Type":"ContainerDied","Data":"9acf3b8a4609ee32265b50181b36dd682185c88a8e2fe0bd0c51ec18b9e4cb47"}
Oct 30 13:17:59.491740 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:59.491611    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-df6c3661-7f764d9794-cq64m"
Oct 30 13:17:59.502591 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:59.502570    4364 scope.go:122] "RemoveContainer" containerID="86ea448ef0edc8f9d9e4529d7ee60513aa4b4acf04b74f17fa97c9f02304372d"
Oct 30 13:17:59.502935 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:17:59.502909    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86ea448ef0edc8f9d9e4529d7ee60513aa4b4acf04b74f17fa97c9f02304372d\": container with ID starting with 86ea448ef0edc8f9d9e4529d7ee60513aa4b4acf04b74f17fa97c9f02304372d not found: ID does not exist" containerID="86ea448ef0edc8f9d9e4529d7ee60513aa4b4acf04b74f17fa97c9f02304372d"
Oct 30 13:17:59.503023 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:59.502939    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86ea448ef0edc8f9d9e4529d7ee60513aa4b4acf04b74f17fa97c9f02304372d"} err="failed to get container status \"86ea448ef0edc8f9d9e4529d7ee60513aa4b4acf04b74f17fa97c9f02304372d\": rpc error: code = NotFound desc = could not find container \"86ea448ef0edc8f9d9e4529d7ee60513aa4b4acf04b74f17fa97c9f02304372d\": container with ID starting with 86ea448ef0edc8f9d9e4529d7ee60513aa4b4acf04b74f17fa97c9f02304372d not found: ID does not exist"
Oct 30 13:17:59.503023 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:59.502955    4364 scope.go:122] "RemoveContainer" containerID="f247962a8f3f262f84177b9495a9bb7231e51ff3c47580a505605210a1c52783"
Oct 30 13:17:59.512794 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:59.512775    4364 scope.go:122] "RemoveContainer" containerID="f247962a8f3f262f84177b9495a9bb7231e51ff3c47580a505605210a1c52783"
Oct 30 13:17:59.513061 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:17:59.513032    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f247962a8f3f262f84177b9495a9bb7231e51ff3c47580a505605210a1c52783\": container with ID starting with f247962a8f3f262f84177b9495a9bb7231e51ff3c47580a505605210a1c52783 not found: ID does not exist" containerID="f247962a8f3f262f84177b9495a9bb7231e51ff3c47580a505605210a1c52783"
Oct 30 13:17:59.513114 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:59.513060    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f247962a8f3f262f84177b9495a9bb7231e51ff3c47580a505605210a1c52783"} err="failed to get container status \"f247962a8f3f262f84177b9495a9bb7231e51ff3c47580a505605210a1c52783\": rpc error: code = NotFound desc = could not find container \"f247962a8f3f262f84177b9495a9bb7231e51ff3c47580a505605210a1c52783\": container with ID starting with f247962a8f3f262f84177b9495a9bb7231e51ff3c47580a505605210a1c52783 not found: ID does not exist"
Oct 30 13:17:59.573842 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:59.573817    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-df6c3661-5866c5b5c5-qfrtz"]
Oct 30 13:17:59.676806 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:59.676768    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-df6c3661-5866c5b5c5-qfrtz"]
Oct 30 13:17:59.895232 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:59.864663    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-df6c3661-7f764d9794-cq64m"]
Oct 30 13:17:59.972842 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:17:59.972098    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-df6c3661-7f764d9794-cq64m"]
Oct 30 13:18:00.975082 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:00.975055    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42c43460-b498-432d-a8e5-a68d5046aba6" path="/var/lib/kubelet/pods/42c43460-b498-432d-a8e5-a68d5046aba6/volumes"
Oct 30 13:18:00.975471 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:00.975455    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5298fc7-1369-4e44-8ee3-f058d1e20c37" path="/var/lib/kubelet/pods/d5298fc7-1369-4e44-8ee3-f058d1e20c37/volumes"
Oct 30 13:18:01.002912 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:01.002887    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq"
Oct 30 13:18:01.044703 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:01.044675    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d4826806-75a8-47d6-866b-415116519d9d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4826806-75a8-47d6-866b-415116519d9d-istio-podinfo\") pod \"d4826806-75a8-47d6-866b-415116519d9d\" (UID: \"d4826806-75a8-47d6-866b-415116519d9d\") "
Oct 30 13:18:01.044824 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:01.044718    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4826806-75a8-47d6-866b-415116519d9d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4826806-75a8-47d6-866b-415116519d9d-istiod-ca-cert\") pod \"d4826806-75a8-47d6-866b-415116519d9d\" (UID: \"d4826806-75a8-47d6-866b-415116519d9d\") "
Oct 30 13:18:01.044824 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:01.044770    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4826806-75a8-47d6-866b-415116519d9d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4826806-75a8-47d6-866b-415116519d9d-credential-socket\") pod \"d4826806-75a8-47d6-866b-415116519d9d\" (UID: \"d4826806-75a8-47d6-866b-415116519d9d\") "
Oct 30 13:18:01.044824 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:01.044818    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4826806-75a8-47d6-866b-415116519d9d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4826806-75a8-47d6-866b-415116519d9d-workload-certs\") pod \"d4826806-75a8-47d6-866b-415116519d9d\" (UID: \"d4826806-75a8-47d6-866b-415116519d9d\") "
Oct 30 13:18:01.044991 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:01.044975    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4826806-75a8-47d6-866b-415116519d9d-kube-api-access-xgwnz\" (UniqueName: \"kubernetes.io/projected/d4826806-75a8-47d6-866b-415116519d9d-kube-api-access-xgwnz\") pod \"d4826806-75a8-47d6-866b-415116519d9d\" (UID: \"d4826806-75a8-47d6-866b-415116519d9d\") "
Oct 30 13:18:01.045078 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:01.045048    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4826806-75a8-47d6-866b-415116519d9d-credential-socket" pod "d4826806-75a8-47d6-866b-415116519d9d" (UID: "d4826806-75a8-47d6-866b-415116519d9d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:18:01.045135 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:01.045082    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4826806-75a8-47d6-866b-415116519d9d-workload-certs" pod "d4826806-75a8-47d6-866b-415116519d9d" (UID: "d4826806-75a8-47d6-866b-415116519d9d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:18:01.045135 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:01.045115    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4826806-75a8-47d6-866b-415116519d9d-istio-token\" (UniqueName: \"kubernetes.io/projected/d4826806-75a8-47d6-866b-415116519d9d-istio-token\") pod \"d4826806-75a8-47d6-866b-415116519d9d\" (UID: \"d4826806-75a8-47d6-866b-415116519d9d\") "
Oct 30 13:18:01.045239 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:01.045150    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4826806-75a8-47d6-866b-415116519d9d-istiod-ca-cert" pod "d4826806-75a8-47d6-866b-415116519d9d" (UID: "d4826806-75a8-47d6-866b-415116519d9d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:18:01.045239 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:01.045180    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4826806-75a8-47d6-866b-415116519d9d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4826806-75a8-47d6-866b-415116519d9d-workload-socket\") pod \"d4826806-75a8-47d6-866b-415116519d9d\" (UID: \"d4826806-75a8-47d6-866b-415116519d9d\") "
Oct 30 13:18:01.045239 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:01.045210    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4826806-75a8-47d6-866b-415116519d9d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4826806-75a8-47d6-866b-415116519d9d-istio-envoy\") pod \"d4826806-75a8-47d6-866b-415116519d9d\" (UID: \"d4826806-75a8-47d6-866b-415116519d9d\") "
Oct 30 13:18:01.045391 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:01.045308    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4826806-75a8-47d6-866b-415116519d9d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4826806-75a8-47d6-866b-415116519d9d-istio-data\") pod \"d4826806-75a8-47d6-866b-415116519d9d\" (UID: \"d4826806-75a8-47d6-866b-415116519d9d\") "
Oct 30 13:18:01.045484 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:01.045461    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4826806-75a8-47d6-866b-415116519d9d-workload-socket" pod "d4826806-75a8-47d6-866b-415116519d9d" (UID: "d4826806-75a8-47d6-866b-415116519d9d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:18:01.045754 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:01.045688    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4826806-75a8-47d6-866b-415116519d9d-istio-data" pod "d4826806-75a8-47d6-866b-415116519d9d" (UID: "d4826806-75a8-47d6-866b-415116519d9d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:18:01.046311 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:01.046284    4364 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4826806-75a8-47d6-866b-415116519d9d-workload-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:18:01.046420 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:01.046314    4364 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4826806-75a8-47d6-866b-415116519d9d-istio-data\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:18:01.046420 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:01.046329    4364 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4826806-75a8-47d6-866b-415116519d9d-istiod-ca-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:18:01.046420 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:01.046343    4364 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4826806-75a8-47d6-866b-415116519d9d-credential-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:18:01.046420 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:01.046356    4364 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4826806-75a8-47d6-866b-415116519d9d-workload-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:18:01.047362 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:01.047332    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4826806-75a8-47d6-866b-415116519d9d-kube-api-access-xgwnz" pod "d4826806-75a8-47d6-866b-415116519d9d" (UID: "d4826806-75a8-47d6-866b-415116519d9d"). InnerVolumeSpecName "kube-api-access-xgwnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:18:01.047362 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:01.047355    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4826806-75a8-47d6-866b-415116519d9d-istio-token" pod "d4826806-75a8-47d6-866b-415116519d9d" (UID: "d4826806-75a8-47d6-866b-415116519d9d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:18:01.047532 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:01.047434    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d4826806-75a8-47d6-866b-415116519d9d-istio-podinfo" pod "d4826806-75a8-47d6-866b-415116519d9d" (UID: "d4826806-75a8-47d6-866b-415116519d9d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:18:01.048094 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:01.048072    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4826806-75a8-47d6-866b-415116519d9d-istio-envoy" pod "d4826806-75a8-47d6-866b-415116519d9d" (UID: "d4826806-75a8-47d6-866b-415116519d9d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:18:01.147783 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:01.147755    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgwnz\" (UniqueName: \"kubernetes.io/projected/d4826806-75a8-47d6-866b-415116519d9d-kube-api-access-xgwnz\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:18:01.147783 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:01.147781    4364 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4826806-75a8-47d6-866b-415116519d9d-istio-token\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:18:01.147783 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:01.147790    4364 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4826806-75a8-47d6-866b-415116519d9d-istio-envoy\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:18:01.147985 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:01.147798    4364 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4826806-75a8-47d6-866b-415116519d9d-istio-podinfo\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:18:01.511740 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:01.511704    4364 generic.go:356] "Generic (PLEG): container finished" podID="d4826806-75a8-47d6-866b-415116519d9d" containerID="1b14ca834bdf6d9785e98377ab7a04e200530a0de03e2c2a2622fc357ba80a6d" exitCode=0
Oct 30 13:18:01.511921 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:01.511753    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq"
Oct 30 13:18:01.511921 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:01.511793    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq" event={"ID":"d4826806-75a8-47d6-866b-415116519d9d","Type":"ContainerDied","Data":"1b14ca834bdf6d9785e98377ab7a04e200530a0de03e2c2a2622fc357ba80a6d"}
Oct 30 13:18:01.511921 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:01.511851    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq" event={"ID":"d4826806-75a8-47d6-866b-415116519d9d","Type":"ContainerDied","Data":"e693417bc6d5c9edf796d38661cf128b9edf5229d0a252f30ad3906a71e89847"}
Oct 30 13:18:01.511921 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:01.511877    4364 scope.go:122] "RemoveContainer" containerID="1b14ca834bdf6d9785e98377ab7a04e200530a0de03e2c2a2622fc357ba80a6d"
Oct 30 13:18:01.523578 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:01.523557    4364 scope.go:122] "RemoveContainer" containerID="1b14ca834bdf6d9785e98377ab7a04e200530a0de03e2c2a2622fc357ba80a6d"
Oct 30 13:18:01.523836 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:18:01.523819    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b14ca834bdf6d9785e98377ab7a04e200530a0de03e2c2a2622fc357ba80a6d\": container with ID starting with 1b14ca834bdf6d9785e98377ab7a04e200530a0de03e2c2a2622fc357ba80a6d not found: ID does not exist" containerID="1b14ca834bdf6d9785e98377ab7a04e200530a0de03e2c2a2622fc357ba80a6d"
Oct 30 13:18:01.523883 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:01.523843    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b14ca834bdf6d9785e98377ab7a04e200530a0de03e2c2a2622fc357ba80a6d"} err="failed to get container status \"1b14ca834bdf6d9785e98377ab7a04e200530a0de03e2c2a2622fc357ba80a6d\": rpc error: code = NotFound desc = could not find container \"1b14ca834bdf6d9785e98377ab7a04e200530a0de03e2c2a2622fc357ba80a6d\": container with ID starting with 1b14ca834bdf6d9785e98377ab7a04e200530a0de03e2c2a2622fc357ba80a6d not found: ID does not exist"
Oct 30 13:18:01.569660 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:01.569633    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq"]
Oct 30 13:18:01.581883 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:01.581860    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-hzvmq"]
Oct 30 13:18:02.974000 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:02.973963    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4826806-75a8-47d6-866b-415116519d9d" path="/var/lib/kubelet/pods/d4826806-75a8-47d6-866b-415116519d9d/volumes"
Oct 30 13:18:10.435668 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:10.435631    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2daeae-5f484f5b8-wzx8f"]
Oct 30 13:18:10.467313 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:10.467283    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2daeae-5b6fb8f887-22zjb"]
Oct 30 13:18:12.112624 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:18:12.104113    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:18:12.323627 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:12.318713    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz" podUID="a568f1ce-602d-4069-a41c-9fe1a7635c5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:18:14.572205 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:14.572172    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-77254d-55f7f48fd6-twrk7"]
Oct 30 13:18:14.572936 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:14.572922    4364 cpu_manager.go:414] "containerMap: removing container" podUID="42c43460-b498-432d-a8e5-a68d5046aba6" containerName="payload-pre-processing"
Oct 30 13:18:14.573002 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:14.572938    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42c43460-b498-432d-a8e5-a68d5046aba6" containerName="payload-pre-processing"
Oct 30 13:18:14.573002 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:14.572951    4364 cpu_manager.go:414] "containerMap: removing container" podUID="d5298fc7-1369-4e44-8ee3-f058d1e20c37" containerName="payload-processing"
Oct 30 13:18:14.573002 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:14.572956    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5298fc7-1369-4e44-8ee3-f058d1e20c37" containerName="payload-processing"
Oct 30 13:18:14.573002 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:14.572969    4364 cpu_manager.go:414] "containerMap: removing container" podUID="8ea01099-fc92-469c-99cf-875275e18a04" containerName="discovery"
Oct 30 13:18:14.573002 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:14.572974    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ea01099-fc92-469c-99cf-875275e18a04" containerName="discovery"
Oct 30 13:18:14.573002 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:14.572981    4364 cpu_manager.go:414] "containerMap: removing container" podUID="f10e7b19-6e46-4e00-b631-6bc0a94b98a7" containerName="revoke-keys"
Oct 30 13:18:14.573002 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:14.572986    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f10e7b19-6e46-4e00-b631-6bc0a94b98a7" containerName="revoke-keys"
Oct 30 13:18:14.573002 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:14.572997    4364 cpu_manager.go:414] "containerMap: removing container" podUID="79f77504-1067-40e5-9ea6-1f7b752d0f8d" containerName="payload-processing"
Oct 30 13:18:14.573002 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:14.573002    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79f77504-1067-40e5-9ea6-1f7b752d0f8d" containerName="payload-processing"
Oct 30 13:18:14.573277 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:14.573015    4364 cpu_manager.go:414] "containerMap: removing container" podUID="d4826806-75a8-47d6-866b-415116519d9d" containerName="istio-proxy"
Oct 30 13:18:14.573277 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:14.573020    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4826806-75a8-47d6-866b-415116519d9d" containerName="istio-proxy"
Oct 30 13:18:14.573277 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:14.573033    4364 cpu_manager.go:414] "containerMap: removing container" podUID="afeac9f9-fa22-47b0-8446-9683377383e2" containerName="payload-processing"
Oct 30 13:18:14.573277 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:14.573037    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afeac9f9-fa22-47b0-8446-9683377383e2" containerName="payload-processing"
Oct 30 13:18:14.573277 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:14.573053    4364 cpu_manager.go:414] "containerMap: removing container" podUID="b564990b-f890-4e82-a667-c24363ec69d2" containerName="payload-pre-processing"
Oct 30 13:18:14.573277 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:14.573059    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b564990b-f890-4e82-a667-c24363ec69d2" containerName="payload-pre-processing"
Oct 30 13:18:14.573277 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:14.573069    4364 cpu_manager.go:414] "containerMap: removing container" podUID="3fb63d98-c5f7-49fd-9ba0-b58f7329bd63" containerName="payload-pre-processing"
Oct 30 13:18:14.573277 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:14.573073    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fb63d98-c5f7-49fd-9ba0-b58f7329bd63" containerName="payload-pre-processing"
Oct 30 13:18:14.573277 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:14.573087    4364 cpu_manager.go:414] "containerMap: removing container" podUID="bbbe037e-4f22-4ecd-9d73-d8d32419647f" containerName="payload-pre-processing"
Oct 30 13:18:14.573277 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:14.573092    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbbe037e-4f22-4ecd-9d73-d8d32419647f" containerName="payload-pre-processing"
Oct 30 13:18:14.573277 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:14.573190    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="afeac9f9-fa22-47b0-8446-9683377383e2" containerName="payload-processing"
Oct 30 13:18:14.573277 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:14.573202    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="b564990b-f890-4e82-a667-c24363ec69d2" containerName="payload-pre-processing"
Oct 30 13:18:14.573277 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:14.573209    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fb63d98-c5f7-49fd-9ba0-b58f7329bd63" containerName="payload-pre-processing"
Oct 30 13:18:14.573277 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:14.573217    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbbe037e-4f22-4ecd-9d73-d8d32419647f" containerName="payload-pre-processing"
Oct 30 13:18:14.573277 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:14.573225    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="42c43460-b498-432d-a8e5-a68d5046aba6" containerName="payload-pre-processing"
Oct 30 13:18:14.573277 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:14.573231    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5298fc7-1369-4e44-8ee3-f058d1e20c37" containerName="payload-processing"
Oct 30 13:18:14.573277 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:14.573237    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ea01099-fc92-469c-99cf-875275e18a04" containerName="discovery"
Oct 30 13:18:14.573277 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:14.573243    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="f10e7b19-6e46-4e00-b631-6bc0a94b98a7" containerName="revoke-keys"
Oct 30 13:18:14.573277 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:14.573249    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="79f77504-1067-40e5-9ea6-1f7b752d0f8d" containerName="payload-processing"
Oct 30 13:18:14.573277 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:14.573256    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4826806-75a8-47d6-866b-415116519d9d" containerName="istio-proxy"
Oct 30 13:18:14.577112 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:14.577097    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-77254d-55f7f48fd6-twrk7"
Oct 30 13:18:14.599129 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:14.599103    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-77254d-serving-cert\""
Oct 30 13:18:14.605591 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:14.605565    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-77254d-55f7f48fd6-twrk7"]
Oct 30 13:18:14.620998 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:14.620970    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cf49036-ea5d-4282-b11e-5ff1c01262d4-metrics-certs\") pod \"maas-api-e2e-inf-a-77254d-55f7f48fd6-twrk7\" (UID: \"2cf49036-ea5d-4282-b11e-5ff1c01262d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-77254d-55f7f48fd6-twrk7"
Oct 30 13:18:14.621163 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:14.621101    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgjbj\" (UniqueName: \"kubernetes.io/projected/2cf49036-ea5d-4282-b11e-5ff1c01262d4-kube-api-access-wgjbj\") pod \"maas-api-e2e-inf-a-77254d-55f7f48fd6-twrk7\" (UID: \"2cf49036-ea5d-4282-b11e-5ff1c01262d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-77254d-55f7f48fd6-twrk7"
Oct 30 13:18:14.621279 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:14.621261    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cf49036-ea5d-4282-b11e-5ff1c01262d4-maas-api-tls\") pod \"maas-api-e2e-inf-a-77254d-55f7f48fd6-twrk7\" (UID: \"2cf49036-ea5d-4282-b11e-5ff1c01262d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-77254d-55f7f48fd6-twrk7"
Oct 30 13:18:14.621358 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:14.621297    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cf49036-ea5d-4282-b11e-5ff1c01262d4-openshift-service-ca\") pod \"maas-api-e2e-inf-a-77254d-55f7f48fd6-twrk7\" (UID: \"2cf49036-ea5d-4282-b11e-5ff1c01262d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-77254d-55f7f48fd6-twrk7"
Oct 30 13:18:14.722712 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:14.722685    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cf49036-ea5d-4282-b11e-5ff1c01262d4-metrics-certs\") pod \"maas-api-e2e-inf-a-77254d-55f7f48fd6-twrk7\" (UID: \"2cf49036-ea5d-4282-b11e-5ff1c01262d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-77254d-55f7f48fd6-twrk7"
Oct 30 13:18:14.722821 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:14.722742    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgjbj\" (UniqueName: \"kubernetes.io/projected/2cf49036-ea5d-4282-b11e-5ff1c01262d4-kube-api-access-wgjbj\") pod \"maas-api-e2e-inf-a-77254d-55f7f48fd6-twrk7\" (UID: \"2cf49036-ea5d-4282-b11e-5ff1c01262d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-77254d-55f7f48fd6-twrk7"
Oct 30 13:18:14.722874 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:14.722822    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cf49036-ea5d-4282-b11e-5ff1c01262d4-maas-api-tls\") pod \"maas-api-e2e-inf-a-77254d-55f7f48fd6-twrk7\" (UID: \"2cf49036-ea5d-4282-b11e-5ff1c01262d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-77254d-55f7f48fd6-twrk7"
Oct 30 13:18:14.722874 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:14.722841    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cf49036-ea5d-4282-b11e-5ff1c01262d4-openshift-service-ca\") pod \"maas-api-e2e-inf-a-77254d-55f7f48fd6-twrk7\" (UID: \"2cf49036-ea5d-4282-b11e-5ff1c01262d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-77254d-55f7f48fd6-twrk7"
Oct 30 13:18:14.723538 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:14.723509    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cf49036-ea5d-4282-b11e-5ff1c01262d4-openshift-service-ca\") pod \"maas-api-e2e-inf-a-77254d-55f7f48fd6-twrk7\" (UID: \"2cf49036-ea5d-4282-b11e-5ff1c01262d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-77254d-55f7f48fd6-twrk7"
Oct 30 13:18:14.725125 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:14.725103    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cf49036-ea5d-4282-b11e-5ff1c01262d4-metrics-certs\") pod \"maas-api-e2e-inf-a-77254d-55f7f48fd6-twrk7\" (UID: \"2cf49036-ea5d-4282-b11e-5ff1c01262d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-77254d-55f7f48fd6-twrk7"
Oct 30 13:18:14.725273 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:14.725250    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cf49036-ea5d-4282-b11e-5ff1c01262d4-maas-api-tls\") pod \"maas-api-e2e-inf-a-77254d-55f7f48fd6-twrk7\" (UID: \"2cf49036-ea5d-4282-b11e-5ff1c01262d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-77254d-55f7f48fd6-twrk7"
Oct 30 13:18:14.746209 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:14.746181    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgjbj\" (UniqueName: \"kubernetes.io/projected/2cf49036-ea5d-4282-b11e-5ff1c01262d4-kube-api-access-wgjbj\") pod \"maas-api-e2e-inf-a-77254d-55f7f48fd6-twrk7\" (UID: \"2cf49036-ea5d-4282-b11e-5ff1c01262d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-77254d-55f7f48fd6-twrk7"
Oct 30 13:18:14.891904 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:14.891681    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-77254d-55f7f48fd6-twrk7"
Oct 30 13:18:15.221032 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:15.221001    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-77254d-55f7f48fd6-twrk7"]
Oct 30 13:18:15.221316 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:18:15.221285    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cf49036_ea5d_4282_b11e_5ff1c01262d4.slice/crio-c9f8c2a7d1e4be43b8a4784e5c3c52415b27880ab7e9f43aa9ec3b05169eae72 WatchSource:0}: Error finding container c9f8c2a7d1e4be43b8a4784e5c3c52415b27880ab7e9f43aa9ec3b05169eae72: Status 404 returned error can't find the container with id c9f8c2a7d1e4be43b8a4784e5c3c52415b27880ab7e9f43aa9ec3b05169eae72
Oct 30 13:18:15.436678 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:15.436632    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-2daeae-5f484f5b8-wzx8f" podUID="b770e4ef-cfb0-41d8-90dc-2b95d06f4c9e" containerName="payload-processing" containerID="cri-o://9098920df2c800094b3dd393a78cd78e61e724684bab34f423da74faa0cdcf91" gracePeriod=25
Oct 30 13:18:15.468402 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:15.468345    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2daeae-5b6fb8f887-22zjb" podUID="dfb4ff14-8dac-4cd5-944a-2aee68792901" containerName="payload-pre-processing" containerID="cri-o://fd76b622842d35ed4b232c19a127cc56d00de9a5038cbd5d744442ac4e2032d8" gracePeriod=25
Oct 30 13:18:15.858002 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:15.857975    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2daeae-5f484f5b8-wzx8f"
Oct 30 13:18:15.881979 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:15.881956    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2daeae-5b6fb8f887-22zjb"
Oct 30 13:18:15.947915 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:15.947883    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b770e4ef-cfb0-41d8-90dc-2b95d06f4c9e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b770e4ef-cfb0-41d8-90dc-2b95d06f4c9e-plugins-config-volume\") pod \"b770e4ef-cfb0-41d8-90dc-2b95d06f4c9e\" (UID: \"b770e4ef-cfb0-41d8-90dc-2b95d06f4c9e\") "
Oct 30 13:18:15.948102 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:15.947970    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfb4ff14-8dac-4cd5-944a-2aee68792901-kube-api-access-59wrg\" (UniqueName: \"kubernetes.io/projected/dfb4ff14-8dac-4cd5-944a-2aee68792901-kube-api-access-59wrg\") pod \"dfb4ff14-8dac-4cd5-944a-2aee68792901\" (UID: \"dfb4ff14-8dac-4cd5-944a-2aee68792901\") "
Oct 30 13:18:15.948170 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:15.948120    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfb4ff14-8dac-4cd5-944a-2aee68792901-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfb4ff14-8dac-4cd5-944a-2aee68792901-plugins-config-volume\") pod \"dfb4ff14-8dac-4cd5-944a-2aee68792901\" (UID: \"dfb4ff14-8dac-4cd5-944a-2aee68792901\") "
Oct 30 13:18:15.948231 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:15.948191    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b770e4ef-cfb0-41d8-90dc-2b95d06f4c9e-kube-api-access-v8k55\" (UniqueName: \"kubernetes.io/projected/b770e4ef-cfb0-41d8-90dc-2b95d06f4c9e-kube-api-access-v8k55\") pod \"b770e4ef-cfb0-41d8-90dc-2b95d06f4c9e\" (UID: \"b770e4ef-cfb0-41d8-90dc-2b95d06f4c9e\") "
Oct 30 13:18:15.948906 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:15.948662    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfb4ff14-8dac-4cd5-944a-2aee68792901-plugins-config-volume" pod "dfb4ff14-8dac-4cd5-944a-2aee68792901" (UID: "dfb4ff14-8dac-4cd5-944a-2aee68792901"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:18:15.949041 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:15.948918    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b770e4ef-cfb0-41d8-90dc-2b95d06f4c9e-plugins-config-volume" pod "b770e4ef-cfb0-41d8-90dc-2b95d06f4c9e" (UID: "b770e4ef-cfb0-41d8-90dc-2b95d06f4c9e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:18:15.949457 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:15.949393    4364 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b770e4ef-cfb0-41d8-90dc-2b95d06f4c9e-plugins-config-volume\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:18:15.949457 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:15.949423    4364 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfb4ff14-8dac-4cd5-944a-2aee68792901-plugins-config-volume\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:18:15.951470 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:15.951447    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfb4ff14-8dac-4cd5-944a-2aee68792901-kube-api-access-59wrg" pod "dfb4ff14-8dac-4cd5-944a-2aee68792901" (UID: "dfb4ff14-8dac-4cd5-944a-2aee68792901"). InnerVolumeSpecName "kube-api-access-59wrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:18:15.951725 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:15.951576    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b770e4ef-cfb0-41d8-90dc-2b95d06f4c9e-kube-api-access-v8k55" pod "b770e4ef-cfb0-41d8-90dc-2b95d06f4c9e" (UID: "b770e4ef-cfb0-41d8-90dc-2b95d06f4c9e"). InnerVolumeSpecName "kube-api-access-v8k55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:18:15.996034 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:15.995967    4364 generic.go:356] "Generic (PLEG): container finished" podID="dfb4ff14-8dac-4cd5-944a-2aee68792901" containerID="fd76b622842d35ed4b232c19a127cc56d00de9a5038cbd5d744442ac4e2032d8" exitCode=0
Oct 30 13:18:15.996034 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:15.996008    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2daeae-5b6fb8f887-22zjb"
Oct 30 13:18:15.996204 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:15.996055    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2daeae-5b6fb8f887-22zjb" event={"ID":"dfb4ff14-8dac-4cd5-944a-2aee68792901","Type":"ContainerDied","Data":"fd76b622842d35ed4b232c19a127cc56d00de9a5038cbd5d744442ac4e2032d8"}
Oct 30 13:18:15.996204 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:15.996103    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2daeae-5b6fb8f887-22zjb" event={"ID":"dfb4ff14-8dac-4cd5-944a-2aee68792901","Type":"ContainerDied","Data":"5b7182cc0b2934e9c9f98c61873f61e6e7d164685647616c2f4e831a40759deb"}
Oct 30 13:18:15.996204 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:15.996128    4364 scope.go:122] "RemoveContainer" containerID="fd76b622842d35ed4b232c19a127cc56d00de9a5038cbd5d744442ac4e2032d8"
Oct 30 13:18:15.997841 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:15.997494    4364 generic.go:356] "Generic (PLEG): container finished" podID="b770e4ef-cfb0-41d8-90dc-2b95d06f4c9e" containerID="9098920df2c800094b3dd393a78cd78e61e724684bab34f423da74faa0cdcf91" exitCode=0
Oct 30 13:18:15.997841 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:15.997675    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2daeae-5f484f5b8-wzx8f" event={"ID":"b770e4ef-cfb0-41d8-90dc-2b95d06f4c9e","Type":"ContainerDied","Data":"9098920df2c800094b3dd393a78cd78e61e724684bab34f423da74faa0cdcf91"}
Oct 30 13:18:15.997841 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:15.997721    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2daeae-5f484f5b8-wzx8f" event={"ID":"b770e4ef-cfb0-41d8-90dc-2b95d06f4c9e","Type":"ContainerDied","Data":"21ee457e0a243777f2662c9c86c7e6130d82d7ba8224fc65ae1075f2ebaf485a"}
Oct 30 13:18:15.997841 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:15.997800    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2daeae-5f484f5b8-wzx8f"
Oct 30 13:18:16.001274 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:16.001247    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-77254d-55f7f48fd6-twrk7" event={"ID":"2cf49036-ea5d-4282-b11e-5ff1c01262d4","Type":"ContainerStarted","Data":"7437f6fb9ff0976fa4b90510e464fa8ff2aea1129600b542083148a2b4cebb0b"}
Oct 30 13:18:16.001397 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:16.001296    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-77254d-55f7f48fd6-twrk7" event={"ID":"2cf49036-ea5d-4282-b11e-5ff1c01262d4","Type":"ContainerStarted","Data":"c9f8c2a7d1e4be43b8a4784e5c3c52415b27880ab7e9f43aa9ec3b05169eae72"}
Oct 30 13:18:16.002073 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:16.002043    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-77254d-55f7f48fd6-twrk7"
Oct 30 13:18:16.012013 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:16.011998    4364 scope.go:122] "RemoveContainer" containerID="fd76b622842d35ed4b232c19a127cc56d00de9a5038cbd5d744442ac4e2032d8"
Oct 30 13:18:16.012291 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:18:16.012271    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd76b622842d35ed4b232c19a127cc56d00de9a5038cbd5d744442ac4e2032d8\": container with ID starting with fd76b622842d35ed4b232c19a127cc56d00de9a5038cbd5d744442ac4e2032d8 not found: ID does not exist" containerID="fd76b622842d35ed4b232c19a127cc56d00de9a5038cbd5d744442ac4e2032d8"
Oct 30 13:18:16.012378 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:16.012300    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd76b622842d35ed4b232c19a127cc56d00de9a5038cbd5d744442ac4e2032d8"} err="failed to get container status \"fd76b622842d35ed4b232c19a127cc56d00de9a5038cbd5d744442ac4e2032d8\": rpc error: code = NotFound desc = could not find container \"fd76b622842d35ed4b232c19a127cc56d00de9a5038cbd5d744442ac4e2032d8\": container with ID starting with fd76b622842d35ed4b232c19a127cc56d00de9a5038cbd5d744442ac4e2032d8 not found: ID does not exist"
Oct 30 13:18:16.012378 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:16.012321    4364 scope.go:122] "RemoveContainer" containerID="9098920df2c800094b3dd393a78cd78e61e724684bab34f423da74faa0cdcf91"
Oct 30 13:18:16.021888 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:16.021866    4364 scope.go:122] "RemoveContainer" containerID="9098920df2c800094b3dd393a78cd78e61e724684bab34f423da74faa0cdcf91"
Oct 30 13:18:16.022161 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:18:16.022141    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9098920df2c800094b3dd393a78cd78e61e724684bab34f423da74faa0cdcf91\": container with ID starting with 9098920df2c800094b3dd393a78cd78e61e724684bab34f423da74faa0cdcf91 not found: ID does not exist" containerID="9098920df2c800094b3dd393a78cd78e61e724684bab34f423da74faa0cdcf91"
Oct 30 13:18:16.022243 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:16.022167    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9098920df2c800094b3dd393a78cd78e61e724684bab34f423da74faa0cdcf91"} err="failed to get container status \"9098920df2c800094b3dd393a78cd78e61e724684bab34f423da74faa0cdcf91\": rpc error: code = NotFound desc = could not find container \"9098920df2c800094b3dd393a78cd78e61e724684bab34f423da74faa0cdcf91\": container with ID starting with 9098920df2c800094b3dd393a78cd78e61e724684bab34f423da74faa0cdcf91 not found: ID does not exist"
Oct 30 13:18:16.051526 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:16.051376    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59wrg\" (UniqueName: \"kubernetes.io/projected/dfb4ff14-8dac-4cd5-944a-2aee68792901-kube-api-access-59wrg\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:18:16.051526 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:16.051411    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8k55\" (UniqueName: \"kubernetes.io/projected/b770e4ef-cfb0-41d8-90dc-2b95d06f4c9e-kube-api-access-v8k55\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:18:16.051816 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:16.051755    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2daeae-5f484f5b8-wzx8f"]
Oct 30 13:18:16.107859 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:16.085884    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2daeae-5f484f5b8-wzx8f"]
Oct 30 13:18:16.138620 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:16.128013    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-77254d-55f7f48fd6-twrk7" podStartSLOduration=1.633875747 podStartE2EDuration="2.127992805s" totalImagesPullingTime="494.117058ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:18:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:18:16.108466699 +0000 UTC m=+1595.274831349" watchObservedRunningTime="2026-10-30 13:18:16.127992805 +0000 UTC m=+1595.294357454"
Oct 30 13:18:16.286073 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:16.285991    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2daeae-5b6fb8f887-22zjb"]
Oct 30 13:18:16.286333 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:16.286285    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2daeae-5b6fb8f887-22zjb"]
Oct 30 13:18:17.014559 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:17.014475    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b770e4ef-cfb0-41d8-90dc-2b95d06f4c9e" path="/var/lib/kubelet/pods/b770e4ef-cfb0-41d8-90dc-2b95d06f4c9e/volumes"
Oct 30 13:18:17.015094 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:17.015075    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfb4ff14-8dac-4cd5-944a-2aee68792901" path="/var/lib/kubelet/pods/dfb4ff14-8dac-4cd5-944a-2aee68792901/volumes"
Oct 30 13:18:20.361967 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:20.361936    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx"]
Oct 30 13:18:20.362549 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:20.362138    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx" podUID="cd6e6964-3e54-48f1-a184-8db6ef093a7b" containerName="istio-proxy" containerID="cri-o://841e79331eff502018036c927e124e95de3674c0570c3c1be13453a228aaf208" gracePeriod=30
Oct 30 13:18:21.329293 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:21.329249    4364 patch_prober.go:29] interesting pod/e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:18:21.329480 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:21.329316    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx" podUID="cd6e6964-3e54-48f1-a184-8db6ef093a7b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:18:21.723078 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:21.723042    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz"]
Oct 30 13:18:21.725156 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:21.725119    4364 cpu_manager.go:414] "containerMap: removing container" podUID="b770e4ef-cfb0-41d8-90dc-2b95d06f4c9e" containerName="payload-processing"
Oct 30 13:18:21.725156 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:21.725145    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b770e4ef-cfb0-41d8-90dc-2b95d06f4c9e" containerName="payload-processing"
Oct 30 13:18:21.725362 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:21.725167    4364 cpu_manager.go:414] "containerMap: removing container" podUID="dfb4ff14-8dac-4cd5-944a-2aee68792901" containerName="payload-pre-processing"
Oct 30 13:18:21.725362 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:21.725173    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfb4ff14-8dac-4cd5-944a-2aee68792901" containerName="payload-pre-processing"
Oct 30 13:18:21.725484 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:21.725387    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="b770e4ef-cfb0-41d8-90dc-2b95d06f4c9e" containerName="payload-processing"
Oct 30 13:18:21.725484 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:21.725402    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfb4ff14-8dac-4cd5-944a-2aee68792901" containerName="payload-pre-processing"
Oct 30 13:18:21.732168 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:21.732137    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz"
Oct 30 13:18:21.747011 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:21.745163    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz"]
Oct 30 13:18:21.753238 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:21.753214    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-b5a369-openshift-default-dockercfg-55749\""
Oct 30 13:18:21.858443 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:21.858404    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52bf85ca-e35d-4a85-85f4-21d6d20a191d-istio-envoy\") pod \"e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz\" (UID: \"52bf85ca-e35d-4a85-85f4-21d6d20a191d\") " pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz"
Oct 30 13:18:21.858660 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:21.858459    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52bf85ca-e35d-4a85-85f4-21d6d20a191d-workload-socket\") pod \"e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz\" (UID: \"52bf85ca-e35d-4a85-85f4-21d6d20a191d\") " pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz"
Oct 30 13:18:21.858660 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:21.858584    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52bf85ca-e35d-4a85-85f4-21d6d20a191d-istio-podinfo\") pod \"e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz\" (UID: \"52bf85ca-e35d-4a85-85f4-21d6d20a191d\") " pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz"
Oct 30 13:18:21.858801 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:21.858679    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52bf85ca-e35d-4a85-85f4-21d6d20a191d-workload-certs\") pod \"e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz\" (UID: \"52bf85ca-e35d-4a85-85f4-21d6d20a191d\") " pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz"
Oct 30 13:18:21.858801 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:21.858706    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52bf85ca-e35d-4a85-85f4-21d6d20a191d-credential-socket\") pod \"e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz\" (UID: \"52bf85ca-e35d-4a85-85f4-21d6d20a191d\") " pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz"
Oct 30 13:18:21.858895 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:21.858795    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52bf85ca-e35d-4a85-85f4-21d6d20a191d-istiod-ca-cert\") pod \"e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz\" (UID: \"52bf85ca-e35d-4a85-85f4-21d6d20a191d\") " pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz"
Oct 30 13:18:21.858958 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:21.858945    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwf9s\" (UniqueName: \"kubernetes.io/projected/52bf85ca-e35d-4a85-85f4-21d6d20a191d-kube-api-access-lwf9s\") pod \"e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz\" (UID: \"52bf85ca-e35d-4a85-85f4-21d6d20a191d\") " pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz"
Oct 30 13:18:21.859070 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:21.859054    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52bf85ca-e35d-4a85-85f4-21d6d20a191d-istio-data\") pod \"e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz\" (UID: \"52bf85ca-e35d-4a85-85f4-21d6d20a191d\") " pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz"
Oct 30 13:18:21.859196 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:21.859179    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52bf85ca-e35d-4a85-85f4-21d6d20a191d-istio-token\") pod \"e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz\" (UID: \"52bf85ca-e35d-4a85-85f4-21d6d20a191d\") " pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz"
Oct 30 13:18:21.962624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:21.960822    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwf9s\" (UniqueName: \"kubernetes.io/projected/52bf85ca-e35d-4a85-85f4-21d6d20a191d-kube-api-access-lwf9s\") pod \"e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz\" (UID: \"52bf85ca-e35d-4a85-85f4-21d6d20a191d\") " pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz"
Oct 30 13:18:21.962624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:21.960964    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52bf85ca-e35d-4a85-85f4-21d6d20a191d-istio-data\") pod \"e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz\" (UID: \"52bf85ca-e35d-4a85-85f4-21d6d20a191d\") " pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz"
Oct 30 13:18:21.962624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:21.961086    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52bf85ca-e35d-4a85-85f4-21d6d20a191d-istio-token\") pod \"e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz\" (UID: \"52bf85ca-e35d-4a85-85f4-21d6d20a191d\") " pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz"
Oct 30 13:18:21.962624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:21.961121    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52bf85ca-e35d-4a85-85f4-21d6d20a191d-istio-envoy\") pod \"e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz\" (UID: \"52bf85ca-e35d-4a85-85f4-21d6d20a191d\") " pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz"
Oct 30 13:18:21.962624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:21.961160    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52bf85ca-e35d-4a85-85f4-21d6d20a191d-workload-socket\") pod \"e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz\" (UID: \"52bf85ca-e35d-4a85-85f4-21d6d20a191d\") " pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz"
Oct 30 13:18:21.962624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:21.961218    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52bf85ca-e35d-4a85-85f4-21d6d20a191d-istio-podinfo\") pod \"e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz\" (UID: \"52bf85ca-e35d-4a85-85f4-21d6d20a191d\") " pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz"
Oct 30 13:18:21.962624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:21.961260    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52bf85ca-e35d-4a85-85f4-21d6d20a191d-workload-certs\") pod \"e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz\" (UID: \"52bf85ca-e35d-4a85-85f4-21d6d20a191d\") " pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz"
Oct 30 13:18:21.962624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:21.961290    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52bf85ca-e35d-4a85-85f4-21d6d20a191d-credential-socket\") pod \"e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz\" (UID: \"52bf85ca-e35d-4a85-85f4-21d6d20a191d\") " pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz"
Oct 30 13:18:21.962624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:21.961334    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52bf85ca-e35d-4a85-85f4-21d6d20a191d-istiod-ca-cert\") pod \"e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz\" (UID: \"52bf85ca-e35d-4a85-85f4-21d6d20a191d\") " pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz"
Oct 30 13:18:21.962624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:21.962261    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52bf85ca-e35d-4a85-85f4-21d6d20a191d-istiod-ca-cert\") pod \"e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz\" (UID: \"52bf85ca-e35d-4a85-85f4-21d6d20a191d\") " pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz"
Oct 30 13:18:22.010694 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:22.002570    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52bf85ca-e35d-4a85-85f4-21d6d20a191d-istio-data\") pod \"e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz\" (UID: \"52bf85ca-e35d-4a85-85f4-21d6d20a191d\") " pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz"
Oct 30 13:18:22.010694 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:22.002942    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52bf85ca-e35d-4a85-85f4-21d6d20a191d-workload-socket\") pod \"e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz\" (UID: \"52bf85ca-e35d-4a85-85f4-21d6d20a191d\") " pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz"
Oct 30 13:18:22.036623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:22.026900    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52bf85ca-e35d-4a85-85f4-21d6d20a191d-workload-certs\") pod \"e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz\" (UID: \"52bf85ca-e35d-4a85-85f4-21d6d20a191d\") " pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz"
Oct 30 13:18:22.036623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:22.032433    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52bf85ca-e35d-4a85-85f4-21d6d20a191d-credential-socket\") pod \"e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz\" (UID: \"52bf85ca-e35d-4a85-85f4-21d6d20a191d\") " pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz"
Oct 30 13:18:22.062626 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:22.046706    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52bf85ca-e35d-4a85-85f4-21d6d20a191d-istio-podinfo\") pod \"e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz\" (UID: \"52bf85ca-e35d-4a85-85f4-21d6d20a191d\") " pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz"
Oct 30 13:18:22.078686 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:22.047646    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwf9s\" (UniqueName: \"kubernetes.io/projected/52bf85ca-e35d-4a85-85f4-21d6d20a191d-kube-api-access-lwf9s\") pod \"e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz\" (UID: \"52bf85ca-e35d-4a85-85f4-21d6d20a191d\") " pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz"
Oct 30 13:18:22.129622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:22.119925    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52bf85ca-e35d-4a85-85f4-21d6d20a191d-istio-envoy\") pod \"e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz\" (UID: \"52bf85ca-e35d-4a85-85f4-21d6d20a191d\") " pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz"
Oct 30 13:18:22.129622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:22.120358    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52bf85ca-e35d-4a85-85f4-21d6d20a191d-istio-token\") pod \"e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz\" (UID: \"52bf85ca-e35d-4a85-85f4-21d6d20a191d\") " pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz"
Oct 30 13:18:22.356525 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:22.356430    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz"
Oct 30 13:18:22.685227 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:22.684148    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz"]
Oct 30 13:18:22.687722 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:18:22.687691    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52bf85ca_e35d_4a85_85f4_21d6d20a191d.slice/crio-613e7dbf5fe99263df85ebd2519cbd4c7f6c7d3133bc542f1dccc85f54a11aaa WatchSource:0}: Error finding container 613e7dbf5fe99263df85ebd2519cbd4c7f6c7d3133bc542f1dccc85f54a11aaa: Status 404 returned error can't find the container with id 613e7dbf5fe99263df85ebd2519cbd4c7f6c7d3133bc542f1dccc85f54a11aaa
Oct 30 13:18:22.691103 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:22.691072    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:18:22.691191 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:22.691138    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:18:22.691191 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:22.691168    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:18:23.032828 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:23.032765    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-77254d-55f7f48fd6-twrk7"
Oct 30 13:18:23.123387 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:23.123344    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz" event={"ID":"52bf85ca-e35d-4a85-85f4-21d6d20a191d","Type":"ContainerStarted","Data":"383280d0ea2ed966b15a3e751091cf11b0522a50adeef0edab2295f885e8120d"}
Oct 30 13:18:23.123542 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:23.123417    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz" event={"ID":"52bf85ca-e35d-4a85-85f4-21d6d20a191d","Type":"ContainerStarted","Data":"613e7dbf5fe99263df85ebd2519cbd4c7f6c7d3133bc542f1dccc85f54a11aaa"}
Oct 30 13:18:23.174199 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:23.174141    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz" podStartSLOduration=2.174125537 podStartE2EDuration="2.174125537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:18:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:18:23.168264818 +0000 UTC m=+1602.334629466" watchObservedRunningTime="2026-10-30 13:18:23.174125537 +0000 UTC m=+1602.340490186"
Oct 30 13:18:23.367983 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:23.361814    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz"
Oct 30 13:18:23.388890 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:23.388859    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz"
Oct 30 13:18:24.130821 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:24.130792    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz"
Oct 30 13:18:24.131967 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:24.131948    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz"
Oct 30 13:18:25.654414 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:25.654358    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx"
Oct 30 13:18:25.734513 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:25.734480    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd6e6964-3e54-48f1-a184-8db6ef093a7b-kube-api-access-54qcz\" (UniqueName: \"kubernetes.io/projected/cd6e6964-3e54-48f1-a184-8db6ef093a7b-kube-api-access-54qcz\") pod \"cd6e6964-3e54-48f1-a184-8db6ef093a7b\" (UID: \"cd6e6964-3e54-48f1-a184-8db6ef093a7b\") "
Oct 30 13:18:25.737849 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:25.734539    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd6e6964-3e54-48f1-a184-8db6ef093a7b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd6e6964-3e54-48f1-a184-8db6ef093a7b-istio-data\") pod \"cd6e6964-3e54-48f1-a184-8db6ef093a7b\" (UID: \"cd6e6964-3e54-48f1-a184-8db6ef093a7b\") "
Oct 30 13:18:25.737849 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:25.734825    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd6e6964-3e54-48f1-a184-8db6ef093a7b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd6e6964-3e54-48f1-a184-8db6ef093a7b-workload-socket\") pod \"cd6e6964-3e54-48f1-a184-8db6ef093a7b\" (UID: \"cd6e6964-3e54-48f1-a184-8db6ef093a7b\") "
Oct 30 13:18:25.737849 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:25.734934    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd6e6964-3e54-48f1-a184-8db6ef093a7b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd6e6964-3e54-48f1-a184-8db6ef093a7b-istio-envoy\") pod \"cd6e6964-3e54-48f1-a184-8db6ef093a7b\" (UID: \"cd6e6964-3e54-48f1-a184-8db6ef093a7b\") "
Oct 30 13:18:25.737849 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:25.735002    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd6e6964-3e54-48f1-a184-8db6ef093a7b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd6e6964-3e54-48f1-a184-8db6ef093a7b-istiod-ca-cert\") pod \"cd6e6964-3e54-48f1-a184-8db6ef093a7b\" (UID: \"cd6e6964-3e54-48f1-a184-8db6ef093a7b\") "
Oct 30 13:18:25.737849 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:25.735212    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd6e6964-3e54-48f1-a184-8db6ef093a7b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd6e6964-3e54-48f1-a184-8db6ef093a7b-workload-certs\") pod \"cd6e6964-3e54-48f1-a184-8db6ef093a7b\" (UID: \"cd6e6964-3e54-48f1-a184-8db6ef093a7b\") "
Oct 30 13:18:25.737849 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:25.735413    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd6e6964-3e54-48f1-a184-8db6ef093a7b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd6e6964-3e54-48f1-a184-8db6ef093a7b-credential-socket\") pod \"cd6e6964-3e54-48f1-a184-8db6ef093a7b\" (UID: \"cd6e6964-3e54-48f1-a184-8db6ef093a7b\") "
Oct 30 13:18:25.737849 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:25.735536    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd6e6964-3e54-48f1-a184-8db6ef093a7b-istio-token\" (UniqueName: \"kubernetes.io/projected/cd6e6964-3e54-48f1-a184-8db6ef093a7b-istio-token\") pod \"cd6e6964-3e54-48f1-a184-8db6ef093a7b\" (UID: \"cd6e6964-3e54-48f1-a184-8db6ef093a7b\") "
Oct 30 13:18:25.737849 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:25.735677    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cd6e6964-3e54-48f1-a184-8db6ef093a7b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd6e6964-3e54-48f1-a184-8db6ef093a7b-istio-podinfo\") pod \"cd6e6964-3e54-48f1-a184-8db6ef093a7b\" (UID: \"cd6e6964-3e54-48f1-a184-8db6ef093a7b\") "
Oct 30 13:18:25.737849 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:25.736238    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd6e6964-3e54-48f1-a184-8db6ef093a7b-credential-socket" pod "cd6e6964-3e54-48f1-a184-8db6ef093a7b" (UID: "cd6e6964-3e54-48f1-a184-8db6ef093a7b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:18:25.737849 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:25.736297    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd6e6964-3e54-48f1-a184-8db6ef093a7b-workload-certs" pod "cd6e6964-3e54-48f1-a184-8db6ef093a7b" (UID: "cd6e6964-3e54-48f1-a184-8db6ef093a7b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:18:25.737849 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:25.736363    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd6e6964-3e54-48f1-a184-8db6ef093a7b-workload-socket" pod "cd6e6964-3e54-48f1-a184-8db6ef093a7b" (UID: "cd6e6964-3e54-48f1-a184-8db6ef093a7b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:18:25.737849 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:25.736561    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd6e6964-3e54-48f1-a184-8db6ef093a7b-istiod-ca-cert" pod "cd6e6964-3e54-48f1-a184-8db6ef093a7b" (UID: "cd6e6964-3e54-48f1-a184-8db6ef093a7b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:18:25.737849 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:25.736687    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd6e6964-3e54-48f1-a184-8db6ef093a7b-istio-data" pod "cd6e6964-3e54-48f1-a184-8db6ef093a7b" (UID: "cd6e6964-3e54-48f1-a184-8db6ef093a7b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:18:25.737849 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:25.736968    4364 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd6e6964-3e54-48f1-a184-8db6ef093a7b-workload-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:18:25.737849 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:25.736991    4364 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd6e6964-3e54-48f1-a184-8db6ef093a7b-credential-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:18:25.737849 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:25.737004    4364 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd6e6964-3e54-48f1-a184-8db6ef093a7b-istio-data\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:18:25.737849 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:25.737017    4364 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd6e6964-3e54-48f1-a184-8db6ef093a7b-workload-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:18:25.737849 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:25.737030    4364 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd6e6964-3e54-48f1-a184-8db6ef093a7b-istiod-ca-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:18:25.739317 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:25.739222    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd6e6964-3e54-48f1-a184-8db6ef093a7b-istio-token" pod "cd6e6964-3e54-48f1-a184-8db6ef093a7b" (UID: "cd6e6964-3e54-48f1-a184-8db6ef093a7b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:18:25.739384 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:25.739364    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd6e6964-3e54-48f1-a184-8db6ef093a7b-kube-api-access-54qcz" pod "cd6e6964-3e54-48f1-a184-8db6ef093a7b" (UID: "cd6e6964-3e54-48f1-a184-8db6ef093a7b"). InnerVolumeSpecName "kube-api-access-54qcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:18:25.740111 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:25.739985    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd6e6964-3e54-48f1-a184-8db6ef093a7b-istio-envoy" pod "cd6e6964-3e54-48f1-a184-8db6ef093a7b" (UID: "cd6e6964-3e54-48f1-a184-8db6ef093a7b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:18:25.740927 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:25.740820    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd6e6964-3e54-48f1-a184-8db6ef093a7b-istio-podinfo" pod "cd6e6964-3e54-48f1-a184-8db6ef093a7b" (UID: "cd6e6964-3e54-48f1-a184-8db6ef093a7b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:18:25.838157 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:25.838134    4364 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd6e6964-3e54-48f1-a184-8db6ef093a7b-istio-token\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:18:25.838157 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:25.838158    4364 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd6e6964-3e54-48f1-a184-8db6ef093a7b-istio-podinfo\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:18:25.838336 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:25.838167    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54qcz\" (UniqueName: \"kubernetes.io/projected/cd6e6964-3e54-48f1-a184-8db6ef093a7b-kube-api-access-54qcz\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:18:25.838336 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:25.838176    4364 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd6e6964-3e54-48f1-a184-8db6ef093a7b-istio-envoy\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:18:26.148259 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:26.148226    4364 generic.go:356] "Generic (PLEG): container finished" podID="cd6e6964-3e54-48f1-a184-8db6ef093a7b" containerID="841e79331eff502018036c927e124e95de3674c0570c3c1be13453a228aaf208" exitCode=0
Oct 30 13:18:26.148430 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:26.148275    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx"
Oct 30 13:18:26.148430 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:26.148313    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx" event={"ID":"cd6e6964-3e54-48f1-a184-8db6ef093a7b","Type":"ContainerDied","Data":"841e79331eff502018036c927e124e95de3674c0570c3c1be13453a228aaf208"}
Oct 30 13:18:26.148430 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:26.148367    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx" event={"ID":"cd6e6964-3e54-48f1-a184-8db6ef093a7b","Type":"ContainerDied","Data":"2e83e0b4ac590f5b7665a2d9a7bfaf600fa0842f2a66aca2cfa53086fa7b6574"}
Oct 30 13:18:26.148430 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:26.148389    4364 scope.go:122] "RemoveContainer" containerID="841e79331eff502018036c927e124e95de3674c0570c3c1be13453a228aaf208"
Oct 30 13:18:26.163174 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:26.163155    4364 scope.go:122] "RemoveContainer" containerID="5bcc210571b46ecb21eae031ebf90fca908e5b4d1d757005c3eec4e48ba9b6be"
Oct 30 13:18:26.174907 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:26.174890    4364 scope.go:122] "RemoveContainer" containerID="841e79331eff502018036c927e124e95de3674c0570c3c1be13453a228aaf208"
Oct 30 13:18:26.175157 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:18:26.175139    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"841e79331eff502018036c927e124e95de3674c0570c3c1be13453a228aaf208\": container with ID starting with 841e79331eff502018036c927e124e95de3674c0570c3c1be13453a228aaf208 not found: ID does not exist" containerID="841e79331eff502018036c927e124e95de3674c0570c3c1be13453a228aaf208"
Oct 30 13:18:26.175207 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:26.175167    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"841e79331eff502018036c927e124e95de3674c0570c3c1be13453a228aaf208"} err="failed to get container status \"841e79331eff502018036c927e124e95de3674c0570c3c1be13453a228aaf208\": rpc error: code = NotFound desc = could not find container \"841e79331eff502018036c927e124e95de3674c0570c3c1be13453a228aaf208\": container with ID starting with 841e79331eff502018036c927e124e95de3674c0570c3c1be13453a228aaf208 not found: ID does not exist"
Oct 30 13:18:26.175207 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:26.175182    4364 scope.go:122] "RemoveContainer" containerID="5bcc210571b46ecb21eae031ebf90fca908e5b4d1d757005c3eec4e48ba9b6be"
Oct 30 13:18:26.175422 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:18:26.175406    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bcc210571b46ecb21eae031ebf90fca908e5b4d1d757005c3eec4e48ba9b6be\": container with ID starting with 5bcc210571b46ecb21eae031ebf90fca908e5b4d1d757005c3eec4e48ba9b6be not found: ID does not exist" containerID="5bcc210571b46ecb21eae031ebf90fca908e5b4d1d757005c3eec4e48ba9b6be"
Oct 30 13:18:26.175460 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:26.175435    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bcc210571b46ecb21eae031ebf90fca908e5b4d1d757005c3eec4e48ba9b6be"} err="failed to get container status \"5bcc210571b46ecb21eae031ebf90fca908e5b4d1d757005c3eec4e48ba9b6be\": rpc error: code = NotFound desc = could not find container \"5bcc210571b46ecb21eae031ebf90fca908e5b4d1d757005c3eec4e48ba9b6be\": container with ID starting with 5bcc210571b46ecb21eae031ebf90fca908e5b4d1d757005c3eec4e48ba9b6be not found: ID does not exist"
Oct 30 13:18:26.217309 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:26.217286    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx"]
Oct 30 13:18:26.229652 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:26.229624    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-6b0881-openshift-default-65655644fc-sd7wx"]
Oct 30 13:18:26.972079 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:26.972052    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd6e6964-3e54-48f1-a184-8db6ef093a7b" path="/var/lib/kubelet/pods/cd6e6964-3e54-48f1-a184-8db6ef093a7b/volumes"
Oct 30 13:18:34.369505 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:34.369468    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9fcb69-6f45cd6b55-7z62k"]
Oct 30 13:18:34.370434 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:34.370385    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9fcb69-6f45cd6b55-7z62k" podUID="287c395c-95cf-4133-9319-f0a2c3ae60b6" containerName="maas-api" containerID="cri-o://40cc225669b3ca2888983b08e228ec1f5f0bbc8bf8270bbe1d72b373579005ac" gracePeriod=30
Oct 30 13:18:34.955641 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:34.952855    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9fcb69-6f45cd6b55-7z62k"
Oct 30 13:18:35.137621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:35.133538    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/287c395c-95cf-4133-9319-f0a2c3ae60b6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/287c395c-95cf-4133-9319-f0a2c3ae60b6-openshift-service-ca\") pod \"287c395c-95cf-4133-9319-f0a2c3ae60b6\" (UID: \"287c395c-95cf-4133-9319-f0a2c3ae60b6\") "
Oct 30 13:18:35.137621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:35.133637    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/287c395c-95cf-4133-9319-f0a2c3ae60b6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/287c395c-95cf-4133-9319-f0a2c3ae60b6-maas-api-tls\") pod \"287c395c-95cf-4133-9319-f0a2c3ae60b6\" (UID: \"287c395c-95cf-4133-9319-f0a2c3ae60b6\") "
Oct 30 13:18:35.137621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:35.133723    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/287c395c-95cf-4133-9319-f0a2c3ae60b6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/287c395c-95cf-4133-9319-f0a2c3ae60b6-metrics-certs\") pod \"287c395c-95cf-4133-9319-f0a2c3ae60b6\" (UID: \"287c395c-95cf-4133-9319-f0a2c3ae60b6\") "
Oct 30 13:18:35.137621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:35.133800    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/287c395c-95cf-4133-9319-f0a2c3ae60b6-kube-api-access-8n7xs\" (UniqueName: \"kubernetes.io/projected/287c395c-95cf-4133-9319-f0a2c3ae60b6-kube-api-access-8n7xs\") pod \"287c395c-95cf-4133-9319-f0a2c3ae60b6\" (UID: \"287c395c-95cf-4133-9319-f0a2c3ae60b6\") "
Oct 30 13:18:35.137621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:35.135149    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/287c395c-95cf-4133-9319-f0a2c3ae60b6-openshift-service-ca" pod "287c395c-95cf-4133-9319-f0a2c3ae60b6" (UID: "287c395c-95cf-4133-9319-f0a2c3ae60b6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:18:35.137621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:35.136750    4364 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/287c395c-95cf-4133-9319-f0a2c3ae60b6-openshift-service-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:18:35.140743 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:35.138948    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/287c395c-95cf-4133-9319-f0a2c3ae60b6-metrics-certs" pod "287c395c-95cf-4133-9319-f0a2c3ae60b6" (UID: "287c395c-95cf-4133-9319-f0a2c3ae60b6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:18:35.140743 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:35.139577    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/287c395c-95cf-4133-9319-f0a2c3ae60b6-kube-api-access-8n7xs" pod "287c395c-95cf-4133-9319-f0a2c3ae60b6" (UID: "287c395c-95cf-4133-9319-f0a2c3ae60b6"). InnerVolumeSpecName "kube-api-access-8n7xs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:18:35.142691 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:35.142662    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/287c395c-95cf-4133-9319-f0a2c3ae60b6-maas-api-tls" pod "287c395c-95cf-4133-9319-f0a2c3ae60b6" (UID: "287c395c-95cf-4133-9319-f0a2c3ae60b6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:18:35.241771 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:35.238333    4364 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/287c395c-95cf-4133-9319-f0a2c3ae60b6-metrics-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:18:35.241771 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:35.238374    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8n7xs\" (UniqueName: \"kubernetes.io/projected/287c395c-95cf-4133-9319-f0a2c3ae60b6-kube-api-access-8n7xs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:18:35.241771 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:35.238391    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/287c395c-95cf-4133-9319-f0a2c3ae60b6-maas-api-tls\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:18:35.270527 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:35.265860    4364 generic.go:356] "Generic (PLEG): container finished" podID="287c395c-95cf-4133-9319-f0a2c3ae60b6" containerID="40cc225669b3ca2888983b08e228ec1f5f0bbc8bf8270bbe1d72b373579005ac" exitCode=0
Oct 30 13:18:35.270527 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:35.265923    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9fcb69-6f45cd6b55-7z62k" event={"ID":"287c395c-95cf-4133-9319-f0a2c3ae60b6","Type":"ContainerDied","Data":"40cc225669b3ca2888983b08e228ec1f5f0bbc8bf8270bbe1d72b373579005ac"}
Oct 30 13:18:35.270527 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:35.265975    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9fcb69-6f45cd6b55-7z62k" event={"ID":"287c395c-95cf-4133-9319-f0a2c3ae60b6","Type":"ContainerDied","Data":"1c338424063abb80f185f2da4d48fded152550e40982c7a6c76063712e6cfbca"}
Oct 30 13:18:35.270527 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:35.266010    4364 scope.go:122] "RemoveContainer" containerID="40cc225669b3ca2888983b08e228ec1f5f0bbc8bf8270bbe1d72b373579005ac"
Oct 30 13:18:35.270527 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:35.266199    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9fcb69-6f45cd6b55-7z62k"
Oct 30 13:18:35.309851 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:35.304788    4364 scope.go:122] "RemoveContainer" containerID="40cc225669b3ca2888983b08e228ec1f5f0bbc8bf8270bbe1d72b373579005ac"
Oct 30 13:18:35.309851 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:18:35.305227    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40cc225669b3ca2888983b08e228ec1f5f0bbc8bf8270bbe1d72b373579005ac\": container with ID starting with 40cc225669b3ca2888983b08e228ec1f5f0bbc8bf8270bbe1d72b373579005ac not found: ID does not exist" containerID="40cc225669b3ca2888983b08e228ec1f5f0bbc8bf8270bbe1d72b373579005ac"
Oct 30 13:18:35.309851 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:35.305262    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40cc225669b3ca2888983b08e228ec1f5f0bbc8bf8270bbe1d72b373579005ac"} err="failed to get container status \"40cc225669b3ca2888983b08e228ec1f5f0bbc8bf8270bbe1d72b373579005ac\": rpc error: code = NotFound desc = could not find container \"40cc225669b3ca2888983b08e228ec1f5f0bbc8bf8270bbe1d72b373579005ac\": container with ID starting with 40cc225669b3ca2888983b08e228ec1f5f0bbc8bf8270bbe1d72b373579005ac not found: ID does not exist"
Oct 30 13:18:35.344584 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:35.344546    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9fcb69-6f45cd6b55-7z62k"]
Oct 30 13:18:35.349788 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:35.349758    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9fcb69-6f45cd6b55-7z62k"]
Oct 30 13:18:36.972139 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:36.972107    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="287c395c-95cf-4133-9319-f0a2c3ae60b6" path="/var/lib/kubelet/pods/287c395c-95cf-4133-9319-f0a2c3ae60b6/volumes"
Oct 30 13:18:37.735452 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:37.735411    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9fcb69-6f45cd6b55-7z62k" podUID="287c395c-95cf-4133-9319-f0a2c3ae60b6" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.152:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:18:41.574470 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:18:41.574418    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:18:45.526296 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:45.526253    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:18:47.120619 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:47.114648    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8af0fe-6c66bcb66d-lh75p"]
Oct 30 13:18:47.120619 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:47.114896    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8af0fe-6c66bcb66d-lh75p" podUID="f8e76a41-e80b-4b68-b18a-eae91fc341e6" containerName="maas-api" containerID="cri-o://b7c943a82c260bf763ee4b7dfe623b3f6c3a9a6248adc8b86f2e0e7a64d36013" gracePeriod=30
Oct 30 13:18:47.518797 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:47.518256    4364 generic.go:356] "Generic (PLEG): container finished" podID="f8e76a41-e80b-4b68-b18a-eae91fc341e6" containerID="b7c943a82c260bf763ee4b7dfe623b3f6c3a9a6248adc8b86f2e0e7a64d36013" exitCode=0
Oct 30 13:18:47.518797 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:47.518428    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8af0fe-6c66bcb66d-lh75p" event={"ID":"f8e76a41-e80b-4b68-b18a-eae91fc341e6","Type":"ContainerDied","Data":"b7c943a82c260bf763ee4b7dfe623b3f6c3a9a6248adc8b86f2e0e7a64d36013"}
Oct 30 13:18:47.917675 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:47.917642    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8af0fe-6c66bcb66d-lh75p"
Oct 30 13:18:47.994107 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:47.994075    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8e76a41-e80b-4b68-b18a-eae91fc341e6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8e76a41-e80b-4b68-b18a-eae91fc341e6-metrics-certs\") pod \"f8e76a41-e80b-4b68-b18a-eae91fc341e6\" (UID: \"f8e76a41-e80b-4b68-b18a-eae91fc341e6\") "
Oct 30 13:18:47.994387 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:47.994365    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8e76a41-e80b-4b68-b18a-eae91fc341e6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8e76a41-e80b-4b68-b18a-eae91fc341e6-maas-api-tls\") pod \"f8e76a41-e80b-4b68-b18a-eae91fc341e6\" (UID: \"f8e76a41-e80b-4b68-b18a-eae91fc341e6\") "
Oct 30 13:18:47.994559 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:47.994538    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8e76a41-e80b-4b68-b18a-eae91fc341e6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8e76a41-e80b-4b68-b18a-eae91fc341e6-openshift-service-ca\") pod \"f8e76a41-e80b-4b68-b18a-eae91fc341e6\" (UID: \"f8e76a41-e80b-4b68-b18a-eae91fc341e6\") "
Oct 30 13:18:47.994692 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:47.994677    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8e76a41-e80b-4b68-b18a-eae91fc341e6-kube-api-access-rntgj\" (UniqueName: \"kubernetes.io/projected/f8e76a41-e80b-4b68-b18a-eae91fc341e6-kube-api-access-rntgj\") pod \"f8e76a41-e80b-4b68-b18a-eae91fc341e6\" (UID: \"f8e76a41-e80b-4b68-b18a-eae91fc341e6\") "
Oct 30 13:18:47.995005 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:47.994977    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8e76a41-e80b-4b68-b18a-eae91fc341e6-openshift-service-ca" pod "f8e76a41-e80b-4b68-b18a-eae91fc341e6" (UID: "f8e76a41-e80b-4b68-b18a-eae91fc341e6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:18:47.995532 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:47.995509    4364 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8e76a41-e80b-4b68-b18a-eae91fc341e6-openshift-service-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:18:47.997998 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:47.997971    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8e76a41-e80b-4b68-b18a-eae91fc341e6-metrics-certs" pod "f8e76a41-e80b-4b68-b18a-eae91fc341e6" (UID: "f8e76a41-e80b-4b68-b18a-eae91fc341e6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:18:47.998118 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:47.998012    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8e76a41-e80b-4b68-b18a-eae91fc341e6-maas-api-tls" pod "f8e76a41-e80b-4b68-b18a-eae91fc341e6" (UID: "f8e76a41-e80b-4b68-b18a-eae91fc341e6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:18:47.998118 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:47.998065    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8e76a41-e80b-4b68-b18a-eae91fc341e6-kube-api-access-rntgj" pod "f8e76a41-e80b-4b68-b18a-eae91fc341e6" (UID: "f8e76a41-e80b-4b68-b18a-eae91fc341e6"). InnerVolumeSpecName "kube-api-access-rntgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:18:48.101354 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:48.099257    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8e76a41-e80b-4b68-b18a-eae91fc341e6-maas-api-tls\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:18:48.101354 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:48.099300    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rntgj\" (UniqueName: \"kubernetes.io/projected/f8e76a41-e80b-4b68-b18a-eae91fc341e6-kube-api-access-rntgj\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:18:48.101354 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:48.099315    4364 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8e76a41-e80b-4b68-b18a-eae91fc341e6-metrics-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:18:48.528258 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:48.528228    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8af0fe-6c66bcb66d-lh75p"
Oct 30 13:18:48.528704 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:48.528263    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8af0fe-6c66bcb66d-lh75p" event={"ID":"f8e76a41-e80b-4b68-b18a-eae91fc341e6","Type":"ContainerDied","Data":"23a4c45aa8b7cf7397cf102787adbe365c7cbf2597c72a9a5606ad00f8a7503b"}
Oct 30 13:18:48.528704 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:48.528330    4364 scope.go:122] "RemoveContainer" containerID="b7c943a82c260bf763ee4b7dfe623b3f6c3a9a6248adc8b86f2e0e7a64d36013"
Oct 30 13:18:48.584380 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:48.584360    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8af0fe-6c66bcb66d-lh75p"]
Oct 30 13:18:48.595398 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:48.595379    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8af0fe-6c66bcb66d-lh75p"]
Oct 30 13:18:48.972049 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:48.972015    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8e76a41-e80b-4b68-b18a-eae91fc341e6" path="/var/lib/kubelet/pods/f8e76a41-e80b-4b68-b18a-eae91fc341e6/volumes"
Oct 30 13:18:53.689397 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:53.689363    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-84942a-54bbf86548-25252"]
Oct 30 13:18:56.949256 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:56.949222    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd"]
Oct 30 13:18:56.950177 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:56.950154    4364 cpu_manager.go:414] "containerMap: removing container" podUID="f8e76a41-e80b-4b68-b18a-eae91fc341e6" containerName="maas-api"
Oct 30 13:18:56.950297 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:56.950180    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8e76a41-e80b-4b68-b18a-eae91fc341e6" containerName="maas-api"
Oct 30 13:18:56.950297 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:56.950206    4364 cpu_manager.go:414] "containerMap: removing container" podUID="287c395c-95cf-4133-9319-f0a2c3ae60b6" containerName="maas-api"
Oct 30 13:18:56.950297 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:56.950214    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="287c395c-95cf-4133-9319-f0a2c3ae60b6" containerName="maas-api"
Oct 30 13:18:56.950297 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:56.950268    4364 cpu_manager.go:414] "containerMap: removing container" podUID="cd6e6964-3e54-48f1-a184-8db6ef093a7b" containerName="istio-proxy"
Oct 30 13:18:56.950297 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:56.950276    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd6e6964-3e54-48f1-a184-8db6ef093a7b" containerName="istio-proxy"
Oct 30 13:18:56.950545 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:56.950301    4364 cpu_manager.go:414] "containerMap: removing container" podUID="cd6e6964-3e54-48f1-a184-8db6ef093a7b" containerName="istio-proxy"
Oct 30 13:18:56.950545 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:56.950309    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd6e6964-3e54-48f1-a184-8db6ef093a7b" containerName="istio-proxy"
Oct 30 13:18:56.950545 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:56.950520    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd6e6964-3e54-48f1-a184-8db6ef093a7b" containerName="istio-proxy"
Oct 30 13:18:56.950545 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:56.950541    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8e76a41-e80b-4b68-b18a-eae91fc341e6" containerName="maas-api"
Oct 30 13:18:56.950795 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:56.950555    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="287c395c-95cf-4133-9319-f0a2c3ae60b6" containerName="maas-api"
Oct 30 13:18:56.958623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:56.958555    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd"
Oct 30 13:18:56.985695 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:56.985659    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd"]
Oct 30 13:18:57.004825 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:57.004794    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f77c6bd-36f6-45cc-92ed-b22fa718700a-workload-socket\") pod \"e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd\" (UID: \"0f77c6bd-36f6-45cc-92ed-b22fa718700a\") " pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd"
Oct 30 13:18:57.005000 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:57.004868    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f77c6bd-36f6-45cc-92ed-b22fa718700a-credential-socket\") pod \"e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd\" (UID: \"0f77c6bd-36f6-45cc-92ed-b22fa718700a\") " pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd"
Oct 30 13:18:57.005000 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:57.004942    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f77c6bd-36f6-45cc-92ed-b22fa718700a-istio-token\") pod \"e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd\" (UID: \"0f77c6bd-36f6-45cc-92ed-b22fa718700a\") " pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd"
Oct 30 13:18:57.005000 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:57.004997    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f77c6bd-36f6-45cc-92ed-b22fa718700a-istio-podinfo\") pod \"e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd\" (UID: \"0f77c6bd-36f6-45cc-92ed-b22fa718700a\") " pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd"
Oct 30 13:18:57.005186 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:57.005040    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f77c6bd-36f6-45cc-92ed-b22fa718700a-istiod-ca-cert\") pod \"e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd\" (UID: \"0f77c6bd-36f6-45cc-92ed-b22fa718700a\") " pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd"
Oct 30 13:18:57.005186 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:57.005059    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f77c6bd-36f6-45cc-92ed-b22fa718700a-workload-certs\") pod \"e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd\" (UID: \"0f77c6bd-36f6-45cc-92ed-b22fa718700a\") " pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd"
Oct 30 13:18:57.005186 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:57.005094    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f77c6bd-36f6-45cc-92ed-b22fa718700a-istio-data\") pod \"e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd\" (UID: \"0f77c6bd-36f6-45cc-92ed-b22fa718700a\") " pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd"
Oct 30 13:18:57.005186 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:57.005168    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f77c6bd-36f6-45cc-92ed-b22fa718700a-istio-envoy\") pod \"e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd\" (UID: \"0f77c6bd-36f6-45cc-92ed-b22fa718700a\") " pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd"
Oct 30 13:18:57.005439 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:57.005201    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj29s\" (UniqueName: \"kubernetes.io/projected/0f77c6bd-36f6-45cc-92ed-b22fa718700a-kube-api-access-bj29s\") pod \"e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd\" (UID: \"0f77c6bd-36f6-45cc-92ed-b22fa718700a\") " pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd"
Oct 30 13:18:57.298673 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:57.297583    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-ddb271-openshift-default-dockercfg-7fnrt\""
Oct 30 13:18:57.330227 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:57.326154    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f77c6bd-36f6-45cc-92ed-b22fa718700a-istio-envoy\") pod \"e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd\" (UID: \"0f77c6bd-36f6-45cc-92ed-b22fa718700a\") " pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd"
Oct 30 13:18:57.330531 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:57.330502    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj29s\" (UniqueName: \"kubernetes.io/projected/0f77c6bd-36f6-45cc-92ed-b22fa718700a-kube-api-access-bj29s\") pod \"e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd\" (UID: \"0f77c6bd-36f6-45cc-92ed-b22fa718700a\") " pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd"
Oct 30 13:18:57.334579 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:57.334552    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f77c6bd-36f6-45cc-92ed-b22fa718700a-workload-socket\") pod \"e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd\" (UID: \"0f77c6bd-36f6-45cc-92ed-b22fa718700a\") " pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd"
Oct 30 13:18:57.341675 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:57.335036    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f77c6bd-36f6-45cc-92ed-b22fa718700a-workload-socket\") pod \"e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd\" (UID: \"0f77c6bd-36f6-45cc-92ed-b22fa718700a\") " pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd"
Oct 30 13:18:57.341675 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:57.335369    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f77c6bd-36f6-45cc-92ed-b22fa718700a-credential-socket\") pod \"e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd\" (UID: \"0f77c6bd-36f6-45cc-92ed-b22fa718700a\") " pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd"
Oct 30 13:18:57.341934 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:57.341913    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f77c6bd-36f6-45cc-92ed-b22fa718700a-credential-socket\") pod \"e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd\" (UID: \"0f77c6bd-36f6-45cc-92ed-b22fa718700a\") " pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd"
Oct 30 13:18:57.342098 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:57.341950    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f77c6bd-36f6-45cc-92ed-b22fa718700a-istio-envoy\") pod \"e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd\" (UID: \"0f77c6bd-36f6-45cc-92ed-b22fa718700a\") " pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd"
Oct 30 13:18:57.342287 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:57.342269    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f77c6bd-36f6-45cc-92ed-b22fa718700a-istio-token\") pod \"e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd\" (UID: \"0f77c6bd-36f6-45cc-92ed-b22fa718700a\") " pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd"
Oct 30 13:18:57.348865 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:57.348839    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f77c6bd-36f6-45cc-92ed-b22fa718700a-istio-podinfo\") pod \"e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd\" (UID: \"0f77c6bd-36f6-45cc-92ed-b22fa718700a\") " pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd"
Oct 30 13:18:57.360430 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:57.360406    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f77c6bd-36f6-45cc-92ed-b22fa718700a-istiod-ca-cert\") pod \"e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd\" (UID: \"0f77c6bd-36f6-45cc-92ed-b22fa718700a\") " pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd"
Oct 30 13:18:57.414076 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:57.407465    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f77c6bd-36f6-45cc-92ed-b22fa718700a-workload-certs\") pod \"e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd\" (UID: \"0f77c6bd-36f6-45cc-92ed-b22fa718700a\") " pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd"
Oct 30 13:18:57.414076 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:57.412805    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f77c6bd-36f6-45cc-92ed-b22fa718700a-istio-data\") pod \"e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd\" (UID: \"0f77c6bd-36f6-45cc-92ed-b22fa718700a\") " pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd"
Oct 30 13:18:57.414076 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:57.361280    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f77c6bd-36f6-45cc-92ed-b22fa718700a-istiod-ca-cert\") pod \"e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd\" (UID: \"0f77c6bd-36f6-45cc-92ed-b22fa718700a\") " pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd"
Oct 30 13:18:57.414076 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:57.408177    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f77c6bd-36f6-45cc-92ed-b22fa718700a-workload-certs\") pod \"e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd\" (UID: \"0f77c6bd-36f6-45cc-92ed-b22fa718700a\") " pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd"
Oct 30 13:18:57.414076 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:57.396571    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj29s\" (UniqueName: \"kubernetes.io/projected/0f77c6bd-36f6-45cc-92ed-b22fa718700a-kube-api-access-bj29s\") pod \"e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd\" (UID: \"0f77c6bd-36f6-45cc-92ed-b22fa718700a\") " pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd"
Oct 30 13:18:57.414986 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:57.414866    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f77c6bd-36f6-45cc-92ed-b22fa718700a-istio-data\") pod \"e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd\" (UID: \"0f77c6bd-36f6-45cc-92ed-b22fa718700a\") " pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd"
Oct 30 13:18:57.417916 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:57.417892    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f77c6bd-36f6-45cc-92ed-b22fa718700a-istio-podinfo\") pod \"e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd\" (UID: \"0f77c6bd-36f6-45cc-92ed-b22fa718700a\") " pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd"
Oct 30 13:18:57.430292 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:57.430263    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f77c6bd-36f6-45cc-92ed-b22fa718700a-istio-token\") pod \"e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd\" (UID: \"0f77c6bd-36f6-45cc-92ed-b22fa718700a\") " pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd"
Oct 30 13:18:57.578919 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:57.578834    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd"
Oct 30 13:18:58.065762 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:18:58.065730    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f77c6bd_36f6_45cc_92ed_b22fa718700a.slice/crio-1556a0ead9df68d09cf4d9504f6cb8b6b5485ceeccab6b25b1c88accaf98bf01 WatchSource:0}: Error finding container 1556a0ead9df68d09cf4d9504f6cb8b6b5485ceeccab6b25b1c88accaf98bf01: Status 404 returned error can't find the container with id 1556a0ead9df68d09cf4d9504f6cb8b6b5485ceeccab6b25b1c88accaf98bf01
Oct 30 13:18:58.067313 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:58.067282    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd"]
Oct 30 13:18:58.071285 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:58.071249    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:18:58.071409 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:58.071321    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:18:58.071409 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:58.071355    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:18:58.700059 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:58.700008    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-84942a-54bbf86548-25252" podUID="30144d1b-99d3-41fc-9ed9-bc1d18c60a4c" containerName="payload-processing" containerID="cri-o://17018eec7fad9e6a04fd065e76069262f2a990b76b74b30d03b110077c1d21ff" gracePeriod=25
Oct 30 13:18:58.721029 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:58.720990    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd" event={"ID":"0f77c6bd-36f6-45cc-92ed-b22fa718700a","Type":"ContainerStarted","Data":"844465097f0200cdee28e6245e758d4eccfe729bfcb546db88580734c0d748d6"}
Oct 30 13:18:58.721291 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:58.721264    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd" event={"ID":"0f77c6bd-36f6-45cc-92ed-b22fa718700a","Type":"ContainerStarted","Data":"1556a0ead9df68d09cf4d9504f6cb8b6b5485ceeccab6b25b1c88accaf98bf01"}
Oct 30 13:18:58.880622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:58.875740    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd" podStartSLOduration=2.875719074 podStartE2EDuration="2.875719074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:18:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:18:58.824579777 +0000 UTC m=+1637.990944426" watchObservedRunningTime="2026-10-30 13:18:58.875719074 +0000 UTC m=+1638.042083723"
Oct 30 13:18:59.295685 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:59.295650    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-84942a-54bbf86548-25252"
Oct 30 13:18:59.411106 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:59.411063    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30144d1b-99d3-41fc-9ed9-bc1d18c60a4c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30144d1b-99d3-41fc-9ed9-bc1d18c60a4c-plugins-config-volume\") pod \"30144d1b-99d3-41fc-9ed9-bc1d18c60a4c\" (UID: \"30144d1b-99d3-41fc-9ed9-bc1d18c60a4c\") "
Oct 30 13:18:59.411249 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:59.411213    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30144d1b-99d3-41fc-9ed9-bc1d18c60a4c-kube-api-access-8kgh9\" (UniqueName: \"kubernetes.io/projected/30144d1b-99d3-41fc-9ed9-bc1d18c60a4c-kube-api-access-8kgh9\") pod \"30144d1b-99d3-41fc-9ed9-bc1d18c60a4c\" (UID: \"30144d1b-99d3-41fc-9ed9-bc1d18c60a4c\") "
Oct 30 13:18:59.411439 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:59.411419    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30144d1b-99d3-41fc-9ed9-bc1d18c60a4c-plugins-config-volume" pod "30144d1b-99d3-41fc-9ed9-bc1d18c60a4c" (UID: "30144d1b-99d3-41fc-9ed9-bc1d18c60a4c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:18:59.411827 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:59.411804    4364 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30144d1b-99d3-41fc-9ed9-bc1d18c60a4c-plugins-config-volume\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:18:59.413238 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:59.413217    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30144d1b-99d3-41fc-9ed9-bc1d18c60a4c-kube-api-access-8kgh9" pod "30144d1b-99d3-41fc-9ed9-bc1d18c60a4c" (UID: "30144d1b-99d3-41fc-9ed9-bc1d18c60a4c"). InnerVolumeSpecName "kube-api-access-8kgh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:18:59.513058 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:59.513029    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kgh9\" (UniqueName: \"kubernetes.io/projected/30144d1b-99d3-41fc-9ed9-bc1d18c60a4c-kube-api-access-8kgh9\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:18:59.579251 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:59.579224    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd"
Oct 30 13:18:59.584940 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:59.584918    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd"
Oct 30 13:18:59.730475 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:59.730413    4364 generic.go:356] "Generic (PLEG): container finished" podID="30144d1b-99d3-41fc-9ed9-bc1d18c60a4c" containerID="17018eec7fad9e6a04fd065e76069262f2a990b76b74b30d03b110077c1d21ff" exitCode=0
Oct 30 13:18:59.730475 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:59.730460    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-84942a-54bbf86548-25252"
Oct 30 13:18:59.730674 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:59.730501    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-84942a-54bbf86548-25252" event={"ID":"30144d1b-99d3-41fc-9ed9-bc1d18c60a4c","Type":"ContainerDied","Data":"17018eec7fad9e6a04fd065e76069262f2a990b76b74b30d03b110077c1d21ff"}
Oct 30 13:18:59.730674 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:59.730553    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-84942a-54bbf86548-25252" event={"ID":"30144d1b-99d3-41fc-9ed9-bc1d18c60a4c","Type":"ContainerDied","Data":"e04b214c8ae937518926e947ed7b6a19d47b8945f067988026242c3e9b5d6b03"}
Oct 30 13:18:59.730674 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:59.730585    4364 scope.go:122] "RemoveContainer" containerID="17018eec7fad9e6a04fd065e76069262f2a990b76b74b30d03b110077c1d21ff"
Oct 30 13:18:59.730941 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:59.730917    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd"
Oct 30 13:18:59.731939 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:59.731810    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd"
Oct 30 13:18:59.742841 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:59.742824    4364 scope.go:122] "RemoveContainer" containerID="17018eec7fad9e6a04fd065e76069262f2a990b76b74b30d03b110077c1d21ff"
Oct 30 13:18:59.743122 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:18:59.743093    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17018eec7fad9e6a04fd065e76069262f2a990b76b74b30d03b110077c1d21ff\": container with ID starting with 17018eec7fad9e6a04fd065e76069262f2a990b76b74b30d03b110077c1d21ff not found: ID does not exist" containerID="17018eec7fad9e6a04fd065e76069262f2a990b76b74b30d03b110077c1d21ff"
Oct 30 13:18:59.743200 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:59.743132    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17018eec7fad9e6a04fd065e76069262f2a990b76b74b30d03b110077c1d21ff"} err="failed to get container status \"17018eec7fad9e6a04fd065e76069262f2a990b76b74b30d03b110077c1d21ff\": rpc error: code = NotFound desc = could not find container \"17018eec7fad9e6a04fd065e76069262f2a990b76b74b30d03b110077c1d21ff\": container with ID starting with 17018eec7fad9e6a04fd065e76069262f2a990b76b74b30d03b110077c1d21ff not found: ID does not exist"
Oct 30 13:18:59.784984 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:59.784962    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-84942a-54bbf86548-25252"]
Oct 30 13:18:59.791380 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:18:59.791359    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-84942a-54bbf86548-25252"]
Oct 30 13:19:01.047627 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:01.046379    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30144d1b-99d3-41fc-9ed9-bc1d18c60a4c" path="/var/lib/kubelet/pods/30144d1b-99d3-41fc-9ed9-bc1d18c60a4c/volumes"
Oct 30 13:19:03.243726 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:03.243636    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ddb271-789bc5d4c9-p97mb"]
Oct 30 13:19:03.244572 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:03.244554    4364 cpu_manager.go:414] "containerMap: removing container" podUID="30144d1b-99d3-41fc-9ed9-bc1d18c60a4c" containerName="payload-processing"
Oct 30 13:19:03.244572 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:03.244572    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30144d1b-99d3-41fc-9ed9-bc1d18c60a4c" containerName="payload-processing"
Oct 30 13:19:03.244730 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:03.244697    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="30144d1b-99d3-41fc-9ed9-bc1d18c60a4c" containerName="payload-processing"
Oct 30 13:19:03.244730 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:03.244713    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd6e6964-3e54-48f1-a184-8db6ef093a7b" containerName="istio-proxy"
Oct 30 13:19:03.250065 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:03.250046    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ddb271-789bc5d4c9-p97mb"
Oct 30 13:19:03.267052 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:03.267027    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ddb271-789bc5d4c9-p97mb"]
Oct 30 13:19:03.272107 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:03.272087    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-ddb271\""
Oct 30 13:19:03.327855 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:03.327822    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2bde24e-a303-4111-adf3-123d6df17c00-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ddb271-789bc5d4c9-p97mb\" (UID: \"b2bde24e-a303-4111-adf3-123d6df17c00\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ddb271-789bc5d4c9-p97mb"
Oct 30 13:19:03.328027 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:03.327892    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pls2t\" (UniqueName: \"kubernetes.io/projected/b2bde24e-a303-4111-adf3-123d6df17c00-kube-api-access-pls2t\") pod \"payload-pre-processing-e2e-inf-b-ddb271-789bc5d4c9-p97mb\" (UID: \"b2bde24e-a303-4111-adf3-123d6df17c00\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ddb271-789bc5d4c9-p97mb"
Oct 30 13:19:03.429866 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:03.429837    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2bde24e-a303-4111-adf3-123d6df17c00-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ddb271-789bc5d4c9-p97mb\" (UID: \"b2bde24e-a303-4111-adf3-123d6df17c00\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ddb271-789bc5d4c9-p97mb"
Oct 30 13:19:03.430017 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:03.429883    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pls2t\" (UniqueName: \"kubernetes.io/projected/b2bde24e-a303-4111-adf3-123d6df17c00-kube-api-access-pls2t\") pod \"payload-pre-processing-e2e-inf-b-ddb271-789bc5d4c9-p97mb\" (UID: \"b2bde24e-a303-4111-adf3-123d6df17c00\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ddb271-789bc5d4c9-p97mb"
Oct 30 13:19:03.430577 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:03.430552    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2bde24e-a303-4111-adf3-123d6df17c00-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ddb271-789bc5d4c9-p97mb\" (UID: \"b2bde24e-a303-4111-adf3-123d6df17c00\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ddb271-789bc5d4c9-p97mb"
Oct 30 13:19:03.509270 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:03.507288    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pls2t\" (UniqueName: \"kubernetes.io/projected/b2bde24e-a303-4111-adf3-123d6df17c00-kube-api-access-pls2t\") pod \"payload-pre-processing-e2e-inf-b-ddb271-789bc5d4c9-p97mb\" (UID: \"b2bde24e-a303-4111-adf3-123d6df17c00\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ddb271-789bc5d4c9-p97mb"
Oct 30 13:19:03.572665 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:03.570876    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ddb271-789bc5d4c9-p97mb"
Oct 30 13:19:04.095799 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:04.095768    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ddb271-789bc5d4c9-p97mb"]
Oct 30 13:19:04.096303 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:19:04.096279    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2bde24e_a303_4111_adf3_123d6df17c00.slice/crio-0c4a8b9aad98ba42d7e3586f03d202c6a01f552c3cea850a27ec0d2a18c6cfd0 WatchSource:0}: Error finding container 0c4a8b9aad98ba42d7e3586f03d202c6a01f552c3cea850a27ec0d2a18c6cfd0: Status 404 returned error can't find the container with id 0c4a8b9aad98ba42d7e3586f03d202c6a01f552c3cea850a27ec0d2a18c6cfd0
Oct 30 13:19:04.833079 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:04.833039    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ddb271-789bc5d4c9-p97mb" event={"ID":"b2bde24e-a303-4111-adf3-123d6df17c00","Type":"ContainerStarted","Data":"9852aa57b3462a6707f95d9d51449df20f381de671eb9faa42d6d79d66b0702b"}
Oct 30 13:19:04.833515 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:04.833103    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ddb271-789bc5d4c9-p97mb" event={"ID":"b2bde24e-a303-4111-adf3-123d6df17c00","Type":"ContainerStarted","Data":"0c4a8b9aad98ba42d7e3586f03d202c6a01f552c3cea850a27ec0d2a18c6cfd0"}
Oct 30 13:19:04.833515 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:04.833135    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ddb271-789bc5d4c9-p97mb"
Oct 30 13:19:04.879580 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:04.879537    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ddb271-789bc5d4c9-p97mb" podStartSLOduration=1.879524534 podStartE2EDuration="1.879524534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:19:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:19:04.874786759 +0000 UTC m=+1644.041151408" watchObservedRunningTime="2026-10-30 13:19:04.879524534 +0000 UTC m=+1644.045889181"
Oct 30 13:19:05.645179 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:05.645148    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n"]
Oct 30 13:19:05.921946 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:05.916155    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n" podUID="93fa52c8-91ad-40ef-a3b0-46c740175647" containerName="istio-proxy" containerID="cri-o://8842099cc4a6b23133e36bc48ce1c11bfc89522294efe950a46721e681ce1455" gracePeriod=30
Oct 30 13:19:06.648422 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:06.648385    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-31721c-6cc5477859-4pglh"]
Oct 30 13:19:06.652969 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:06.652948    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-31721c-6cc5477859-4pglh"
Oct 30 13:19:06.673694 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:06.673669    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-31721c-6cc5477859-4pglh"]
Oct 30 13:19:06.679119 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:06.679095    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-31721c-dockercfg-6ms8p\""
Oct 30 13:19:06.707341 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:06.707316    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-31721c\""
Oct 30 13:19:06.783624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:06.782729    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qm5m\" (UniqueName: \"kubernetes.io/projected/f9acff8a-e701-4ab6-ba2f-9c8484b7bdcb-kube-api-access-5qm5m\") pod \"payload-pre-processing-e2e-inf-a-31721c-6cc5477859-4pglh\" (UID: \"f9acff8a-e701-4ab6-ba2f-9c8484b7bdcb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-31721c-6cc5477859-4pglh"
Oct 30 13:19:06.794620 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:06.784816    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9acff8a-e701-4ab6-ba2f-9c8484b7bdcb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-31721c-6cc5477859-4pglh\" (UID: \"f9acff8a-e701-4ab6-ba2f-9c8484b7bdcb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-31721c-6cc5477859-4pglh"
Oct 30 13:19:07.057017 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:07.056919    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qm5m\" (UniqueName: \"kubernetes.io/projected/f9acff8a-e701-4ab6-ba2f-9c8484b7bdcb-kube-api-access-5qm5m\") pod \"payload-pre-processing-e2e-inf-a-31721c-6cc5477859-4pglh\" (UID: \"f9acff8a-e701-4ab6-ba2f-9c8484b7bdcb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-31721c-6cc5477859-4pglh"
Oct 30 13:19:07.057482 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:07.057141    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9acff8a-e701-4ab6-ba2f-9c8484b7bdcb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-31721c-6cc5477859-4pglh\" (UID: \"f9acff8a-e701-4ab6-ba2f-9c8484b7bdcb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-31721c-6cc5477859-4pglh"
Oct 30 13:19:07.059574 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:07.059533    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9acff8a-e701-4ab6-ba2f-9c8484b7bdcb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-31721c-6cc5477859-4pglh\" (UID: \"f9acff8a-e701-4ab6-ba2f-9c8484b7bdcb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-31721c-6cc5477859-4pglh"
Oct 30 13:19:07.090013 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:07.089981    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qm5m\" (UniqueName: \"kubernetes.io/projected/f9acff8a-e701-4ab6-ba2f-9c8484b7bdcb-kube-api-access-5qm5m\") pod \"payload-pre-processing-e2e-inf-a-31721c-6cc5477859-4pglh\" (UID: \"f9acff8a-e701-4ab6-ba2f-9c8484b7bdcb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-31721c-6cc5477859-4pglh"
Oct 30 13:19:07.271087 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:07.271062    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-31721c-6cc5477859-4pglh"
Oct 30 13:19:07.581503 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:07.581475    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-31721c-6cc5477859-4pglh"]
Oct 30 13:19:07.582064 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:19:07.582038    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9acff8a_e701_4ab6_ba2f_9c8484b7bdcb.slice/crio-55793fdb9b2b1a25a724900a99d843c1a36cd31a008c574bb3c53434662ea8c7 WatchSource:0}: Error finding container 55793fdb9b2b1a25a724900a99d843c1a36cd31a008c574bb3c53434662ea8c7: Status 404 returned error can't find the container with id 55793fdb9b2b1a25a724900a99d843c1a36cd31a008c574bb3c53434662ea8c7
Oct 30 13:19:08.089556 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:08.089517    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-31721c-6cc5477859-4pglh" event={"ID":"f9acff8a-e701-4ab6-ba2f-9c8484b7bdcb","Type":"ContainerStarted","Data":"724a012bc8418d5b787734379ce78cb0cf2836b453fc3c18c02d73d566878338"}
Oct 30 13:19:08.090007 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:08.089580    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-31721c-6cc5477859-4pglh"
Oct 30 13:19:08.090007 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:08.089618    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-31721c-6cc5477859-4pglh" event={"ID":"f9acff8a-e701-4ab6-ba2f-9c8484b7bdcb","Type":"ContainerStarted","Data":"55793fdb9b2b1a25a724900a99d843c1a36cd31a008c574bb3c53434662ea8c7"}
Oct 30 13:19:08.175536 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:08.175474    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-31721c-6cc5477859-4pglh" podStartSLOduration=2.175457553 podStartE2EDuration="2.175457553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:19:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:19:08.17059319 +0000 UTC m=+1647.336957838" watchObservedRunningTime="2026-10-30 13:19:08.175457553 +0000 UTC m=+1647.341822202"
Oct 30 13:19:08.678843 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:08.678810    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4644a1-74c746f55f-rkfmk"]
Oct 30 13:19:09.103007 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:09.102922    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2daeae-76cf799c89-c8sgx"]
Oct 30 13:19:09.103498 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:09.103143    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2daeae-76cf799c89-c8sgx" podUID="80cbfd1b-0a69-4fdb-880b-561f4bc6cd90" containerName="maas-api" containerID="cri-o://22bd6fcb36d43d58c0b38c39f53d8187b51a15136ce086d2afbdfa84f7cf71f1" gracePeriod=30
Oct 30 13:19:09.429319 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:09.429295    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2daeae-76cf799c89-c8sgx"
Oct 30 13:19:09.507921 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:09.507892    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80cbfd1b-0a69-4fdb-880b-561f4bc6cd90-metrics-certs\" (UniqueName: \"kubernetes.io/secret/80cbfd1b-0a69-4fdb-880b-561f4bc6cd90-metrics-certs\") pod \"80cbfd1b-0a69-4fdb-880b-561f4bc6cd90\" (UID: \"80cbfd1b-0a69-4fdb-880b-561f4bc6cd90\") "
Oct 30 13:19:09.508085 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:09.507933    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80cbfd1b-0a69-4fdb-880b-561f4bc6cd90-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80cbfd1b-0a69-4fdb-880b-561f4bc6cd90-maas-api-tls\") pod \"80cbfd1b-0a69-4fdb-880b-561f4bc6cd90\" (UID: \"80cbfd1b-0a69-4fdb-880b-561f4bc6cd90\") "
Oct 30 13:19:09.508085 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:09.508030    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80cbfd1b-0a69-4fdb-880b-561f4bc6cd90-kube-api-access-flsxc\" (UniqueName: \"kubernetes.io/projected/80cbfd1b-0a69-4fdb-880b-561f4bc6cd90-kube-api-access-flsxc\") pod \"80cbfd1b-0a69-4fdb-880b-561f4bc6cd90\" (UID: \"80cbfd1b-0a69-4fdb-880b-561f4bc6cd90\") "
Oct 30 13:19:09.508218 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:09.508162    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80cbfd1b-0a69-4fdb-880b-561f4bc6cd90-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80cbfd1b-0a69-4fdb-880b-561f4bc6cd90-openshift-service-ca\") pod \"80cbfd1b-0a69-4fdb-880b-561f4bc6cd90\" (UID: \"80cbfd1b-0a69-4fdb-880b-561f4bc6cd90\") "
Oct 30 13:19:09.508562 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:09.508531    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80cbfd1b-0a69-4fdb-880b-561f4bc6cd90-openshift-service-ca" pod "80cbfd1b-0a69-4fdb-880b-561f4bc6cd90" (UID: "80cbfd1b-0a69-4fdb-880b-561f4bc6cd90"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:19:09.509138 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:09.509112    4364 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80cbfd1b-0a69-4fdb-880b-561f4bc6cd90-openshift-service-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:19:09.510114 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:09.510087    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80cbfd1b-0a69-4fdb-880b-561f4bc6cd90-metrics-certs" pod "80cbfd1b-0a69-4fdb-880b-561f4bc6cd90" (UID: "80cbfd1b-0a69-4fdb-880b-561f4bc6cd90"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:19:09.510200 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:09.510154    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80cbfd1b-0a69-4fdb-880b-561f4bc6cd90-kube-api-access-flsxc" pod "80cbfd1b-0a69-4fdb-880b-561f4bc6cd90" (UID: "80cbfd1b-0a69-4fdb-880b-561f4bc6cd90"). InnerVolumeSpecName "kube-api-access-flsxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:19:09.510200 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:09.510162    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80cbfd1b-0a69-4fdb-880b-561f4bc6cd90-maas-api-tls" pod "80cbfd1b-0a69-4fdb-880b-561f4bc6cd90" (UID: "80cbfd1b-0a69-4fdb-880b-561f4bc6cd90"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:19:09.651715 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:09.642376    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flsxc\" (UniqueName: \"kubernetes.io/projected/80cbfd1b-0a69-4fdb-880b-561f4bc6cd90-kube-api-access-flsxc\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:19:09.651715 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:09.642419    4364 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80cbfd1b-0a69-4fdb-880b-561f4bc6cd90-metrics-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:19:09.651715 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:09.642435    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80cbfd1b-0a69-4fdb-880b-561f4bc6cd90-maas-api-tls\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:19:09.813776 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:09.813744    4364 patch_prober.go:29] interesting pod/e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:19:09.814085 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:09.814044    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n" podUID="93fa52c8-91ad-40ef-a3b0-46c740175647" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:19:10.125256 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:10.125218    4364 generic.go:356] "Generic (PLEG): container finished" podID="80cbfd1b-0a69-4fdb-880b-561f4bc6cd90" containerID="22bd6fcb36d43d58c0b38c39f53d8187b51a15136ce086d2afbdfa84f7cf71f1" exitCode=0
Oct 30 13:19:10.125780 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:10.125372    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2daeae-76cf799c89-c8sgx" event={"ID":"80cbfd1b-0a69-4fdb-880b-561f4bc6cd90","Type":"ContainerDied","Data":"22bd6fcb36d43d58c0b38c39f53d8187b51a15136ce086d2afbdfa84f7cf71f1"}
Oct 30 13:19:10.125780 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:10.125418    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2daeae-76cf799c89-c8sgx" event={"ID":"80cbfd1b-0a69-4fdb-880b-561f4bc6cd90","Type":"ContainerDied","Data":"89ce86cb66c725d68fa3be87e1ceff28dc20e7be3083bbf297dbc4a5f9e651af"}
Oct 30 13:19:10.125780 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:10.125443    4364 scope.go:122] "RemoveContainer" containerID="22bd6fcb36d43d58c0b38c39f53d8187b51a15136ce086d2afbdfa84f7cf71f1"
Oct 30 13:19:10.125780 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:10.125593    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2daeae-76cf799c89-c8sgx"
Oct 30 13:19:10.151474 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:10.143771    4364 scope.go:122] "RemoveContainer" containerID="22bd6fcb36d43d58c0b38c39f53d8187b51a15136ce086d2afbdfa84f7cf71f1"
Oct 30 13:19:10.151474 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:19:10.145261    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22bd6fcb36d43d58c0b38c39f53d8187b51a15136ce086d2afbdfa84f7cf71f1\": container with ID starting with 22bd6fcb36d43d58c0b38c39f53d8187b51a15136ce086d2afbdfa84f7cf71f1 not found: ID does not exist" containerID="22bd6fcb36d43d58c0b38c39f53d8187b51a15136ce086d2afbdfa84f7cf71f1"
Oct 30 13:19:10.151474 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:10.145302    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22bd6fcb36d43d58c0b38c39f53d8187b51a15136ce086d2afbdfa84f7cf71f1"} err="failed to get container status \"22bd6fcb36d43d58c0b38c39f53d8187b51a15136ce086d2afbdfa84f7cf71f1\": rpc error: code = NotFound desc = could not find container \"22bd6fcb36d43d58c0b38c39f53d8187b51a15136ce086d2afbdfa84f7cf71f1\": container with ID starting with 22bd6fcb36d43d58c0b38c39f53d8187b51a15136ce086d2afbdfa84f7cf71f1 not found: ID does not exist"
Oct 30 13:19:10.217268 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:10.215550    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2daeae-76cf799c89-c8sgx"]
Oct 30 13:19:10.220478 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:10.220447    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2daeae-76cf799c89-c8sgx"]
Oct 30 13:19:11.012076 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:11.012025    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80cbfd1b-0a69-4fdb-880b-561f4bc6cd90" path="/var/lib/kubelet/pods/80cbfd1b-0a69-4fdb-880b-561f4bc6cd90/volumes"
Oct 30 13:19:11.136249 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:11.136217    4364 generic.go:356] "Generic (PLEG): container finished" podID="93fa52c8-91ad-40ef-a3b0-46c740175647" containerID="8842099cc4a6b23133e36bc48ce1c11bfc89522294efe950a46721e681ce1455" exitCode=0
Oct 30 13:19:11.136918 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:11.136651    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n" event={"ID":"93fa52c8-91ad-40ef-a3b0-46c740175647","Type":"ContainerDied","Data":"8842099cc4a6b23133e36bc48ce1c11bfc89522294efe950a46721e681ce1455"}
Oct 30 13:19:11.322335 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:11.322313    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n"
Oct 30 13:19:11.522868 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:11.522792    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93fa52c8-91ad-40ef-a3b0-46c740175647-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93fa52c8-91ad-40ef-a3b0-46c740175647-credential-socket\") pod \"93fa52c8-91ad-40ef-a3b0-46c740175647\" (UID: \"93fa52c8-91ad-40ef-a3b0-46c740175647\") "
Oct 30 13:19:11.523015 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:11.522874    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93fa52c8-91ad-40ef-a3b0-46c740175647-istio-token\" (UniqueName: \"kubernetes.io/projected/93fa52c8-91ad-40ef-a3b0-46c740175647-istio-token\") pod \"93fa52c8-91ad-40ef-a3b0-46c740175647\" (UID: \"93fa52c8-91ad-40ef-a3b0-46c740175647\") "
Oct 30 13:19:11.523015 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:11.522965    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93fa52c8-91ad-40ef-a3b0-46c740175647-kube-api-access-8qskw\" (UniqueName: \"kubernetes.io/projected/93fa52c8-91ad-40ef-a3b0-46c740175647-kube-api-access-8qskw\") pod \"93fa52c8-91ad-40ef-a3b0-46c740175647\" (UID: \"93fa52c8-91ad-40ef-a3b0-46c740175647\") "
Oct 30 13:19:11.523015 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:11.522997    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93fa52c8-91ad-40ef-a3b0-46c740175647-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93fa52c8-91ad-40ef-a3b0-46c740175647-istiod-ca-cert\") pod \"93fa52c8-91ad-40ef-a3b0-46c740175647\" (UID: \"93fa52c8-91ad-40ef-a3b0-46c740175647\") "
Oct 30 13:19:11.523178 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:11.523036    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93fa52c8-91ad-40ef-a3b0-46c740175647-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93fa52c8-91ad-40ef-a3b0-46c740175647-istio-envoy\") pod \"93fa52c8-91ad-40ef-a3b0-46c740175647\" (UID: \"93fa52c8-91ad-40ef-a3b0-46c740175647\") "
Oct 30 13:19:11.523178 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:11.523082    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93fa52c8-91ad-40ef-a3b0-46c740175647-credential-socket" pod "93fa52c8-91ad-40ef-a3b0-46c740175647" (UID: "93fa52c8-91ad-40ef-a3b0-46c740175647"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:19:11.523178 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:11.523169    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93fa52c8-91ad-40ef-a3b0-46c740175647-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93fa52c8-91ad-40ef-a3b0-46c740175647-workload-certs\") pod \"93fa52c8-91ad-40ef-a3b0-46c740175647\" (UID: \"93fa52c8-91ad-40ef-a3b0-46c740175647\") "
Oct 30 13:19:11.523345 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:11.523235    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93fa52c8-91ad-40ef-a3b0-46c740175647-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93fa52c8-91ad-40ef-a3b0-46c740175647-istio-data\") pod \"93fa52c8-91ad-40ef-a3b0-46c740175647\" (UID: \"93fa52c8-91ad-40ef-a3b0-46c740175647\") "
Oct 30 13:19:11.523428 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:11.523406    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/93fa52c8-91ad-40ef-a3b0-46c740175647-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93fa52c8-91ad-40ef-a3b0-46c740175647-istio-podinfo\") pod \"93fa52c8-91ad-40ef-a3b0-46c740175647\" (UID: \"93fa52c8-91ad-40ef-a3b0-46c740175647\") "
Oct 30 13:19:11.523485 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:11.523475    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93fa52c8-91ad-40ef-a3b0-46c740175647-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93fa52c8-91ad-40ef-a3b0-46c740175647-workload-socket\") pod \"93fa52c8-91ad-40ef-a3b0-46c740175647\" (UID: \"93fa52c8-91ad-40ef-a3b0-46c740175647\") "
Oct 30 13:19:11.523543 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:11.523417    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93fa52c8-91ad-40ef-a3b0-46c740175647-istiod-ca-cert" pod "93fa52c8-91ad-40ef-a3b0-46c740175647" (UID: "93fa52c8-91ad-40ef-a3b0-46c740175647"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:19:11.523543 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:11.523513    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93fa52c8-91ad-40ef-a3b0-46c740175647-workload-certs" pod "93fa52c8-91ad-40ef-a3b0-46c740175647" (UID: "93fa52c8-91ad-40ef-a3b0-46c740175647"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:19:11.523826 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:11.523805    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93fa52c8-91ad-40ef-a3b0-46c740175647-workload-socket" pod "93fa52c8-91ad-40ef-a3b0-46c740175647" (UID: "93fa52c8-91ad-40ef-a3b0-46c740175647"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:19:11.524027 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:11.523999    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93fa52c8-91ad-40ef-a3b0-46c740175647-istio-data" pod "93fa52c8-91ad-40ef-a3b0-46c740175647" (UID: "93fa52c8-91ad-40ef-a3b0-46c740175647"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:19:11.524358 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:11.524336    4364 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93fa52c8-91ad-40ef-a3b0-46c740175647-workload-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:19:11.524459 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:11.524363    4364 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93fa52c8-91ad-40ef-a3b0-46c740175647-credential-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:19:11.524459 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:11.524380    4364 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93fa52c8-91ad-40ef-a3b0-46c740175647-istiod-ca-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:19:11.524459 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:11.524394    4364 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93fa52c8-91ad-40ef-a3b0-46c740175647-workload-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:19:11.524459 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:11.524407    4364 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93fa52c8-91ad-40ef-a3b0-46c740175647-istio-data\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:19:11.525479 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:11.525454    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93fa52c8-91ad-40ef-a3b0-46c740175647-kube-api-access-8qskw" pod "93fa52c8-91ad-40ef-a3b0-46c740175647" (UID: "93fa52c8-91ad-40ef-a3b0-46c740175647"). InnerVolumeSpecName "kube-api-access-8qskw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:19:11.525657 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:11.525633    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93fa52c8-91ad-40ef-a3b0-46c740175647-istio-token" pod "93fa52c8-91ad-40ef-a3b0-46c740175647" (UID: "93fa52c8-91ad-40ef-a3b0-46c740175647"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:19:11.525738 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:11.525708    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93fa52c8-91ad-40ef-a3b0-46c740175647-istio-envoy" pod "93fa52c8-91ad-40ef-a3b0-46c740175647" (UID: "93fa52c8-91ad-40ef-a3b0-46c740175647"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:19:11.525816 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:11.525788    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/93fa52c8-91ad-40ef-a3b0-46c740175647-istio-podinfo" pod "93fa52c8-91ad-40ef-a3b0-46c740175647" (UID: "93fa52c8-91ad-40ef-a3b0-46c740175647"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:19:11.531154 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:19:11.531129    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:19:11.625523 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:11.625495    4364 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93fa52c8-91ad-40ef-a3b0-46c740175647-istio-podinfo\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:19:11.625523 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:11.625522    4364 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93fa52c8-91ad-40ef-a3b0-46c740175647-istio-token\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:19:11.625717 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:11.625531    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qskw\" (UniqueName: \"kubernetes.io/projected/93fa52c8-91ad-40ef-a3b0-46c740175647-kube-api-access-8qskw\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:19:11.625717 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:11.625540    4364 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93fa52c8-91ad-40ef-a3b0-46c740175647-istio-envoy\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:19:12.147808 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:12.147780    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n"
Oct 30 13:19:12.148261 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:12.147782    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n" event={"ID":"93fa52c8-91ad-40ef-a3b0-46c740175647","Type":"ContainerDied","Data":"adc70631b7d9be779aad9e349f1ee5db3843836aa2136a9e7a81ebcd51e5a586"}
Oct 30 13:19:12.148261 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:12.147916    4364 scope.go:122] "RemoveContainer" containerID="8842099cc4a6b23133e36bc48ce1c11bfc89522294efe950a46721e681ce1455"
Oct 30 13:19:12.267625 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:12.260831    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n"]
Oct 30 13:19:12.271030 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:12.269151    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-df6c3661-openshift-default-578b8967cc-z7f2n"]
Oct 30 13:19:12.972659 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:12.972620    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93fa52c8-91ad-40ef-a3b0-46c740175647" path="/var/lib/kubelet/pods/93fa52c8-91ad-40ef-a3b0-46c740175647/volumes"
Oct 30 13:19:13.699623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:13.698654    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4644a1-74c746f55f-rkfmk" podUID="ad81d931-2563-49a1-aebe-a7f0f29cda94" containerName="payload-pre-processing" containerID="cri-o://e9b0fba95ed493f6476f97398d85c7c3f4b5375db66321e1857e9f4c0f75626e" gracePeriod=25
Oct 30 13:19:14.062670 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:14.062644    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4644a1-74c746f55f-rkfmk"
Oct 30 13:19:14.161544 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:14.161515    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad81d931-2563-49a1-aebe-a7f0f29cda94-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad81d931-2563-49a1-aebe-a7f0f29cda94-plugins-config-volume\") pod \"ad81d931-2563-49a1-aebe-a7f0f29cda94\" (UID: \"ad81d931-2563-49a1-aebe-a7f0f29cda94\") "
Oct 30 13:19:14.161739 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:14.161665    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad81d931-2563-49a1-aebe-a7f0f29cda94-kube-api-access-kn92k\" (UniqueName: \"kubernetes.io/projected/ad81d931-2563-49a1-aebe-a7f0f29cda94-kube-api-access-kn92k\") pod \"ad81d931-2563-49a1-aebe-a7f0f29cda94\" (UID: \"ad81d931-2563-49a1-aebe-a7f0f29cda94\") "
Oct 30 13:19:14.161961 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:14.161936    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad81d931-2563-49a1-aebe-a7f0f29cda94-plugins-config-volume" pod "ad81d931-2563-49a1-aebe-a7f0f29cda94" (UID: "ad81d931-2563-49a1-aebe-a7f0f29cda94"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:19:14.162462 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:14.162439    4364 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad81d931-2563-49a1-aebe-a7f0f29cda94-plugins-config-volume\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:19:14.164797 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:14.164767    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad81d931-2563-49a1-aebe-a7f0f29cda94-kube-api-access-kn92k" pod "ad81d931-2563-49a1-aebe-a7f0f29cda94" (UID: "ad81d931-2563-49a1-aebe-a7f0f29cda94"). InnerVolumeSpecName "kube-api-access-kn92k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:19:14.168653 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:14.168629    4364 generic.go:356] "Generic (PLEG): container finished" podID="ad81d931-2563-49a1-aebe-a7f0f29cda94" containerID="e9b0fba95ed493f6476f97398d85c7c3f4b5375db66321e1857e9f4c0f75626e" exitCode=0
Oct 30 13:19:14.168763 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:14.168665    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4644a1-74c746f55f-rkfmk" event={"ID":"ad81d931-2563-49a1-aebe-a7f0f29cda94","Type":"ContainerDied","Data":"e9b0fba95ed493f6476f97398d85c7c3f4b5375db66321e1857e9f4c0f75626e"}
Oct 30 13:19:14.168763 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:14.168738    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4644a1-74c746f55f-rkfmk" event={"ID":"ad81d931-2563-49a1-aebe-a7f0f29cda94","Type":"ContainerDied","Data":"86f3a087716d3e234010f495a2feba469e5af0841ae58c536ddd699eed7dcc1b"}
Oct 30 13:19:14.168763 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:14.168677    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4644a1-74c746f55f-rkfmk"
Oct 30 13:19:14.168933 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:14.168770    4364 scope.go:122] "RemoveContainer" containerID="e9b0fba95ed493f6476f97398d85c7c3f4b5375db66321e1857e9f4c0f75626e"
Oct 30 13:19:14.257611 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:14.239801    4364 scope.go:122] "RemoveContainer" containerID="e9b0fba95ed493f6476f97398d85c7c3f4b5375db66321e1857e9f4c0f75626e"
Oct 30 13:19:14.257611 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:19:14.241354    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9b0fba95ed493f6476f97398d85c7c3f4b5375db66321e1857e9f4c0f75626e\": container with ID starting with e9b0fba95ed493f6476f97398d85c7c3f4b5375db66321e1857e9f4c0f75626e not found: ID does not exist" containerID="e9b0fba95ed493f6476f97398d85c7c3f4b5375db66321e1857e9f4c0f75626e"
Oct 30 13:19:14.257611 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:14.241397    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9b0fba95ed493f6476f97398d85c7c3f4b5375db66321e1857e9f4c0f75626e"} err="failed to get container status \"e9b0fba95ed493f6476f97398d85c7c3f4b5375db66321e1857e9f4c0f75626e\": rpc error: code = NotFound desc = could not find container \"e9b0fba95ed493f6476f97398d85c7c3f4b5375db66321e1857e9f4c0f75626e\": container with ID starting with e9b0fba95ed493f6476f97398d85c7c3f4b5375db66321e1857e9f4c0f75626e not found: ID does not exist"
Oct 30 13:19:14.270680 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:14.268267    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kn92k\" (UniqueName: \"kubernetes.io/projected/ad81d931-2563-49a1-aebe-a7f0f29cda94-kube-api-access-kn92k\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:19:14.316667 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:14.315322    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4644a1-74c746f55f-rkfmk"]
Oct 30 13:19:14.348782 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:14.348739    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4644a1-74c746f55f-rkfmk"]
Oct 30 13:19:14.989773 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:14.989727    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad81d931-2563-49a1-aebe-a7f0f29cda94" path="/var/lib/kubelet/pods/ad81d931-2563-49a1-aebe-a7f0f29cda94/volumes"
Oct 30 13:19:15.920913 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:15.920883    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ddb271-789bc5d4c9-p97mb"
Oct 30 13:19:17.904664 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:17.900882    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm"]
Oct 30 13:19:17.904664 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:17.901126    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm" podUID="9ef21594-2f1f-4b43-9c5f-529c9f9763f7" containerName="istio-proxy" containerID="cri-o://800d907b4ccaf08e8c7e840e40094698c1c300512259106b4b43ee082b51f297" gracePeriod=30
Oct 30 13:19:18.499556 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:18.499525    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2"]
Oct 30 13:19:18.500673 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:18.500648    4364 cpu_manager.go:414] "containerMap: removing container" podUID="80cbfd1b-0a69-4fdb-880b-561f4bc6cd90" containerName="maas-api"
Oct 30 13:19:18.500673 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:18.500671    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80cbfd1b-0a69-4fdb-880b-561f4bc6cd90" containerName="maas-api"
Oct 30 13:19:18.500866 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:18.500686    4364 cpu_manager.go:414] "containerMap: removing container" podUID="ad81d931-2563-49a1-aebe-a7f0f29cda94" containerName="payload-pre-processing"
Oct 30 13:19:18.500866 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:18.500694    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad81d931-2563-49a1-aebe-a7f0f29cda94" containerName="payload-pre-processing"
Oct 30 13:19:18.500866 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:18.500730    4364 cpu_manager.go:414] "containerMap: removing container" podUID="93fa52c8-91ad-40ef-a3b0-46c740175647" containerName="istio-proxy"
Oct 30 13:19:18.500866 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:18.500738    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93fa52c8-91ad-40ef-a3b0-46c740175647" containerName="istio-proxy"
Oct 30 13:19:18.501069 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:18.500907    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="80cbfd1b-0a69-4fdb-880b-561f4bc6cd90" containerName="maas-api"
Oct 30 13:19:18.501069 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:18.500921    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad81d931-2563-49a1-aebe-a7f0f29cda94" containerName="payload-pre-processing"
Oct 30 13:19:18.501069 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:18.500938    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="93fa52c8-91ad-40ef-a3b0-46c740175647" containerName="istio-proxy"
Oct 30 13:19:18.506682 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:18.506654    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2"
Oct 30 13:19:18.527634 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:18.527580    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2"]
Oct 30 13:19:18.533642 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:18.532373    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-ff4285-openshift-default-dockercfg-4t4rj\""
Oct 30 13:19:18.605869 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:18.605840    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd17aa36-e0ae-4652-a85f-20945233ccc0-workload-socket\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2\" (UID: \"fd17aa36-e0ae-4652-a85f-20945233ccc0\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2"
Oct 30 13:19:18.606049 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:18.605907    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd17aa36-e0ae-4652-a85f-20945233ccc0-istio-data\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2\" (UID: \"fd17aa36-e0ae-4652-a85f-20945233ccc0\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2"
Oct 30 13:19:18.606049 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:18.606011    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8bnk\" (UniqueName: \"kubernetes.io/projected/fd17aa36-e0ae-4652-a85f-20945233ccc0-kube-api-access-d8bnk\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2\" (UID: \"fd17aa36-e0ae-4652-a85f-20945233ccc0\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2"
Oct 30 13:19:18.606211 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:18.606194    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd17aa36-e0ae-4652-a85f-20945233ccc0-istiod-ca-cert\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2\" (UID: \"fd17aa36-e0ae-4652-a85f-20945233ccc0\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2"
Oct 30 13:19:18.606327 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:18.606309    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd17aa36-e0ae-4652-a85f-20945233ccc0-istio-envoy\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2\" (UID: \"fd17aa36-e0ae-4652-a85f-20945233ccc0\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2"
Oct 30 13:19:18.606384 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:18.606370    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd17aa36-e0ae-4652-a85f-20945233ccc0-istio-token\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2\" (UID: \"fd17aa36-e0ae-4652-a85f-20945233ccc0\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2"
Oct 30 13:19:18.606428 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:18.606412    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd17aa36-e0ae-4652-a85f-20945233ccc0-credential-socket\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2\" (UID: \"fd17aa36-e0ae-4652-a85f-20945233ccc0\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2"
Oct 30 13:19:18.606497 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:18.606485    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd17aa36-e0ae-4652-a85f-20945233ccc0-workload-certs\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2\" (UID: \"fd17aa36-e0ae-4652-a85f-20945233ccc0\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2"
Oct 30 13:19:18.606533 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:18.606524    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd17aa36-e0ae-4652-a85f-20945233ccc0-istio-podinfo\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2\" (UID: \"fd17aa36-e0ae-4652-a85f-20945233ccc0\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2"
Oct 30 13:19:18.726622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:18.722341    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd17aa36-e0ae-4652-a85f-20945233ccc0-istiod-ca-cert\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2\" (UID: \"fd17aa36-e0ae-4652-a85f-20945233ccc0\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2"
Oct 30 13:19:18.726622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:18.722453    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd17aa36-e0ae-4652-a85f-20945233ccc0-istio-envoy\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2\" (UID: \"fd17aa36-e0ae-4652-a85f-20945233ccc0\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2"
Oct 30 13:19:18.726622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:18.722528    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd17aa36-e0ae-4652-a85f-20945233ccc0-istio-token\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2\" (UID: \"fd17aa36-e0ae-4652-a85f-20945233ccc0\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2"
Oct 30 13:19:18.726622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:18.722576    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd17aa36-e0ae-4652-a85f-20945233ccc0-credential-socket\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2\" (UID: \"fd17aa36-e0ae-4652-a85f-20945233ccc0\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2"
Oct 30 13:19:18.726622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:18.722635    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd17aa36-e0ae-4652-a85f-20945233ccc0-workload-certs\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2\" (UID: \"fd17aa36-e0ae-4652-a85f-20945233ccc0\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2"
Oct 30 13:19:18.726622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:18.722703    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd17aa36-e0ae-4652-a85f-20945233ccc0-istio-podinfo\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2\" (UID: \"fd17aa36-e0ae-4652-a85f-20945233ccc0\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2"
Oct 30 13:19:18.726622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:18.722823    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd17aa36-e0ae-4652-a85f-20945233ccc0-workload-socket\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2\" (UID: \"fd17aa36-e0ae-4652-a85f-20945233ccc0\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2"
Oct 30 13:19:18.726622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:18.722880    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd17aa36-e0ae-4652-a85f-20945233ccc0-istio-data\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2\" (UID: \"fd17aa36-e0ae-4652-a85f-20945233ccc0\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2"
Oct 30 13:19:18.726622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:18.722925    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8bnk\" (UniqueName: \"kubernetes.io/projected/fd17aa36-e0ae-4652-a85f-20945233ccc0-kube-api-access-d8bnk\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2\" (UID: \"fd17aa36-e0ae-4652-a85f-20945233ccc0\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2"
Oct 30 13:19:18.733737 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:18.733291    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd17aa36-e0ae-4652-a85f-20945233ccc0-workload-certs\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2\" (UID: \"fd17aa36-e0ae-4652-a85f-20945233ccc0\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2"
Oct 30 13:19:18.734990 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:18.734568    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd17aa36-e0ae-4652-a85f-20945233ccc0-workload-socket\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2\" (UID: \"fd17aa36-e0ae-4652-a85f-20945233ccc0\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2"
Oct 30 13:19:18.742112 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:18.742072    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd17aa36-e0ae-4652-a85f-20945233ccc0-istio-data\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2\" (UID: \"fd17aa36-e0ae-4652-a85f-20945233ccc0\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2"
Oct 30 13:19:18.742697 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:18.742667    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd17aa36-e0ae-4652-a85f-20945233ccc0-credential-socket\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2\" (UID: \"fd17aa36-e0ae-4652-a85f-20945233ccc0\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2"
Oct 30 13:19:18.745396 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:18.745373    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd17aa36-e0ae-4652-a85f-20945233ccc0-istiod-ca-cert\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2\" (UID: \"fd17aa36-e0ae-4652-a85f-20945233ccc0\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2"
Oct 30 13:19:18.793964 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:18.793232    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd17aa36-e0ae-4652-a85f-20945233ccc0-istio-token\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2\" (UID: \"fd17aa36-e0ae-4652-a85f-20945233ccc0\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2"
Oct 30 13:19:18.802619 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:18.796208    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd17aa36-e0ae-4652-a85f-20945233ccc0-istio-podinfo\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2\" (UID: \"fd17aa36-e0ae-4652-a85f-20945233ccc0\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2"
Oct 30 13:19:18.806343 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:18.806282    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd17aa36-e0ae-4652-a85f-20945233ccc0-istio-envoy\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2\" (UID: \"fd17aa36-e0ae-4652-a85f-20945233ccc0\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2"
Oct 30 13:19:18.822406 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:18.822376    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8bnk\" (UniqueName: \"kubernetes.io/projected/fd17aa36-e0ae-4652-a85f-20945233ccc0-kube-api-access-d8bnk\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2\" (UID: \"fd17aa36-e0ae-4652-a85f-20945233ccc0\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2"
Oct 30 13:19:18.871279 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:18.855831    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2"
Oct 30 13:19:19.107794 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:19.107767    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-31721c-6cc5477859-4pglh"
Oct 30 13:19:19.273607 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:19.273565    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx"]
Oct 30 13:19:19.279404 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:19.279381    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx"
Oct 30 13:19:19.303983 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:19.303955    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx"]
Oct 30 13:19:19.313085 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:19.313026    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-0ad384-openshift-default-dockercfg-tf9dh\""
Oct 30 13:19:19.358586 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:19.358459    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2"]
Oct 30 13:19:19.364228 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:19:19.362861    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd17aa36_e0ae_4652_a85f_20945233ccc0.slice/crio-774d1b268fdcd67b576c6dc19fde9c2285bc1cd45daa7c439283ee7ca3529c7a WatchSource:0}: Error finding container 774d1b268fdcd67b576c6dc19fde9c2285bc1cd45daa7c439283ee7ca3529c7a: Status 404 returned error can't find the container with id 774d1b268fdcd67b576c6dc19fde9c2285bc1cd45daa7c439283ee7ca3529c7a
Oct 30 13:19:19.366933 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:19.366903    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/945bfd4e-0d0d-499f-8457-5f37a0c48e28-istiod-ca-cert\") pod \"e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx\" (UID: \"945bfd4e-0d0d-499f-8457-5f37a0c48e28\") " pod="openshift-ingress/e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx"
Oct 30 13:19:19.367043 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:19.367001    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/945bfd4e-0d0d-499f-8457-5f37a0c48e28-workload-certs\") pod \"e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx\" (UID: \"945bfd4e-0d0d-499f-8457-5f37a0c48e28\") " pod="openshift-ingress/e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx"
Oct 30 13:19:19.367043 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:19.367035    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/945bfd4e-0d0d-499f-8457-5f37a0c48e28-credential-socket\") pod \"e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx\" (UID: \"945bfd4e-0d0d-499f-8457-5f37a0c48e28\") " pod="openshift-ingress/e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx"
Oct 30 13:19:19.367228 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:19.367130    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/945bfd4e-0d0d-499f-8457-5f37a0c48e28-workload-socket\") pod \"e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx\" (UID: \"945bfd4e-0d0d-499f-8457-5f37a0c48e28\") " pod="openshift-ingress/e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx"
Oct 30 13:19:19.367228 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:19.367183    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/945bfd4e-0d0d-499f-8457-5f37a0c48e28-istio-token\") pod \"e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx\" (UID: \"945bfd4e-0d0d-499f-8457-5f37a0c48e28\") " pod="openshift-ingress/e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx"
Oct 30 13:19:19.367228 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:19.367219    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpqkw\" (UniqueName: \"kubernetes.io/projected/945bfd4e-0d0d-499f-8457-5f37a0c48e28-kube-api-access-zpqkw\") pod \"e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx\" (UID: \"945bfd4e-0d0d-499f-8457-5f37a0c48e28\") " pod="openshift-ingress/e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx"
Oct 30 13:19:19.367403 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:19.367289    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/945bfd4e-0d0d-499f-8457-5f37a0c48e28-istio-podinfo\") pod \"e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx\" (UID: \"945bfd4e-0d0d-499f-8457-5f37a0c48e28\") " pod="openshift-ingress/e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx"
Oct 30 13:19:19.367403 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:19.367359    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/945bfd4e-0d0d-499f-8457-5f37a0c48e28-istio-data\") pod \"e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx\" (UID: \"945bfd4e-0d0d-499f-8457-5f37a0c48e28\") " pod="openshift-ingress/e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx"
Oct 30 13:19:19.367514 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:19.367410    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/945bfd4e-0d0d-499f-8457-5f37a0c48e28-istio-envoy\") pod \"e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx\" (UID: \"945bfd4e-0d0d-499f-8457-5f37a0c48e28\") " pod="openshift-ingress/e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx"
Oct 30 13:19:19.369390 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:19.369333    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:19:19.370152 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:19.370075    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:19:19.370398 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:19.370301    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:19:19.468894 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:19.468865    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/945bfd4e-0d0d-499f-8457-5f37a0c48e28-istio-envoy\") pod \"e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx\" (UID: \"945bfd4e-0d0d-499f-8457-5f37a0c48e28\") " pod="openshift-ingress/e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx"
Oct 30 13:19:19.469007 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:19.468972    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/945bfd4e-0d0d-499f-8457-5f37a0c48e28-istiod-ca-cert\") pod \"e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx\" (UID: \"945bfd4e-0d0d-499f-8457-5f37a0c48e28\") " pod="openshift-ingress/e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx"
Oct 30 13:19:19.469097 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:19.469084    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/945bfd4e-0d0d-499f-8457-5f37a0c48e28-workload-certs\") pod \"e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx\" (UID: \"945bfd4e-0d0d-499f-8457-5f37a0c48e28\") " pod="openshift-ingress/e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx"
Oct 30 13:19:19.469137 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:19.469119    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/945bfd4e-0d0d-499f-8457-5f37a0c48e28-credential-socket\") pod \"e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx\" (UID: \"945bfd4e-0d0d-499f-8457-5f37a0c48e28\") " pod="openshift-ingress/e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx"
Oct 30 13:19:19.469286 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:19.469258    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/945bfd4e-0d0d-499f-8457-5f37a0c48e28-workload-socket\") pod \"e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx\" (UID: \"945bfd4e-0d0d-499f-8457-5f37a0c48e28\") " pod="openshift-ingress/e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx"
Oct 30 13:19:19.469398 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:19.469330    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/945bfd4e-0d0d-499f-8457-5f37a0c48e28-istio-token\") pod \"e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx\" (UID: \"945bfd4e-0d0d-499f-8457-5f37a0c48e28\") " pod="openshift-ingress/e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx"
Oct 30 13:19:19.469398 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:19.469372    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpqkw\" (UniqueName: \"kubernetes.io/projected/945bfd4e-0d0d-499f-8457-5f37a0c48e28-kube-api-access-zpqkw\") pod \"e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx\" (UID: \"945bfd4e-0d0d-499f-8457-5f37a0c48e28\") " pod="openshift-ingress/e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx"
Oct 30 13:19:19.469535 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:19.469451    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/945bfd4e-0d0d-499f-8457-5f37a0c48e28-istio-podinfo\") pod \"e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx\" (UID: \"945bfd4e-0d0d-499f-8457-5f37a0c48e28\") " pod="openshift-ingress/e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx"
Oct 30 13:19:19.469593 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:19.469540    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/945bfd4e-0d0d-499f-8457-5f37a0c48e28-workload-certs\") pod \"e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx\" (UID: \"945bfd4e-0d0d-499f-8457-5f37a0c48e28\") " pod="openshift-ingress/e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx"
Oct 30 13:19:19.469593 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:19.469571    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/945bfd4e-0d0d-499f-8457-5f37a0c48e28-workload-socket\") pod \"e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx\" (UID: \"945bfd4e-0d0d-499f-8457-5f37a0c48e28\") " pod="openshift-ingress/e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx"
Oct 30 13:19:19.469593 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:19.469544    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/945bfd4e-0d0d-499f-8457-5f37a0c48e28-istio-data\") pod \"e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx\" (UID: \"945bfd4e-0d0d-499f-8457-5f37a0c48e28\") " pod="openshift-ingress/e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx"
Oct 30 13:19:19.469774 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:19.469666    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/945bfd4e-0d0d-499f-8457-5f37a0c48e28-credential-socket\") pod \"e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx\" (UID: \"945bfd4e-0d0d-499f-8457-5f37a0c48e28\") " pod="openshift-ingress/e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx"
Oct 30 13:19:19.469949 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:19.469918    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/945bfd4e-0d0d-499f-8457-5f37a0c48e28-istio-data\") pod \"e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx\" (UID: \"945bfd4e-0d0d-499f-8457-5f37a0c48e28\") " pod="openshift-ingress/e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx"
Oct 30 13:19:19.470060 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:19.469932    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/945bfd4e-0d0d-499f-8457-5f37a0c48e28-istiod-ca-cert\") pod \"e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx\" (UID: \"945bfd4e-0d0d-499f-8457-5f37a0c48e28\") " pod="openshift-ingress/e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx"
Oct 30 13:19:19.471426 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:19.471402    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/945bfd4e-0d0d-499f-8457-5f37a0c48e28-istio-envoy\") pod \"e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx\" (UID: \"945bfd4e-0d0d-499f-8457-5f37a0c48e28\") " pod="openshift-ingress/e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx"
Oct 30 13:19:19.471989 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:19.471965    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/945bfd4e-0d0d-499f-8457-5f37a0c48e28-istio-podinfo\") pod \"e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx\" (UID: \"945bfd4e-0d0d-499f-8457-5f37a0c48e28\") " pod="openshift-ingress/e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx"
Oct 30 13:19:19.493636 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:19.493210    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpqkw\" (UniqueName: \"kubernetes.io/projected/945bfd4e-0d0d-499f-8457-5f37a0c48e28-kube-api-access-zpqkw\") pod \"e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx\" (UID: \"945bfd4e-0d0d-499f-8457-5f37a0c48e28\") " pod="openshift-ingress/e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx"
Oct 30 13:19:19.493636 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:19.493291    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/945bfd4e-0d0d-499f-8457-5f37a0c48e28-istio-token\") pod \"e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx\" (UID: \"945bfd4e-0d0d-499f-8457-5f37a0c48e28\") " pod="openshift-ingress/e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx"
Oct 30 13:19:19.611670 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:19.607940    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx"
Oct 30 13:19:20.168096 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:20.168022    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx"]
Oct 30 13:19:20.173189 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:19:20.173155    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod945bfd4e_0d0d_499f_8457_5f37a0c48e28.slice/crio-66e62a9d96da5d5116d3a5cb176bccd9cc88fb4fa1b2f8f7f8ccf284f9d68771 WatchSource:0}: Error finding container 66e62a9d96da5d5116d3a5cb176bccd9cc88fb4fa1b2f8f7f8ccf284f9d68771: Status 404 returned error can't find the container with id 66e62a9d96da5d5116d3a5cb176bccd9cc88fb4fa1b2f8f7f8ccf284f9d68771
Oct 30 13:19:20.177155 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:20.177113    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:19:20.177275 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:20.177203    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:19:20.177275 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:20.177247    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:19:20.303894 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:20.302274    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2" event={"ID":"fd17aa36-e0ae-4652-a85f-20945233ccc0","Type":"ContainerStarted","Data":"b96ac07b11db5c0d57da34eada5d105abdfb0f1c9e05bd72d8f06f3357c2d022"}
Oct 30 13:19:20.303894 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:20.302349    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2" event={"ID":"fd17aa36-e0ae-4652-a85f-20945233ccc0","Type":"ContainerStarted","Data":"774d1b268fdcd67b576c6dc19fde9c2285bc1cd45daa7c439283ee7ca3529c7a"}
Oct 30 13:19:20.307858 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:20.307797    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx" event={"ID":"945bfd4e-0d0d-499f-8457-5f37a0c48e28","Type":"ContainerStarted","Data":"66e62a9d96da5d5116d3a5cb176bccd9cc88fb4fa1b2f8f7f8ccf284f9d68771"}
Oct 30 13:19:20.358578 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:20.358519    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2" podStartSLOduration=2.358499278 podStartE2EDuration="2.358499278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:19:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:19:20.354964673 +0000 UTC m=+1659.521329333" watchObservedRunningTime="2026-10-30 13:19:20.358499278 +0000 UTC m=+1659.524863928"
Oct 30 13:19:20.859884 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:20.859842    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2"
Oct 30 13:19:20.867910 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:20.867882    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2"
Oct 30 13:19:21.050755 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:21.050722    4364 patch_prober.go:29] interesting pod/e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:19:21.050919 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:21.050771    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj" podUID="dc3dc010-a693-406c-ba1e-55593789e70a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:19:21.318011 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:21.317974    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx" event={"ID":"945bfd4e-0d0d-499f-8457-5f37a0c48e28","Type":"ContainerStarted","Data":"632629d92e8dc7487e27cd045c846c6f44737ecfe34b41e6779ba9ddb88775af"}
Oct 30 13:19:21.319796 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:21.319772    4364 generic.go:356] "Generic (PLEG): container finished" podID="dc3dc010-a693-406c-ba1e-55593789e70a" containerID="f905015273e255c0722a45299627b4b2fb5bcd0811c8d44578957967701da44c" exitCode=0
Oct 30 13:19:21.319910 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:21.319840    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj" event={"ID":"dc3dc010-a693-406c-ba1e-55593789e70a","Type":"ContainerDied","Data":"f905015273e255c0722a45299627b4b2fb5bcd0811c8d44578957967701da44c"}
Oct 30 13:19:21.320111 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:21.320092    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2"
Oct 30 13:19:21.320350 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:21.320335    4364 scope.go:122] "RemoveContainer" containerID="f905015273e255c0722a45299627b4b2fb5bcd0811c8d44578957967701da44c"
Oct 30 13:19:21.321173 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:21.321150    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2"
Oct 30 13:19:21.321922 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:21.321869    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:19:21.321977 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:21.321933    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:19:21.321977 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:21.321962    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:19:21.364077 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:21.364028    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx" podStartSLOduration=2.3640116239999998 podStartE2EDuration="2.364011624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:19:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:19:21.362078686 +0000 UTC m=+1660.528443332" watchObservedRunningTime="2026-10-30 13:19:21.364011624 +0000 UTC m=+1660.530376277"
Oct 30 13:19:21.609173 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:21.609093    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx"
Oct 30 13:19:21.670620 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:21.648274    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx"
Oct 30 13:19:22.330448 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:22.330412    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj" event={"ID":"dc3dc010-a693-406c-ba1e-55593789e70a","Type":"ContainerStarted","Data":"b146f090c42801191e551e76cfefed623e03a5676c488624db81f6a54666589d"}
Oct 30 13:19:22.331025 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:22.331004    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx"
Oct 30 13:19:22.331907 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:22.331886    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx"
Oct 30 13:19:22.674810 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:22.674783    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj"
Oct 30 13:19:22.675015 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:22.674994    4364 patch_prober.go:29] interesting pod/e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Oct 30 13:19:22.675068 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:22.675047    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj" podUID="dc3dc010-a693-406c-ba1e-55593789e70a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Oct 30 13:19:23.254328 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:23.254303    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm"
Oct 30 13:19:23.343585 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:23.343547    4364 generic.go:356] "Generic (PLEG): container finished" podID="9ef21594-2f1f-4b43-9c5f-529c9f9763f7" containerID="800d907b4ccaf08e8c7e840e40094698c1c300512259106b4b43ee082b51f297" exitCode=0
Oct 30 13:19:23.344078 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:23.343588    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm"
Oct 30 13:19:23.344078 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:23.343631    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm" event={"ID":"9ef21594-2f1f-4b43-9c5f-529c9f9763f7","Type":"ContainerDied","Data":"800d907b4ccaf08e8c7e840e40094698c1c300512259106b4b43ee082b51f297"}
Oct 30 13:19:23.344078 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:23.343697    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm" event={"ID":"9ef21594-2f1f-4b43-9c5f-529c9f9763f7","Type":"ContainerDied","Data":"3a77f3235de1d4a14463b962dd9ce64acb54589f3205fff191d293a0cd610a7f"}
Oct 30 13:19:23.344078 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:23.343733    4364 scope.go:122] "RemoveContainer" containerID="800d907b4ccaf08e8c7e840e40094698c1c300512259106b4b43ee082b51f297"
Oct 30 13:19:23.361826 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:23.361802    4364 scope.go:122] "RemoveContainer" containerID="e07d871ed6035bbaa3a9a1cf1365580f2ebcbb398ca33e4cebc9e511aa7a3527"
Oct 30 13:19:23.375586 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:23.375565    4364 scope.go:122] "RemoveContainer" containerID="800d907b4ccaf08e8c7e840e40094698c1c300512259106b4b43ee082b51f297"
Oct 30 13:19:23.376327 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:19:23.375877    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"800d907b4ccaf08e8c7e840e40094698c1c300512259106b4b43ee082b51f297\": container with ID starting with 800d907b4ccaf08e8c7e840e40094698c1c300512259106b4b43ee082b51f297 not found: ID does not exist" containerID="800d907b4ccaf08e8c7e840e40094698c1c300512259106b4b43ee082b51f297"
Oct 30 13:19:23.376327 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:23.375909    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"800d907b4ccaf08e8c7e840e40094698c1c300512259106b4b43ee082b51f297"} err="failed to get container status \"800d907b4ccaf08e8c7e840e40094698c1c300512259106b4b43ee082b51f297\": rpc error: code = NotFound desc = could not find container \"800d907b4ccaf08e8c7e840e40094698c1c300512259106b4b43ee082b51f297\": container with ID starting with 800d907b4ccaf08e8c7e840e40094698c1c300512259106b4b43ee082b51f297 not found: ID does not exist"
Oct 30 13:19:23.376327 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:23.375965    4364 scope.go:122] "RemoveContainer" containerID="e07d871ed6035bbaa3a9a1cf1365580f2ebcbb398ca33e4cebc9e511aa7a3527"
Oct 30 13:19:23.376327 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:19:23.376243    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e07d871ed6035bbaa3a9a1cf1365580f2ebcbb398ca33e4cebc9e511aa7a3527\": container with ID starting with e07d871ed6035bbaa3a9a1cf1365580f2ebcbb398ca33e4cebc9e511aa7a3527 not found: ID does not exist" containerID="e07d871ed6035bbaa3a9a1cf1365580f2ebcbb398ca33e4cebc9e511aa7a3527"
Oct 30 13:19:23.376327 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:23.376272    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e07d871ed6035bbaa3a9a1cf1365580f2ebcbb398ca33e4cebc9e511aa7a3527"} err="failed to get container status \"e07d871ed6035bbaa3a9a1cf1365580f2ebcbb398ca33e4cebc9e511aa7a3527\": rpc error: code = NotFound desc = could not find container \"e07d871ed6035bbaa3a9a1cf1365580f2ebcbb398ca33e4cebc9e511aa7a3527\": container with ID starting with e07d871ed6035bbaa3a9a1cf1365580f2ebcbb398ca33e4cebc9e511aa7a3527 not found: ID does not exist"
Oct 30 13:19:23.406136 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:23.406107    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ef21594-2f1f-4b43-9c5f-529c9f9763f7-istio-token\" (UniqueName: \"kubernetes.io/projected/9ef21594-2f1f-4b43-9c5f-529c9f9763f7-istio-token\") pod \"9ef21594-2f1f-4b43-9c5f-529c9f9763f7\" (UID: \"9ef21594-2f1f-4b43-9c5f-529c9f9763f7\") "
Oct 30 13:19:23.406261 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:23.406149    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ef21594-2f1f-4b43-9c5f-529c9f9763f7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ef21594-2f1f-4b43-9c5f-529c9f9763f7-credential-socket\") pod \"9ef21594-2f1f-4b43-9c5f-529c9f9763f7\" (UID: \"9ef21594-2f1f-4b43-9c5f-529c9f9763f7\") "
Oct 30 13:19:23.406340 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:23.406296    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ef21594-2f1f-4b43-9c5f-529c9f9763f7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ef21594-2f1f-4b43-9c5f-529c9f9763f7-workload-certs\") pod \"9ef21594-2f1f-4b43-9c5f-529c9f9763f7\" (UID: \"9ef21594-2f1f-4b43-9c5f-529c9f9763f7\") "
Oct 30 13:19:23.406500 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:23.406478    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ef21594-2f1f-4b43-9c5f-529c9f9763f7-credential-socket" pod "9ef21594-2f1f-4b43-9c5f-529c9f9763f7" (UID: "9ef21594-2f1f-4b43-9c5f-529c9f9763f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:19:23.406572 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:23.406508    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ef21594-2f1f-4b43-9c5f-529c9f9763f7-workload-certs" pod "9ef21594-2f1f-4b43-9c5f-529c9f9763f7" (UID: "9ef21594-2f1f-4b43-9c5f-529c9f9763f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:19:23.406572 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:23.406549    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ef21594-2f1f-4b43-9c5f-529c9f9763f7-kube-api-access-gfpbm\" (UniqueName: \"kubernetes.io/projected/9ef21594-2f1f-4b43-9c5f-529c9f9763f7-kube-api-access-gfpbm\") pod \"9ef21594-2f1f-4b43-9c5f-529c9f9763f7\" (UID: \"9ef21594-2f1f-4b43-9c5f-529c9f9763f7\") "
Oct 30 13:19:23.406831 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:23.406798    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9ef21594-2f1f-4b43-9c5f-529c9f9763f7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ef21594-2f1f-4b43-9c5f-529c9f9763f7-istio-podinfo\") pod \"9ef21594-2f1f-4b43-9c5f-529c9f9763f7\" (UID: \"9ef21594-2f1f-4b43-9c5f-529c9f9763f7\") "
Oct 30 13:19:23.406906 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:23.406852    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ef21594-2f1f-4b43-9c5f-529c9f9763f7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ef21594-2f1f-4b43-9c5f-529c9f9763f7-workload-socket\") pod \"9ef21594-2f1f-4b43-9c5f-529c9f9763f7\" (UID: \"9ef21594-2f1f-4b43-9c5f-529c9f9763f7\") "
Oct 30 13:19:23.408347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:23.406967    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ef21594-2f1f-4b43-9c5f-529c9f9763f7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ef21594-2f1f-4b43-9c5f-529c9f9763f7-istiod-ca-cert\") pod \"9ef21594-2f1f-4b43-9c5f-529c9f9763f7\" (UID: \"9ef21594-2f1f-4b43-9c5f-529c9f9763f7\") "
Oct 30 13:19:23.408347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:23.407055    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ef21594-2f1f-4b43-9c5f-529c9f9763f7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ef21594-2f1f-4b43-9c5f-529c9f9763f7-istio-data\") pod \"9ef21594-2f1f-4b43-9c5f-529c9f9763f7\" (UID: \"9ef21594-2f1f-4b43-9c5f-529c9f9763f7\") "
Oct 30 13:19:23.408347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:23.407084    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ef21594-2f1f-4b43-9c5f-529c9f9763f7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ef21594-2f1f-4b43-9c5f-529c9f9763f7-istio-envoy\") pod \"9ef21594-2f1f-4b43-9c5f-529c9f9763f7\" (UID: \"9ef21594-2f1f-4b43-9c5f-529c9f9763f7\") "
Oct 30 13:19:23.408347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:23.407842    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ef21594-2f1f-4b43-9c5f-529c9f9763f7-workload-socket" pod "9ef21594-2f1f-4b43-9c5f-529c9f9763f7" (UID: "9ef21594-2f1f-4b43-9c5f-529c9f9763f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:19:23.408347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:23.407943    4364 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ef21594-2f1f-4b43-9c5f-529c9f9763f7-credential-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:19:23.408347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:23.407963    4364 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ef21594-2f1f-4b43-9c5f-529c9f9763f7-workload-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:19:23.408347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:23.408164    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ef21594-2f1f-4b43-9c5f-529c9f9763f7-istio-data" pod "9ef21594-2f1f-4b43-9c5f-529c9f9763f7" (UID: "9ef21594-2f1f-4b43-9c5f-529c9f9763f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:19:23.408347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:23.408315    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ef21594-2f1f-4b43-9c5f-529c9f9763f7-istiod-ca-cert" pod "9ef21594-2f1f-4b43-9c5f-529c9f9763f7" (UID: "9ef21594-2f1f-4b43-9c5f-529c9f9763f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:19:23.408916 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:23.408896    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ef21594-2f1f-4b43-9c5f-529c9f9763f7-istio-token" pod "9ef21594-2f1f-4b43-9c5f-529c9f9763f7" (UID: "9ef21594-2f1f-4b43-9c5f-529c9f9763f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:19:23.409550 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:23.409515    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ef21594-2f1f-4b43-9c5f-529c9f9763f7-kube-api-access-gfpbm" pod "9ef21594-2f1f-4b43-9c5f-529c9f9763f7" (UID: "9ef21594-2f1f-4b43-9c5f-529c9f9763f7"). InnerVolumeSpecName "kube-api-access-gfpbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:19:23.410906 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:23.410886    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ef21594-2f1f-4b43-9c5f-529c9f9763f7-istio-envoy" pod "9ef21594-2f1f-4b43-9c5f-529c9f9763f7" (UID: "9ef21594-2f1f-4b43-9c5f-529c9f9763f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:19:23.411166 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:23.411151    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ef21594-2f1f-4b43-9c5f-529c9f9763f7-istio-podinfo" pod "9ef21594-2f1f-4b43-9c5f-529c9f9763f7" (UID: "9ef21594-2f1f-4b43-9c5f-529c9f9763f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:19:23.509380 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:23.509327    4364 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ef21594-2f1f-4b43-9c5f-529c9f9763f7-istio-podinfo\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:19:23.509501 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:23.509413    4364 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ef21594-2f1f-4b43-9c5f-529c9f9763f7-workload-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:19:23.509501 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:23.509428    4364 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ef21594-2f1f-4b43-9c5f-529c9f9763f7-istiod-ca-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:19:23.509501 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:23.509439    4364 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ef21594-2f1f-4b43-9c5f-529c9f9763f7-istio-data\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:19:23.509501 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:23.509447    4364 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ef21594-2f1f-4b43-9c5f-529c9f9763f7-istio-envoy\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:19:23.509501 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:23.509454    4364 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ef21594-2f1f-4b43-9c5f-529c9f9763f7-istio-token\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:19:23.509501 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:23.509462    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfpbm\" (UniqueName: \"kubernetes.io/projected/9ef21594-2f1f-4b43-9c5f-529c9f9763f7-kube-api-access-gfpbm\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:19:23.675091 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:23.675057    4364 patch_prober.go:29] interesting pod/e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Oct 30 13:19:23.675216 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:23.675108    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj" podUID="dc3dc010-a693-406c-ba1e-55593789e70a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Oct 30 13:19:23.706758 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:23.706730    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm"]
Oct 30 13:19:23.714839 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:23.714818    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-2daeae-openshift-default-77b988c447-cc6wm"]
Oct 30 13:19:24.675001 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:24.674961    4364 patch_prober.go:29] interesting pod/e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Oct 30 13:19:24.675466 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:24.675034    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj" podUID="dc3dc010-a693-406c-ba1e-55593789e70a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Oct 30 13:19:24.809785 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:24.809750    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ff4285-59fc768575-tgw8j"]
Oct 30 13:19:24.811101 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:24.811074    4364 cpu_manager.go:414] "containerMap: removing container" podUID="9ef21594-2f1f-4b43-9c5f-529c9f9763f7" containerName="istio-proxy"
Oct 30 13:19:24.811101 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:24.811101    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ef21594-2f1f-4b43-9c5f-529c9f9763f7" containerName="istio-proxy"
Oct 30 13:19:24.811301 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:24.811164    4364 cpu_manager.go:414] "containerMap: removing container" podUID="9ef21594-2f1f-4b43-9c5f-529c9f9763f7" containerName="istio-proxy"
Oct 30 13:19:24.811301 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:24.811173    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ef21594-2f1f-4b43-9c5f-529c9f9763f7" containerName="istio-proxy"
Oct 30 13:19:24.811445 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:24.811425    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ef21594-2f1f-4b43-9c5f-529c9f9763f7" containerName="istio-proxy"
Oct 30 13:19:24.850141 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:24.835612    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ff4285-59fc768575-tgw8j"
Oct 30 13:19:24.850141 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:24.835647    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ff4285-59fc768575-tgw8j"]
Oct 30 13:19:24.927676 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:24.884841    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-ff4285-dockercfg-gs8bw\""
Oct 30 13:19:24.927676 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:24.884922    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-ff4285\""
Oct 30 13:19:25.100627 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:25.096471    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5c6ed73-3541-4945-8d38-cb289a4e05f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-ff4285-59fc768575-tgw8j\" (UID: \"e5c6ed73-3541-4945-8d38-cb289a4e05f8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ff4285-59fc768575-tgw8j"
Oct 30 13:19:25.100627 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:25.096689    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbwl8\" (UniqueName: \"kubernetes.io/projected/e5c6ed73-3541-4945-8d38-cb289a4e05f8-kube-api-access-tbwl8\") pod \"payload-pre-processing-e2e-shared-a-w1-ff4285-59fc768575-tgw8j\" (UID: \"e5c6ed73-3541-4945-8d38-cb289a4e05f8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ff4285-59fc768575-tgw8j"
Oct 30 13:19:25.224901 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:25.224807    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbwl8\" (UniqueName: \"kubernetes.io/projected/e5c6ed73-3541-4945-8d38-cb289a4e05f8-kube-api-access-tbwl8\") pod \"payload-pre-processing-e2e-shared-a-w1-ff4285-59fc768575-tgw8j\" (UID: \"e5c6ed73-3541-4945-8d38-cb289a4e05f8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ff4285-59fc768575-tgw8j"
Oct 30 13:19:25.247959 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:25.225127    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5c6ed73-3541-4945-8d38-cb289a4e05f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-ff4285-59fc768575-tgw8j\" (UID: \"e5c6ed73-3541-4945-8d38-cb289a4e05f8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ff4285-59fc768575-tgw8j"
Oct 30 13:19:25.253623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:25.249700    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5c6ed73-3541-4945-8d38-cb289a4e05f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-ff4285-59fc768575-tgw8j\" (UID: \"e5c6ed73-3541-4945-8d38-cb289a4e05f8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ff4285-59fc768575-tgw8j"
Oct 30 13:19:25.284629 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:25.283661    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ef21594-2f1f-4b43-9c5f-529c9f9763f7" path="/var/lib/kubelet/pods/9ef21594-2f1f-4b43-9c5f-529c9f9763f7/volumes"
Oct 30 13:19:25.284629 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:25.284186    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-ff4285-6b8554d544-6flwq"]
Oct 30 13:19:25.288978 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:25.286807    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ef21594-2f1f-4b43-9c5f-529c9f9763f7" containerName="istio-proxy"
Oct 30 13:19:25.290369 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:25.290348    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ff4285-6b8554d544-6flwq"
Oct 30 13:19:25.306649 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:25.305587    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-ff4285-6b8554d544-6flwq"]
Oct 30 13:19:25.330458 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:25.329725    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbwl8\" (UniqueName: \"kubernetes.io/projected/e5c6ed73-3541-4945-8d38-cb289a4e05f8-kube-api-access-tbwl8\") pod \"payload-pre-processing-e2e-shared-a-w1-ff4285-59fc768575-tgw8j\" (UID: \"e5c6ed73-3541-4945-8d38-cb289a4e05f8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ff4285-59fc768575-tgw8j"
Oct 30 13:19:25.451539 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:25.451511    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43e6fe99-7d56-417f-b9db-1d41a24340d2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-ff4285-6b8554d544-6flwq\" (UID: \"43e6fe99-7d56-417f-b9db-1d41a24340d2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ff4285-6b8554d544-6flwq"
Oct 30 13:19:25.451915 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:25.451889    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7svt\" (UniqueName: \"kubernetes.io/projected/43e6fe99-7d56-417f-b9db-1d41a24340d2-kube-api-access-w7svt\") pod \"payload-processing-e2e-shared-a-w1-ff4285-6b8554d544-6flwq\" (UID: \"43e6fe99-7d56-417f-b9db-1d41a24340d2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ff4285-6b8554d544-6flwq"
Oct 30 13:19:25.553377 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:25.553298    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7svt\" (UniqueName: \"kubernetes.io/projected/43e6fe99-7d56-417f-b9db-1d41a24340d2-kube-api-access-w7svt\") pod \"payload-processing-e2e-shared-a-w1-ff4285-6b8554d544-6flwq\" (UID: \"43e6fe99-7d56-417f-b9db-1d41a24340d2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ff4285-6b8554d544-6flwq"
Oct 30 13:19:25.553377 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:25.553333    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43e6fe99-7d56-417f-b9db-1d41a24340d2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-ff4285-6b8554d544-6flwq\" (UID: \"43e6fe99-7d56-417f-b9db-1d41a24340d2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ff4285-6b8554d544-6flwq"
Oct 30 13:19:25.554065 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:25.554045    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43e6fe99-7d56-417f-b9db-1d41a24340d2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-ff4285-6b8554d544-6flwq\" (UID: \"43e6fe99-7d56-417f-b9db-1d41a24340d2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ff4285-6b8554d544-6flwq"
Oct 30 13:19:25.576469 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:25.576438    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7svt\" (UniqueName: \"kubernetes.io/projected/43e6fe99-7d56-417f-b9db-1d41a24340d2-kube-api-access-w7svt\") pod \"payload-processing-e2e-shared-a-w1-ff4285-6b8554d544-6flwq\" (UID: \"43e6fe99-7d56-417f-b9db-1d41a24340d2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ff4285-6b8554d544-6flwq"
Oct 30 13:19:25.606664 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:25.596823    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ff4285-59fc768575-tgw8j"
Oct 30 13:19:25.664649 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:25.664583    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ff4285-6b8554d544-6flwq"
Oct 30 13:19:25.688643 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:25.676270    4364 patch_prober.go:29] interesting pod/e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Oct 30 13:19:25.688643 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:25.676329    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj" podUID="dc3dc010-a693-406c-ba1e-55593789e70a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Oct 30 13:19:25.925215 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:25.925185    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ff4285-59fc768575-tgw8j"]
Oct 30 13:19:25.926096 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:19:25.926070    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5c6ed73_3541_4945_8d38_cb289a4e05f8.slice/crio-f765a5f6a49fb0e97d299eeb4f1dd9bbc6329deb46183818370f35270249196d WatchSource:0}: Error finding container f765a5f6a49fb0e97d299eeb4f1dd9bbc6329deb46183818370f35270249196d: Status 404 returned error can't find the container with id f765a5f6a49fb0e97d299eeb4f1dd9bbc6329deb46183818370f35270249196d
Oct 30 13:19:26.013711 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:19:26.013676    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43e6fe99_7d56_417f_b9db_1d41a24340d2.slice/crio-3c78831d67fc34bf0467a6bb68667f7b3d9cb2e077080860dbd960bc76a1c505 WatchSource:0}: Error finding container 3c78831d67fc34bf0467a6bb68667f7b3d9cb2e077080860dbd960bc76a1c505: Status 404 returned error can't find the container with id 3c78831d67fc34bf0467a6bb68667f7b3d9cb2e077080860dbd960bc76a1c505
Oct 30 13:19:26.014530 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:26.014502    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-ff4285-6b8554d544-6flwq"]
Oct 30 13:19:26.412969 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:26.412936    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0ad384-554b74dc84-486b2"]
Oct 30 13:19:26.416685 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:26.416662    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0ad384-554b74dc84-486b2"
Oct 30 13:19:26.417400 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:26.417371    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ff4285-6b8554d544-6flwq" event={"ID":"43e6fe99-7d56-417f-b9db-1d41a24340d2","Type":"ContainerStarted","Data":"e7763a4801ab768e54a01a08625bda1aed3ea3e1c6c4323eee61c82fa52e72cd"}
Oct 30 13:19:26.417518 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:26.417422    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ff4285-6b8554d544-6flwq" event={"ID":"43e6fe99-7d56-417f-b9db-1d41a24340d2","Type":"ContainerStarted","Data":"3c78831d67fc34bf0467a6bb68667f7b3d9cb2e077080860dbd960bc76a1c505"}
Oct 30 13:19:26.417518 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:26.417452    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ff4285-6b8554d544-6flwq"
Oct 30 13:19:26.419213 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:26.419195    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ff4285-59fc768575-tgw8j" event={"ID":"e5c6ed73-3541-4945-8d38-cb289a4e05f8","Type":"ContainerStarted","Data":"40100748bc3f22e0b545a30cdee796a670965ce2fadcc87c0aba5ab0d65622fb"}
Oct 30 13:19:26.419305 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:26.419235    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ff4285-59fc768575-tgw8j"
Oct 30 13:19:26.419305 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:26.419251    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ff4285-59fc768575-tgw8j" event={"ID":"e5c6ed73-3541-4945-8d38-cb289a4e05f8","Type":"ContainerStarted","Data":"f765a5f6a49fb0e97d299eeb4f1dd9bbc6329deb46183818370f35270249196d"}
Oct 30 13:19:26.433638 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:26.433622    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-0ad384-dockercfg-rhwnd\""
Oct 30 13:19:26.437955 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:26.437940    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-0ad384\""
Oct 30 13:19:26.466550 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:26.466281    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0ad384-554b74dc84-486b2"]
Oct 30 13:19:26.558063 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:26.557908    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ff4285-6b8554d544-6flwq" podStartSLOduration=2.557895396 podStartE2EDuration="2.557895396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:19:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:19:26.556886061 +0000 UTC m=+1665.723250709" watchObservedRunningTime="2026-10-30 13:19:26.557895396 +0000 UTC m=+1665.724260042"
Oct 30 13:19:26.562285 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:26.562215    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ff4285-59fc768575-tgw8j" podStartSLOduration=2.5622026890000003 podStartE2EDuration="2.562202689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:19:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:19:26.509989731 +0000 UTC m=+1665.676354379" watchObservedRunningTime="2026-10-30 13:19:26.562202689 +0000 UTC m=+1665.728567390"
Oct 30 13:19:26.569708 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:26.569683    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cd7b23d-f2dc-49be-b0a5-0dd233c8faa9-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0ad384-554b74dc84-486b2\" (UID: \"0cd7b23d-f2dc-49be-b0a5-0dd233c8faa9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0ad384-554b74dc84-486b2"
Oct 30 13:19:26.570005 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:26.569975    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbqvq\" (UniqueName: \"kubernetes.io/projected/0cd7b23d-f2dc-49be-b0a5-0dd233c8faa9-kube-api-access-dbqvq\") pod \"payload-processing-e2e-inf-a-0ad384-554b74dc84-486b2\" (UID: \"0cd7b23d-f2dc-49be-b0a5-0dd233c8faa9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0ad384-554b74dc84-486b2"
Oct 30 13:19:26.672559 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:26.672469    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cd7b23d-f2dc-49be-b0a5-0dd233c8faa9-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0ad384-554b74dc84-486b2\" (UID: \"0cd7b23d-f2dc-49be-b0a5-0dd233c8faa9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0ad384-554b74dc84-486b2"
Oct 30 13:19:26.672559 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:26.672550    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbqvq\" (UniqueName: \"kubernetes.io/projected/0cd7b23d-f2dc-49be-b0a5-0dd233c8faa9-kube-api-access-dbqvq\") pod \"payload-processing-e2e-inf-a-0ad384-554b74dc84-486b2\" (UID: \"0cd7b23d-f2dc-49be-b0a5-0dd233c8faa9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0ad384-554b74dc84-486b2"
Oct 30 13:19:26.674299 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:26.674268    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cd7b23d-f2dc-49be-b0a5-0dd233c8faa9-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0ad384-554b74dc84-486b2\" (UID: \"0cd7b23d-f2dc-49be-b0a5-0dd233c8faa9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0ad384-554b74dc84-486b2"
Oct 30 13:19:26.676051 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:26.675079    4364 patch_prober.go:29] interesting pod/e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Oct 30 13:19:26.676051 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:26.675288    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj" podUID="dc3dc010-a693-406c-ba1e-55593789e70a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Oct 30 13:19:26.733685 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:26.732268    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbqvq\" (UniqueName: \"kubernetes.io/projected/0cd7b23d-f2dc-49be-b0a5-0dd233c8faa9-kube-api-access-dbqvq\") pod \"payload-processing-e2e-inf-a-0ad384-554b74dc84-486b2\" (UID: \"0cd7b23d-f2dc-49be-b0a5-0dd233c8faa9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0ad384-554b74dc84-486b2"
Oct 30 13:19:26.743621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:26.740306    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0ad384-554b74dc84-486b2"
Oct 30 13:19:27.375265 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:27.375235    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0ad384-554b74dc84-486b2"]
Oct 30 13:19:27.377058 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:19:27.377017    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cd7b23d_f2dc_49be_b0a5_0dd233c8faa9.slice/crio-cfcc600c59903272c6b37a0544e65cf0f70672ac4121c88c36aa0f2c958a4c39 WatchSource:0}: Error finding container cfcc600c59903272c6b37a0544e65cf0f70672ac4121c88c36aa0f2c958a4c39: Status 404 returned error can't find the container with id cfcc600c59903272c6b37a0544e65cf0f70672ac4121c88c36aa0f2c958a4c39
Oct 30 13:19:27.429448 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:27.429417    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0ad384-554b74dc84-486b2" event={"ID":"0cd7b23d-f2dc-49be-b0a5-0dd233c8faa9","Type":"ContainerStarted","Data":"cfcc600c59903272c6b37a0544e65cf0f70672ac4121c88c36aa0f2c958a4c39"}
Oct 30 13:19:27.675510 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:27.675422    4364 patch_prober.go:29] interesting pod/e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Oct 30 13:19:27.675510 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:27.675481    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj" podUID="dc3dc010-a693-406c-ba1e-55593789e70a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Oct 30 13:19:28.469480 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:28.469439    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0ad384-554b74dc84-486b2" event={"ID":"0cd7b23d-f2dc-49be-b0a5-0dd233c8faa9","Type":"ContainerStarted","Data":"65dd0bb5a787194b4f374511bd09d7eac49760c506055e24073b53b19bed8148"}
Oct 30 13:19:28.471378 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:28.471350    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-0ad384-554b74dc84-486b2"
Oct 30 13:19:28.532759 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:28.532695    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-0ad384-554b74dc84-486b2" podStartSLOduration=2.532671963 podStartE2EDuration="2.532671963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:19:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:19:28.522912341 +0000 UTC m=+1667.689276988" watchObservedRunningTime="2026-10-30 13:19:28.532671963 +0000 UTC m=+1667.699036610"
Oct 30 13:19:28.709622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:28.708683    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj"
Oct 30 13:19:28.709812 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:28.709774    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj"
Oct 30 13:19:28.713463 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:28.712536    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj"
Oct 30 13:19:37.432686 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:37.432655    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ff4285-59fc768575-tgw8j"
Oct 30 13:19:37.433089 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:37.432735    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ff4285-6b8554d544-6flwq"
Oct 30 13:19:40.499809 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:40.499777    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-0ad384-554b74dc84-486b2"
Oct 30 13:19:41.504761 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:19:41.504729    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:19:43.359456 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:43.359423    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn"]
Oct 30 13:19:43.365181 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:43.365158    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn"
Oct 30 13:19:43.392260 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:43.392235    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-8467ed-openshift-default-dockercfg-hc2sc\""
Oct 30 13:19:43.398231 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:43.398200    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn"]
Oct 30 13:19:43.512685 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:43.509270    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebad4af8-38a2-4da7-a1ec-b56ef4c48a79-workload-socket\") pod \"e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn\" (UID: \"ebad4af8-38a2-4da7-a1ec-b56ef4c48a79\") " pod="openshift-ingress/e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn"
Oct 30 13:19:43.512685 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:43.509354    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebad4af8-38a2-4da7-a1ec-b56ef4c48a79-credential-socket\") pod \"e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn\" (UID: \"ebad4af8-38a2-4da7-a1ec-b56ef4c48a79\") " pod="openshift-ingress/e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn"
Oct 30 13:19:43.512685 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:43.509396    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebad4af8-38a2-4da7-a1ec-b56ef4c48a79-istio-envoy\") pod \"e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn\" (UID: \"ebad4af8-38a2-4da7-a1ec-b56ef4c48a79\") " pod="openshift-ingress/e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn"
Oct 30 13:19:43.512685 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:43.509546    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebad4af8-38a2-4da7-a1ec-b56ef4c48a79-istiod-ca-cert\") pod \"e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn\" (UID: \"ebad4af8-38a2-4da7-a1ec-b56ef4c48a79\") " pod="openshift-ingress/e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn"
Oct 30 13:19:43.512685 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:43.509719    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebad4af8-38a2-4da7-a1ec-b56ef4c48a79-istio-token\") pod \"e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn\" (UID: \"ebad4af8-38a2-4da7-a1ec-b56ef4c48a79\") " pod="openshift-ingress/e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn"
Oct 30 13:19:43.512685 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:43.509800    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cvgc\" (UniqueName: \"kubernetes.io/projected/ebad4af8-38a2-4da7-a1ec-b56ef4c48a79-kube-api-access-9cvgc\") pod \"e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn\" (UID: \"ebad4af8-38a2-4da7-a1ec-b56ef4c48a79\") " pod="openshift-ingress/e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn"
Oct 30 13:19:43.512685 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:43.509853    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebad4af8-38a2-4da7-a1ec-b56ef4c48a79-workload-certs\") pod \"e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn\" (UID: \"ebad4af8-38a2-4da7-a1ec-b56ef4c48a79\") " pod="openshift-ingress/e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn"
Oct 30 13:19:43.512685 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:43.509884    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebad4af8-38a2-4da7-a1ec-b56ef4c48a79-istio-podinfo\") pod \"e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn\" (UID: \"ebad4af8-38a2-4da7-a1ec-b56ef4c48a79\") " pod="openshift-ingress/e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn"
Oct 30 13:19:43.512685 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:43.509968    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebad4af8-38a2-4da7-a1ec-b56ef4c48a79-istio-data\") pod \"e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn\" (UID: \"ebad4af8-38a2-4da7-a1ec-b56ef4c48a79\") " pod="openshift-ingress/e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn"
Oct 30 13:19:43.635677 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:43.628657    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebad4af8-38a2-4da7-a1ec-b56ef4c48a79-istiod-ca-cert\") pod \"e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn\" (UID: \"ebad4af8-38a2-4da7-a1ec-b56ef4c48a79\") " pod="openshift-ingress/e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn"
Oct 30 13:19:43.635677 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:43.628825    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebad4af8-38a2-4da7-a1ec-b56ef4c48a79-istio-token\") pod \"e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn\" (UID: \"ebad4af8-38a2-4da7-a1ec-b56ef4c48a79\") " pod="openshift-ingress/e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn"
Oct 30 13:19:43.635677 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:43.628911    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cvgc\" (UniqueName: \"kubernetes.io/projected/ebad4af8-38a2-4da7-a1ec-b56ef4c48a79-kube-api-access-9cvgc\") pod \"e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn\" (UID: \"ebad4af8-38a2-4da7-a1ec-b56ef4c48a79\") " pod="openshift-ingress/e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn"
Oct 30 13:19:43.635677 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:43.628959    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebad4af8-38a2-4da7-a1ec-b56ef4c48a79-workload-certs\") pod \"e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn\" (UID: \"ebad4af8-38a2-4da7-a1ec-b56ef4c48a79\") " pod="openshift-ingress/e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn"
Oct 30 13:19:43.635677 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:43.628987    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebad4af8-38a2-4da7-a1ec-b56ef4c48a79-istio-podinfo\") pod \"e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn\" (UID: \"ebad4af8-38a2-4da7-a1ec-b56ef4c48a79\") " pod="openshift-ingress/e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn"
Oct 30 13:19:43.635677 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:43.629065    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebad4af8-38a2-4da7-a1ec-b56ef4c48a79-istio-data\") pod \"e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn\" (UID: \"ebad4af8-38a2-4da7-a1ec-b56ef4c48a79\") " pod="openshift-ingress/e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn"
Oct 30 13:19:43.635677 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:43.629135    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebad4af8-38a2-4da7-a1ec-b56ef4c48a79-workload-socket\") pod \"e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn\" (UID: \"ebad4af8-38a2-4da7-a1ec-b56ef4c48a79\") " pod="openshift-ingress/e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn"
Oct 30 13:19:43.635677 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:43.629184    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebad4af8-38a2-4da7-a1ec-b56ef4c48a79-credential-socket\") pod \"e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn\" (UID: \"ebad4af8-38a2-4da7-a1ec-b56ef4c48a79\") " pod="openshift-ingress/e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn"
Oct 30 13:19:43.635677 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:43.629222    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebad4af8-38a2-4da7-a1ec-b56ef4c48a79-istio-envoy\") pod \"e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn\" (UID: \"ebad4af8-38a2-4da7-a1ec-b56ef4c48a79\") " pod="openshift-ingress/e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn"
Oct 30 13:19:43.635677 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:43.630053    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebad4af8-38a2-4da7-a1ec-b56ef4c48a79-workload-certs\") pod \"e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn\" (UID: \"ebad4af8-38a2-4da7-a1ec-b56ef4c48a79\") " pod="openshift-ingress/e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn"
Oct 30 13:19:43.635677 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:43.630723    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebad4af8-38a2-4da7-a1ec-b56ef4c48a79-istiod-ca-cert\") pod \"e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn\" (UID: \"ebad4af8-38a2-4da7-a1ec-b56ef4c48a79\") " pod="openshift-ingress/e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn"
Oct 30 13:19:43.635677 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:43.631479    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebad4af8-38a2-4da7-a1ec-b56ef4c48a79-istio-data\") pod \"e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn\" (UID: \"ebad4af8-38a2-4da7-a1ec-b56ef4c48a79\") " pod="openshift-ingress/e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn"
Oct 30 13:19:43.635677 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:43.632103    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebad4af8-38a2-4da7-a1ec-b56ef4c48a79-credential-socket\") pod \"e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn\" (UID: \"ebad4af8-38a2-4da7-a1ec-b56ef4c48a79\") " pod="openshift-ingress/e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn"
Oct 30 13:19:43.635677 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:43.632427    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebad4af8-38a2-4da7-a1ec-b56ef4c48a79-workload-socket\") pod \"e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn\" (UID: \"ebad4af8-38a2-4da7-a1ec-b56ef4c48a79\") " pod="openshift-ingress/e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn"
Oct 30 13:19:43.640165 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:43.640139    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebad4af8-38a2-4da7-a1ec-b56ef4c48a79-istio-envoy\") pod \"e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn\" (UID: \"ebad4af8-38a2-4da7-a1ec-b56ef4c48a79\") " pod="openshift-ingress/e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn"
Oct 30 13:19:43.659233 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:43.658676    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebad4af8-38a2-4da7-a1ec-b56ef4c48a79-istio-podinfo\") pod \"e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn\" (UID: \"ebad4af8-38a2-4da7-a1ec-b56ef4c48a79\") " pod="openshift-ingress/e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn"
Oct 30 13:19:43.685277 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:43.680181    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebad4af8-38a2-4da7-a1ec-b56ef4c48a79-istio-token\") pod \"e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn\" (UID: \"ebad4af8-38a2-4da7-a1ec-b56ef4c48a79\") " pod="openshift-ingress/e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn"
Oct 30 13:19:43.685277 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:43.680780    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cvgc\" (UniqueName: \"kubernetes.io/projected/ebad4af8-38a2-4da7-a1ec-b56ef4c48a79-kube-api-access-9cvgc\") pod \"e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn\" (UID: \"ebad4af8-38a2-4da7-a1ec-b56ef4c48a79\") " pod="openshift-ingress/e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn"
Oct 30 13:19:43.686121 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:43.686099    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn"
Oct 30 13:19:44.256447 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:19:44.256419    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebad4af8_38a2_4da7_a1ec_b56ef4c48a79.slice/crio-c53c3f0f3145151e221e038fe449a4b69378a475f95ae2cb58495d24bc65959c WatchSource:0}: Error finding container c53c3f0f3145151e221e038fe449a4b69378a475f95ae2cb58495d24bc65959c: Status 404 returned error can't find the container with id c53c3f0f3145151e221e038fe449a4b69378a475f95ae2cb58495d24bc65959c
Oct 30 13:19:44.256447 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:44.256434    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn"]
Oct 30 13:19:44.259177 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:44.259149    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:19:44.259247 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:44.259209    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:19:44.259247 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:44.259238    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:19:44.923938 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:44.923895    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn" event={"ID":"ebad4af8-38a2-4da7-a1ec-b56ef4c48a79","Type":"ContainerStarted","Data":"b4e8057dbfc71e745c1ec7b36ba257b389a44a362756b1272c6f5253e3d046f1"}
Oct 30 13:19:44.924329 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:44.923969    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn" event={"ID":"ebad4af8-38a2-4da7-a1ec-b56ef4c48a79","Type":"ContainerStarted","Data":"c53c3f0f3145151e221e038fe449a4b69378a475f95ae2cb58495d24bc65959c"}
Oct 30 13:19:44.972263 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:44.972217    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn" podStartSLOduration=1.972205387 podStartE2EDuration="1.972205387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:19:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:19:44.967996944 +0000 UTC m=+1684.134361591" watchObservedRunningTime="2026-10-30 13:19:44.972205387 +0000 UTC m=+1684.138570033"
Oct 30 13:19:45.687811 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:45.687773    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn"
Oct 30 13:19:45.694184 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:45.694158    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn"
Oct 30 13:19:45.932564 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:45.932524    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn"
Oct 30 13:19:45.933331 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:45.933311    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn"
Oct 30 13:19:49.146281 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:49.146232    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8467ed-86997f5d8c-dxslk"]
Oct 30 13:19:49.149996 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:49.149972    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8467ed-86997f5d8c-dxslk"
Oct 30 13:19:49.175067 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:49.175039    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8467ed-86997f5d8c-dxslk"]
Oct 30 13:19:49.179544 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:49.179520    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-8467ed-dockercfg-wzhv4\""
Oct 30 13:19:49.271617 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:49.270564    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-8467ed\""
Oct 30 13:19:49.396627 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:49.396466    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27c2bf35-ac61-4f5c-80f7-ee4d2b6c4b07-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8467ed-86997f5d8c-dxslk\" (UID: \"27c2bf35-ac61-4f5c-80f7-ee4d2b6c4b07\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8467ed-86997f5d8c-dxslk"
Oct 30 13:19:49.396956 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:49.396931    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngw2k\" (UniqueName: \"kubernetes.io/projected/27c2bf35-ac61-4f5c-80f7-ee4d2b6c4b07-kube-api-access-ngw2k\") pod \"payload-pre-processing-e2e-inf-b-8467ed-86997f5d8c-dxslk\" (UID: \"27c2bf35-ac61-4f5c-80f7-ee4d2b6c4b07\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8467ed-86997f5d8c-dxslk"
Oct 30 13:19:49.503567 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:49.501878    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27c2bf35-ac61-4f5c-80f7-ee4d2b6c4b07-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8467ed-86997f5d8c-dxslk\" (UID: \"27c2bf35-ac61-4f5c-80f7-ee4d2b6c4b07\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8467ed-86997f5d8c-dxslk"
Oct 30 13:19:49.503567 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:49.502005    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngw2k\" (UniqueName: \"kubernetes.io/projected/27c2bf35-ac61-4f5c-80f7-ee4d2b6c4b07-kube-api-access-ngw2k\") pod \"payload-pre-processing-e2e-inf-b-8467ed-86997f5d8c-dxslk\" (UID: \"27c2bf35-ac61-4f5c-80f7-ee4d2b6c4b07\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8467ed-86997f5d8c-dxslk"
Oct 30 13:19:49.503567 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:49.503516    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27c2bf35-ac61-4f5c-80f7-ee4d2b6c4b07-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8467ed-86997f5d8c-dxslk\" (UID: \"27c2bf35-ac61-4f5c-80f7-ee4d2b6c4b07\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8467ed-86997f5d8c-dxslk"
Oct 30 13:19:49.553622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:49.549790    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngw2k\" (UniqueName: \"kubernetes.io/projected/27c2bf35-ac61-4f5c-80f7-ee4d2b6c4b07-kube-api-access-ngw2k\") pod \"payload-pre-processing-e2e-inf-b-8467ed-86997f5d8c-dxslk\" (UID: \"27c2bf35-ac61-4f5c-80f7-ee4d2b6c4b07\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8467ed-86997f5d8c-dxslk"
Oct 30 13:19:49.766544 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:49.766463    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8467ed-86997f5d8c-dxslk"
Oct 30 13:19:50.079127 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:50.079100    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8467ed-86997f5d8c-dxslk"]
Oct 30 13:19:50.079558 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:19:50.079535    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27c2bf35_ac61_4f5c_80f7_ee4d2b6c4b07.slice/crio-7f62ccea32caa5fa0ce845ed9601c09520ee27032cd0fc0b13547e69654d76db WatchSource:0}: Error finding container 7f62ccea32caa5fa0ce845ed9601c09520ee27032cd0fc0b13547e69654d76db: Status 404 returned error can't find the container with id 7f62ccea32caa5fa0ce845ed9601c09520ee27032cd0fc0b13547e69654d76db
Oct 30 13:19:50.324620 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:50.320567    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:19:50.988380 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:50.988343    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8467ed-86997f5d8c-dxslk" event={"ID":"27c2bf35-ac61-4f5c-80f7-ee4d2b6c4b07","Type":"ContainerStarted","Data":"2e3a41e2ca1e44b4c58c41be9647fdec01b56e9d03a3cd51f974913a123f2544"}
Oct 30 13:19:50.988551 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:50.988401    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8467ed-86997f5d8c-dxslk" event={"ID":"27c2bf35-ac61-4f5c-80f7-ee4d2b6c4b07","Type":"ContainerStarted","Data":"7f62ccea32caa5fa0ce845ed9601c09520ee27032cd0fc0b13547e69654d76db"}
Oct 30 13:19:50.988551 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:50.988502    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8467ed-86997f5d8c-dxslk"
Oct 30 13:19:51.030740 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:51.030695    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8467ed-86997f5d8c-dxslk" podStartSLOduration=2.030682647 podStartE2EDuration="2.030682647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:19:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:19:51.027391118 +0000 UTC m=+1690.193755765" watchObservedRunningTime="2026-10-30 13:19:51.030682647 +0000 UTC m=+1690.197047295"
Oct 30 13:19:53.327445 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:53.327412    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-84942a-7d58989cc9-j2l2j"]
Oct 30 13:19:53.327952 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:53.327708    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84942a-7d58989cc9-j2l2j" podUID="644cd57e-9920-4049-a7f6-d4a41a5069e4" containerName="maas-api" containerID="cri-o://037ee5a283f37e7bd69f9b4da00e753a3288dabcc4f160139c97900077649b30" gracePeriod=30
Oct 30 13:19:53.996622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:53.996508    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84942a-7d58989cc9-j2l2j"
Oct 30 13:19:54.110423 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:54.101334    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/644cd57e-9920-4049-a7f6-d4a41a5069e4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/644cd57e-9920-4049-a7f6-d4a41a5069e4-metrics-certs\") pod \"644cd57e-9920-4049-a7f6-d4a41a5069e4\" (UID: \"644cd57e-9920-4049-a7f6-d4a41a5069e4\") "
Oct 30 13:19:54.110423 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:54.101510    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/644cd57e-9920-4049-a7f6-d4a41a5069e4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/644cd57e-9920-4049-a7f6-d4a41a5069e4-maas-api-tls\") pod \"644cd57e-9920-4049-a7f6-d4a41a5069e4\" (UID: \"644cd57e-9920-4049-a7f6-d4a41a5069e4\") "
Oct 30 13:19:54.110423 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:54.101573    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/644cd57e-9920-4049-a7f6-d4a41a5069e4-kube-api-access-mlqs5\" (UniqueName: \"kubernetes.io/projected/644cd57e-9920-4049-a7f6-d4a41a5069e4-kube-api-access-mlqs5\") pod \"644cd57e-9920-4049-a7f6-d4a41a5069e4\" (UID: \"644cd57e-9920-4049-a7f6-d4a41a5069e4\") "
Oct 30 13:19:54.131617 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:54.124775    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/644cd57e-9920-4049-a7f6-d4a41a5069e4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/644cd57e-9920-4049-a7f6-d4a41a5069e4-openshift-service-ca\") pod \"644cd57e-9920-4049-a7f6-d4a41a5069e4\" (UID: \"644cd57e-9920-4049-a7f6-d4a41a5069e4\") "
Oct 30 13:19:54.143030 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:54.142977    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/644cd57e-9920-4049-a7f6-d4a41a5069e4-openshift-service-ca" pod "644cd57e-9920-4049-a7f6-d4a41a5069e4" (UID: "644cd57e-9920-4049-a7f6-d4a41a5069e4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:19:54.168863 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:54.161258    4364 generic.go:356] "Generic (PLEG): container finished" podID="644cd57e-9920-4049-a7f6-d4a41a5069e4" containerID="037ee5a283f37e7bd69f9b4da00e753a3288dabcc4f160139c97900077649b30" exitCode=0
Oct 30 13:19:54.168863 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:54.161388    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84942a-7d58989cc9-j2l2j" event={"ID":"644cd57e-9920-4049-a7f6-d4a41a5069e4","Type":"ContainerDied","Data":"037ee5a283f37e7bd69f9b4da00e753a3288dabcc4f160139c97900077649b30"}
Oct 30 13:19:54.168863 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:54.161437    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84942a-7d58989cc9-j2l2j" event={"ID":"644cd57e-9920-4049-a7f6-d4a41a5069e4","Type":"ContainerDied","Data":"f7e4ae8a0f3253dd1bb2d5bb0331193bc9f09c5c55ff8494db9248c46a343796"}
Oct 30 13:19:54.168863 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:54.161465    4364 scope.go:122] "RemoveContainer" containerID="037ee5a283f37e7bd69f9b4da00e753a3288dabcc4f160139c97900077649b30"
Oct 30 13:19:54.168863 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:54.161654    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84942a-7d58989cc9-j2l2j"
Oct 30 13:19:54.234193 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:54.233005    4364 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/644cd57e-9920-4049-a7f6-d4a41a5069e4-openshift-service-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:19:54.404623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:54.402650    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/644cd57e-9920-4049-a7f6-d4a41a5069e4-kube-api-access-mlqs5" pod "644cd57e-9920-4049-a7f6-d4a41a5069e4" (UID: "644cd57e-9920-4049-a7f6-d4a41a5069e4"). InnerVolumeSpecName "kube-api-access-mlqs5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:19:54.432952 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:54.429863    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/644cd57e-9920-4049-a7f6-d4a41a5069e4-maas-api-tls" pod "644cd57e-9920-4049-a7f6-d4a41a5069e4" (UID: "644cd57e-9920-4049-a7f6-d4a41a5069e4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:19:54.432952 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:54.429938    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/644cd57e-9920-4049-a7f6-d4a41a5069e4-metrics-certs" pod "644cd57e-9920-4049-a7f6-d4a41a5069e4" (UID: "644cd57e-9920-4049-a7f6-d4a41a5069e4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:19:54.447015 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:54.446591    4364 scope.go:122] "RemoveContainer" containerID="037ee5a283f37e7bd69f9b4da00e753a3288dabcc4f160139c97900077649b30"
Oct 30 13:19:54.447185 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:19:54.447022    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"037ee5a283f37e7bd69f9b4da00e753a3288dabcc4f160139c97900077649b30\": container with ID starting with 037ee5a283f37e7bd69f9b4da00e753a3288dabcc4f160139c97900077649b30 not found: ID does not exist" containerID="037ee5a283f37e7bd69f9b4da00e753a3288dabcc4f160139c97900077649b30"
Oct 30 13:19:54.447185 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:54.447075    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"037ee5a283f37e7bd69f9b4da00e753a3288dabcc4f160139c97900077649b30"} err="failed to get container status \"037ee5a283f37e7bd69f9b4da00e753a3288dabcc4f160139c97900077649b30\": rpc error: code = NotFound desc = could not find container \"037ee5a283f37e7bd69f9b4da00e753a3288dabcc4f160139c97900077649b30\": container with ID starting with 037ee5a283f37e7bd69f9b4da00e753a3288dabcc4f160139c97900077649b30 not found: ID does not exist"
Oct 30 13:19:54.478521 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:54.478499    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlqs5\" (UniqueName: \"kubernetes.io/projected/644cd57e-9920-4049-a7f6-d4a41a5069e4-kube-api-access-mlqs5\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:19:54.478521 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:54.478522    4364 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/644cd57e-9920-4049-a7f6-d4a41a5069e4-metrics-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:19:54.478697 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:54.478531    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/644cd57e-9920-4049-a7f6-d4a41a5069e4-maas-api-tls\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:19:54.530770 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:54.530746    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-84942a-7d58989cc9-j2l2j"]
Oct 30 13:19:54.551400 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:54.551374    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-84942a-7d58989cc9-j2l2j"]
Oct 30 13:19:54.663421 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:54.663348    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m"]
Oct 30 13:19:54.663580 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:54.663558    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m" podUID="5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b" containerName="istio-proxy" containerID="cri-o://a1f8f42527e9131515c99a903756f40202e9748f6eb33325be6d202a389d9798" gracePeriod=30
Oct 30 13:19:55.002192 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:55.002107    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="644cd57e-9920-4049-a7f6-d4a41a5069e4" path="/var/lib/kubelet/pods/644cd57e-9920-4049-a7f6-d4a41a5069e4/volumes"
Oct 30 13:19:56.053679 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:56.053647    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f"]
Oct 30 13:19:56.054238 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:56.053840    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f" podUID="086afc53-5c53-42d9-bd7b-b803cc41ed6d" containerName="istio-proxy" containerID="cri-o://ffdeb9551c8182040b980864acb868a5d663b60720f0ecc8d9a610cecf1b979f" gracePeriod=30
Oct 30 13:19:56.777072 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:56.777031    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84942a-7d58989cc9-j2l2j" podUID="644cd57e-9920-4049-a7f6-d4a41a5069e4" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.162:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:19:58.071998 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:58.071967    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ff4285-76c67bfb9c-ckgmb"]
Oct 30 13:19:58.072792 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:58.072774    4364 cpu_manager.go:414] "containerMap: removing container" podUID="644cd57e-9920-4049-a7f6-d4a41a5069e4" containerName="maas-api"
Oct 30 13:19:58.072849 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:58.072793    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="644cd57e-9920-4049-a7f6-d4a41a5069e4" containerName="maas-api"
Oct 30 13:19:58.072935 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:58.072926    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="644cd57e-9920-4049-a7f6-d4a41a5069e4" containerName="maas-api"
Oct 30 13:19:58.076742 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:58.076721    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ff4285-76c67bfb9c-ckgmb"
Oct 30 13:19:58.099156 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:58.099081    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ff4285-76c67bfb9c-ckgmb"]
Oct 30 13:19:58.114156 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:58.114127    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-ff4285-serving-cert\""
Oct 30 13:19:58.168354 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:58.168319    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71576777-83eb-4fb6-af0e-33c177bc2dd0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-ff4285-76c67bfb9c-ckgmb\" (UID: \"71576777-83eb-4fb6-af0e-33c177bc2dd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ff4285-76c67bfb9c-ckgmb"
Oct 30 13:19:58.168514 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:58.168367    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71576777-83eb-4fb6-af0e-33c177bc2dd0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-ff4285-76c67bfb9c-ckgmb\" (UID: \"71576777-83eb-4fb6-af0e-33c177bc2dd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ff4285-76c67bfb9c-ckgmb"
Oct 30 13:19:58.168514 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:58.168497    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2frwx\" (UniqueName: \"kubernetes.io/projected/71576777-83eb-4fb6-af0e-33c177bc2dd0-kube-api-access-2frwx\") pod \"maas-api-e2e-shared-a-w1-ff4285-76c67bfb9c-ckgmb\" (UID: \"71576777-83eb-4fb6-af0e-33c177bc2dd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ff4285-76c67bfb9c-ckgmb"
Oct 30 13:19:58.168739 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:58.168721    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71576777-83eb-4fb6-af0e-33c177bc2dd0-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-ff4285-76c67bfb9c-ckgmb\" (UID: \"71576777-83eb-4fb6-af0e-33c177bc2dd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ff4285-76c67bfb9c-ckgmb"
Oct 30 13:19:58.300896 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:58.300855    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71576777-83eb-4fb6-af0e-33c177bc2dd0-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-ff4285-76c67bfb9c-ckgmb\" (UID: \"71576777-83eb-4fb6-af0e-33c177bc2dd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ff4285-76c67bfb9c-ckgmb"
Oct 30 13:19:58.301259 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:58.301235    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71576777-83eb-4fb6-af0e-33c177bc2dd0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-ff4285-76c67bfb9c-ckgmb\" (UID: \"71576777-83eb-4fb6-af0e-33c177bc2dd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ff4285-76c67bfb9c-ckgmb"
Oct 30 13:19:58.301411 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:58.301396    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71576777-83eb-4fb6-af0e-33c177bc2dd0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-ff4285-76c67bfb9c-ckgmb\" (UID: \"71576777-83eb-4fb6-af0e-33c177bc2dd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ff4285-76c67bfb9c-ckgmb"
Oct 30 13:19:58.301584 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:58.301566    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2frwx\" (UniqueName: \"kubernetes.io/projected/71576777-83eb-4fb6-af0e-33c177bc2dd0-kube-api-access-2frwx\") pod \"maas-api-e2e-shared-a-w1-ff4285-76c67bfb9c-ckgmb\" (UID: \"71576777-83eb-4fb6-af0e-33c177bc2dd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ff4285-76c67bfb9c-ckgmb"
Oct 30 13:19:58.314852 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:58.314817    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71576777-83eb-4fb6-af0e-33c177bc2dd0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-ff4285-76c67bfb9c-ckgmb\" (UID: \"71576777-83eb-4fb6-af0e-33c177bc2dd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ff4285-76c67bfb9c-ckgmb"
Oct 30 13:19:58.332283 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:58.332207    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71576777-83eb-4fb6-af0e-33c177bc2dd0-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-ff4285-76c67bfb9c-ckgmb\" (UID: \"71576777-83eb-4fb6-af0e-33c177bc2dd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ff4285-76c67bfb9c-ckgmb"
Oct 30 13:19:58.364635 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:58.364579    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71576777-83eb-4fb6-af0e-33c177bc2dd0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-ff4285-76c67bfb9c-ckgmb\" (UID: \"71576777-83eb-4fb6-af0e-33c177bc2dd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ff4285-76c67bfb9c-ckgmb"
Oct 30 13:19:58.399390 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:58.399353    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2frwx\" (UniqueName: \"kubernetes.io/projected/71576777-83eb-4fb6-af0e-33c177bc2dd0-kube-api-access-2frwx\") pod \"maas-api-e2e-shared-a-w1-ff4285-76c67bfb9c-ckgmb\" (UID: \"71576777-83eb-4fb6-af0e-33c177bc2dd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ff4285-76c67bfb9c-ckgmb"
Oct 30 13:19:58.406619 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:58.406572    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ff4285-76c67bfb9c-ckgmb"
Oct 30 13:19:58.719244 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:58.719217    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ff4285-76c67bfb9c-ckgmb"]
Oct 30 13:19:58.719356 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:19:58.719337    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71576777_83eb_4fb6_af0e_33c177bc2dd0.slice/crio-531e2707b7fbf1da5b976259b6e501bde6c0b3a79a6b289caa6b68414c51aefe WatchSource:0}: Error finding container 531e2707b7fbf1da5b976259b6e501bde6c0b3a79a6b289caa6b68414c51aefe: Status 404 returned error can't find the container with id 531e2707b7fbf1da5b976259b6e501bde6c0b3a79a6b289caa6b68414c51aefe
Oct 30 13:19:59.236421 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:59.236277    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ff4285-76c67bfb9c-ckgmb" event={"ID":"71576777-83eb-4fb6-af0e-33c177bc2dd0","Type":"ContainerStarted","Data":"30e4c2140211aab0993d7b43c934efbb8fce752dc0dc96c3c461a44620053334"}
Oct 30 13:19:59.236421 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:59.236353    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ff4285-76c67bfb9c-ckgmb" event={"ID":"71576777-83eb-4fb6-af0e-33c177bc2dd0","Type":"ContainerStarted","Data":"531e2707b7fbf1da5b976259b6e501bde6c0b3a79a6b289caa6b68414c51aefe"}
Oct 30 13:19:59.236421 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:59.236407    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ff4285-76c67bfb9c-ckgmb"
Oct 30 13:19:59.294478 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:59.294423    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ff4285-76c67bfb9c-ckgmb" podStartSLOduration=0.915184714 podStartE2EDuration="1.29440567s" totalImagesPullingTime="379.220956ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:19:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:19:59.285795903 +0000 UTC m=+1698.452160562" watchObservedRunningTime="2026-10-30 13:19:59.29440567 +0000 UTC m=+1698.460770317"
Oct 30 13:19:59.707806 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:59.707772    4364 patch_prober.go:29] interesting pod/e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 13:19:59.707963 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:59.707834    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m" podUID="5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 13:19:59.977667 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:19:59.977639    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m"
Oct 30 13:20:00.042790 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:00.042755    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b-istio-data\") pod \"5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b\" (UID: \"5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b\") "
Oct 30 13:20:00.043097 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:00.043079    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b-istio-envoy\") pod \"5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b\" (UID: \"5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b\") "
Oct 30 13:20:00.043301 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:00.043137    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b-istio-data" pod "5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b" (UID: "5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:20:00.043430 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:00.043411    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b-workload-certs\") pod \"5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b\" (UID: \"5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b\") "
Oct 30 13:20:00.043574 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:00.043559    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b-istiod-ca-cert\") pod \"5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b\" (UID: \"5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b\") "
Oct 30 13:20:00.043792 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:00.043773    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b-credential-socket\") pod \"5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b\" (UID: \"5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b\") "
Oct 30 13:20:00.043984 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:00.043851    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b-workload-certs" pod "5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b" (UID: "5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:20:00.044058 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:00.044012    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b-istiod-ca-cert" pod "5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b" (UID: "5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:20:00.044971 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:00.044070    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b-credential-socket" pod "5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b" (UID: "5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:20:00.044971 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:00.044149    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b-istio-podinfo\") pod \"5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b\" (UID: \"5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b\") "
Oct 30 13:20:00.044971 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:00.044301    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b-workload-socket\") pod \"5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b\" (UID: \"5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b\") "
Oct 30 13:20:00.044971 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:00.044360    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b-kube-api-access-r5h9s\" (UniqueName: \"kubernetes.io/projected/5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b-kube-api-access-r5h9s\") pod \"5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b\" (UID: \"5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b\") "
Oct 30 13:20:00.044971 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:00.044445    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b-istio-token\" (UniqueName: \"kubernetes.io/projected/5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b-istio-token\") pod \"5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b\" (UID: \"5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b\") "
Oct 30 13:20:00.044971 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:00.044893    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b-workload-socket" pod "5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b" (UID: "5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:20:00.045636 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:00.045614    4364 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b-istio-data\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:20:00.046239 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:00.045643    4364 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b-workload-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:20:00.046239 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:00.045657    4364 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b-istiod-ca-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:20:00.046239 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:00.045670    4364 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b-credential-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:20:00.046239 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:00.045683    4364 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b-workload-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:20:00.063620 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:00.048753    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b-istio-envoy" pod "5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b" (UID: "5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:20:00.063620 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:00.059861    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b-istio-token" pod "5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b" (UID: "5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:20:00.063620 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:00.061693    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b-istio-podinfo" pod "5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b" (UID: "5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:20:00.063620 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:00.063054    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b-kube-api-access-r5h9s" pod "5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b" (UID: "5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b"). InnerVolumeSpecName "kube-api-access-r5h9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:20:00.147318 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:00.147278    4364 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b-istio-podinfo\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:20:00.147318 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:00.147313    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5h9s\" (UniqueName: \"kubernetes.io/projected/5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b-kube-api-access-r5h9s\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:20:00.147318 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:00.147323    4364 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b-istio-token\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:20:00.147589 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:00.147332    4364 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b-istio-envoy\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:20:00.248088 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:00.247997    4364 generic.go:356] "Generic (PLEG): container finished" podID="5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b" containerID="a1f8f42527e9131515c99a903756f40202e9748f6eb33325be6d202a389d9798" exitCode=0
Oct 30 13:20:00.248088 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:00.248067    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m"
Oct 30 13:20:00.248590 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:00.248095    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m" event={"ID":"5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b","Type":"ContainerDied","Data":"a1f8f42527e9131515c99a903756f40202e9748f6eb33325be6d202a389d9798"}
Oct 30 13:20:00.248590 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:00.248162    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m" event={"ID":"5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b","Type":"ContainerDied","Data":"3d65b03acd7ed56ba1441413bf978b8fb2f575ee319f04d6489532d28d1e745e"}
Oct 30 13:20:00.248590 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:00.248189    4364 scope.go:122] "RemoveContainer" containerID="a1f8f42527e9131515c99a903756f40202e9748f6eb33325be6d202a389d9798"
Oct 30 13:20:00.264950 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:00.264933    4364 scope.go:122] "RemoveContainer" containerID="a1f8f42527e9131515c99a903756f40202e9748f6eb33325be6d202a389d9798"
Oct 30 13:20:00.265258 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:20:00.265234    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1f8f42527e9131515c99a903756f40202e9748f6eb33325be6d202a389d9798\": container with ID starting with a1f8f42527e9131515c99a903756f40202e9748f6eb33325be6d202a389d9798 not found: ID does not exist" containerID="a1f8f42527e9131515c99a903756f40202e9748f6eb33325be6d202a389d9798"
Oct 30 13:20:00.265363 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:00.265264    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1f8f42527e9131515c99a903756f40202e9748f6eb33325be6d202a389d9798"} err="failed to get container status \"a1f8f42527e9131515c99a903756f40202e9748f6eb33325be6d202a389d9798\": rpc error: code = NotFound desc = could not find container \"a1f8f42527e9131515c99a903756f40202e9748f6eb33325be6d202a389d9798\": container with ID starting with a1f8f42527e9131515c99a903756f40202e9748f6eb33325be6d202a389d9798 not found: ID does not exist"
Oct 30 13:20:00.351868 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:00.347712    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m"]
Oct 30 13:20:00.360647 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:00.358795    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-c4j8m"]
Oct 30 13:20:00.771732 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:00.771700    4364 patch_prober.go:29] interesting pod/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:20:00.771901 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:00.771814    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd" podUID="0f77c6bd-36f6-45cc-92ed-b22fa718700a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:20:00.973122 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:00.973093    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b" path="/var/lib/kubelet/pods/5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b/volumes"
Oct 30 13:20:01.276781 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:01.276748    4364 generic.go:356] "Generic (PLEG): container finished" podID="086afc53-5c53-42d9-bd7b-b803cc41ed6d" containerID="ffdeb9551c8182040b980864acb868a5d663b60720f0ecc8d9a610cecf1b979f" exitCode=0
Oct 30 13:20:01.277251 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:01.276843    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f" event={"ID":"086afc53-5c53-42d9-bd7b-b803cc41ed6d","Type":"ContainerDied","Data":"ffdeb9551c8182040b980864acb868a5d663b60720f0ecc8d9a610cecf1b979f"}
Oct 30 13:20:01.369223 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:01.369201    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f"
Oct 30 13:20:01.469353 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:01.469324    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/086afc53-5c53-42d9-bd7b-b803cc41ed6d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/086afc53-5c53-42d9-bd7b-b803cc41ed6d-workload-socket\") pod \"086afc53-5c53-42d9-bd7b-b803cc41ed6d\" (UID: \"086afc53-5c53-42d9-bd7b-b803cc41ed6d\") "
Oct 30 13:20:01.469505 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:01.469400    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/086afc53-5c53-42d9-bd7b-b803cc41ed6d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/086afc53-5c53-42d9-bd7b-b803cc41ed6d-istiod-ca-cert\") pod \"086afc53-5c53-42d9-bd7b-b803cc41ed6d\" (UID: \"086afc53-5c53-42d9-bd7b-b803cc41ed6d\") "
Oct 30 13:20:01.469584 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:01.469572    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/086afc53-5c53-42d9-bd7b-b803cc41ed6d-istio-token\" (UniqueName: \"kubernetes.io/projected/086afc53-5c53-42d9-bd7b-b803cc41ed6d-istio-token\") pod \"086afc53-5c53-42d9-bd7b-b803cc41ed6d\" (UID: \"086afc53-5c53-42d9-bd7b-b803cc41ed6d\") "
Oct 30 13:20:01.469664 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:01.469636    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/086afc53-5c53-42d9-bd7b-b803cc41ed6d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/086afc53-5c53-42d9-bd7b-b803cc41ed6d-workload-certs\") pod \"086afc53-5c53-42d9-bd7b-b803cc41ed6d\" (UID: \"086afc53-5c53-42d9-bd7b-b803cc41ed6d\") "
Oct 30 13:20:01.469664 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:01.469618    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/086afc53-5c53-42d9-bd7b-b803cc41ed6d-workload-socket" pod "086afc53-5c53-42d9-bd7b-b803cc41ed6d" (UID: "086afc53-5c53-42d9-bd7b-b803cc41ed6d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:20:01.469781 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:01.469704    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/086afc53-5c53-42d9-bd7b-b803cc41ed6d-kube-api-access-5z5wr\" (UniqueName: \"kubernetes.io/projected/086afc53-5c53-42d9-bd7b-b803cc41ed6d-kube-api-access-5z5wr\") pod \"086afc53-5c53-42d9-bd7b-b803cc41ed6d\" (UID: \"086afc53-5c53-42d9-bd7b-b803cc41ed6d\") "
Oct 30 13:20:01.469781 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:01.469744    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/086afc53-5c53-42d9-bd7b-b803cc41ed6d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/086afc53-5c53-42d9-bd7b-b803cc41ed6d-credential-socket\") pod \"086afc53-5c53-42d9-bd7b-b803cc41ed6d\" (UID: \"086afc53-5c53-42d9-bd7b-b803cc41ed6d\") "
Oct 30 13:20:01.469882 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:01.469778    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/086afc53-5c53-42d9-bd7b-b803cc41ed6d-istiod-ca-cert" pod "086afc53-5c53-42d9-bd7b-b803cc41ed6d" (UID: "086afc53-5c53-42d9-bd7b-b803cc41ed6d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:20:01.469882 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:01.469799    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/086afc53-5c53-42d9-bd7b-b803cc41ed6d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/086afc53-5c53-42d9-bd7b-b803cc41ed6d-istio-data\") pod \"086afc53-5c53-42d9-bd7b-b803cc41ed6d\" (UID: \"086afc53-5c53-42d9-bd7b-b803cc41ed6d\") "
Oct 30 13:20:01.469882 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:01.469817    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/086afc53-5c53-42d9-bd7b-b803cc41ed6d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/086afc53-5c53-42d9-bd7b-b803cc41ed6d-istio-envoy\") pod \"086afc53-5c53-42d9-bd7b-b803cc41ed6d\" (UID: \"086afc53-5c53-42d9-bd7b-b803cc41ed6d\") "
Oct 30 13:20:01.469882 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:01.469856    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/086afc53-5c53-42d9-bd7b-b803cc41ed6d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/086afc53-5c53-42d9-bd7b-b803cc41ed6d-istio-podinfo\") pod \"086afc53-5c53-42d9-bd7b-b803cc41ed6d\" (UID: \"086afc53-5c53-42d9-bd7b-b803cc41ed6d\") "
Oct 30 13:20:01.470096 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:01.469900    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/086afc53-5c53-42d9-bd7b-b803cc41ed6d-workload-certs" pod "086afc53-5c53-42d9-bd7b-b803cc41ed6d" (UID: "086afc53-5c53-42d9-bd7b-b803cc41ed6d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:20:01.470096 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:01.470078    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/086afc53-5c53-42d9-bd7b-b803cc41ed6d-istio-data" pod "086afc53-5c53-42d9-bd7b-b803cc41ed6d" (UID: "086afc53-5c53-42d9-bd7b-b803cc41ed6d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:20:01.470418 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:01.470392    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/086afc53-5c53-42d9-bd7b-b803cc41ed6d-credential-socket" pod "086afc53-5c53-42d9-bd7b-b803cc41ed6d" (UID: "086afc53-5c53-42d9-bd7b-b803cc41ed6d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:20:01.470701 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:01.470678    4364 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/086afc53-5c53-42d9-bd7b-b803cc41ed6d-workload-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:20:01.470802 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:01.470703    4364 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/086afc53-5c53-42d9-bd7b-b803cc41ed6d-istiod-ca-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:20:01.470802 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:01.470716    4364 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/086afc53-5c53-42d9-bd7b-b803cc41ed6d-workload-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:20:01.470802 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:01.470727    4364 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/086afc53-5c53-42d9-bd7b-b803cc41ed6d-credential-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:20:01.470802 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:01.470739    4364 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/086afc53-5c53-42d9-bd7b-b803cc41ed6d-istio-data\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:20:01.472515 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:01.472490    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/086afc53-5c53-42d9-bd7b-b803cc41ed6d-istio-envoy" pod "086afc53-5c53-42d9-bd7b-b803cc41ed6d" (UID: "086afc53-5c53-42d9-bd7b-b803cc41ed6d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:20:01.472624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:01.472553    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/086afc53-5c53-42d9-bd7b-b803cc41ed6d-kube-api-access-5z5wr" pod "086afc53-5c53-42d9-bd7b-b803cc41ed6d" (UID: "086afc53-5c53-42d9-bd7b-b803cc41ed6d"). InnerVolumeSpecName "kube-api-access-5z5wr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:20:01.472869 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:01.472841    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/086afc53-5c53-42d9-bd7b-b803cc41ed6d-istio-token" pod "086afc53-5c53-42d9-bd7b-b803cc41ed6d" (UID: "086afc53-5c53-42d9-bd7b-b803cc41ed6d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:20:01.473223 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:01.473196    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/086afc53-5c53-42d9-bd7b-b803cc41ed6d-istio-podinfo" pod "086afc53-5c53-42d9-bd7b-b803cc41ed6d" (UID: "086afc53-5c53-42d9-bd7b-b803cc41ed6d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:20:01.572050 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:01.572014    4364 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/086afc53-5c53-42d9-bd7b-b803cc41ed6d-istio-token\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:20:01.572050 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:01.572045    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z5wr\" (UniqueName: \"kubernetes.io/projected/086afc53-5c53-42d9-bd7b-b803cc41ed6d-kube-api-access-5z5wr\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:20:01.572050 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:01.572054    4364 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/086afc53-5c53-42d9-bd7b-b803cc41ed6d-istio-envoy\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:20:01.572279 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:01.572064    4364 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/086afc53-5c53-42d9-bd7b-b803cc41ed6d-istio-podinfo\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:20:02.060198 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:02.060173    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8467ed-86997f5d8c-dxslk"
Oct 30 13:20:02.332376 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:02.332127    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f" event={"ID":"086afc53-5c53-42d9-bd7b-b803cc41ed6d","Type":"ContainerDied","Data":"c2e6b32bf747756b675e0662d9ba84a022c55e0630b2269ead7c3625a37b9a02"}
Oct 30 13:20:02.332376 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:02.332205    4364 scope.go:122] "RemoveContainer" containerID="ffdeb9551c8182040b980864acb868a5d663b60720f0ecc8d9a610cecf1b979f"
Oct 30 13:20:02.332930 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:02.332404    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f"
Oct 30 13:20:02.403531 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:02.403454    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f"]
Oct 30 13:20:02.414931 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:02.414899    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-f575f"]
Oct 30 13:20:02.972397 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:02.972326    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="086afc53-5c53-42d9-bd7b-b803cc41ed6d" path="/var/lib/kubelet/pods/086afc53-5c53-42d9-bd7b-b803cc41ed6d/volumes"
Oct 30 13:20:03.244682 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:03.244652    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-84942a-openshift-default-58974cd687-wtjlv"]
Oct 30 13:20:03.244902 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:03.244876    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-84942a-openshift-default-58974cd687-wtjlv" podUID="900b00e0-ab94-4293-ade2-7e89bd313fea" containerName="istio-proxy" containerID="cri-o://7f8d4538497a7934ffdd59df1c4840ac99336ca49c873227643a5e716acd7885" gracePeriod=30
Oct 30 13:20:03.442617 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:03.441883    4364 generic.go:356] "Generic (PLEG): container finished" podID="0f77c6bd-36f6-45cc-92ed-b22fa718700a" containerID="844465097f0200cdee28e6245e758d4eccfe729bfcb546db88580734c0d748d6" exitCode=0
Oct 30 13:20:03.442617 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:03.441972    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd" event={"ID":"0f77c6bd-36f6-45cc-92ed-b22fa718700a","Type":"ContainerDied","Data":"844465097f0200cdee28e6245e758d4eccfe729bfcb546db88580734c0d748d6"}
Oct 30 13:20:03.458656 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:03.443171    4364 scope.go:122] "RemoveContainer" containerID="844465097f0200cdee28e6245e758d4eccfe729bfcb546db88580734c0d748d6"
Oct 30 13:20:03.480573 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:03.480513    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:20:03.480893 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:03.480867    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:20:03.481077 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:03.481038    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:20:04.677152 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:04.675903    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd" event={"ID":"0f77c6bd-36f6-45cc-92ed-b22fa718700a","Type":"ContainerStarted","Data":"62dc970ba4997d047018b450364fb355469c1de8020f1513bd1986102e16542f"}
Oct 30 13:20:05.282654 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:05.282622    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ff4285-76c67bfb9c-ckgmb"
Oct 30 13:20:05.583005 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:05.582917    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd"
Oct 30 13:20:05.583470 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:05.583434    4364 patch_prober.go:29] interesting pod/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Oct 30 13:20:05.587621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:05.583635    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd" podUID="0f77c6bd-36f6-45cc-92ed-b22fa718700a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Oct 30 13:20:06.597622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:06.590696    4364 patch_prober.go:29] interesting pod/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Oct 30 13:20:06.597622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:06.590768    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd" podUID="0f77c6bd-36f6-45cc-92ed-b22fa718700a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Oct 30 13:20:07.604706 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:07.604668    4364 patch_prober.go:29] interesting pod/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Oct 30 13:20:07.629623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:07.627701    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd" podUID="0f77c6bd-36f6-45cc-92ed-b22fa718700a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Oct 30 13:20:08.604163 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:08.591887    4364 patch_prober.go:29] interesting pod/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Oct 30 13:20:08.604163 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:08.603691    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd" podUID="0f77c6bd-36f6-45cc-92ed-b22fa718700a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Oct 30 13:20:08.612172 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:08.612152    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-84942a-openshift-default-58974cd687-wtjlv"
Oct 30 13:20:08.748500 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:08.748469    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/900b00e0-ab94-4293-ade2-7e89bd313fea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/900b00e0-ab94-4293-ade2-7e89bd313fea-istio-data\") pod \"900b00e0-ab94-4293-ade2-7e89bd313fea\" (UID: \"900b00e0-ab94-4293-ade2-7e89bd313fea\") "
Oct 30 13:20:08.748726 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:08.748576    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/900b00e0-ab94-4293-ade2-7e89bd313fea-istio-token\" (UniqueName: \"kubernetes.io/projected/900b00e0-ab94-4293-ade2-7e89bd313fea-istio-token\") pod \"900b00e0-ab94-4293-ade2-7e89bd313fea\" (UID: \"900b00e0-ab94-4293-ade2-7e89bd313fea\") "
Oct 30 13:20:08.748803 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:08.748785    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/900b00e0-ab94-4293-ade2-7e89bd313fea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/900b00e0-ab94-4293-ade2-7e89bd313fea-istio-podinfo\") pod \"900b00e0-ab94-4293-ade2-7e89bd313fea\" (UID: \"900b00e0-ab94-4293-ade2-7e89bd313fea\") "
Oct 30 13:20:08.748803 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:08.748786    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/900b00e0-ab94-4293-ade2-7e89bd313fea-istio-data" pod "900b00e0-ab94-4293-ade2-7e89bd313fea" (UID: "900b00e0-ab94-4293-ade2-7e89bd313fea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:20:08.748913 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:08.748876    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/900b00e0-ab94-4293-ade2-7e89bd313fea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/900b00e0-ab94-4293-ade2-7e89bd313fea-workload-certs\") pod \"900b00e0-ab94-4293-ade2-7e89bd313fea\" (UID: \"900b00e0-ab94-4293-ade2-7e89bd313fea\") "
Oct 30 13:20:08.749012 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:08.748995    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/900b00e0-ab94-4293-ade2-7e89bd313fea-kube-api-access-pdsjz\" (UniqueName: \"kubernetes.io/projected/900b00e0-ab94-4293-ade2-7e89bd313fea-kube-api-access-pdsjz\") pod \"900b00e0-ab94-4293-ade2-7e89bd313fea\" (UID: \"900b00e0-ab94-4293-ade2-7e89bd313fea\") "
Oct 30 13:20:08.749091 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:08.749036    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/900b00e0-ab94-4293-ade2-7e89bd313fea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/900b00e0-ab94-4293-ade2-7e89bd313fea-workload-socket\") pod \"900b00e0-ab94-4293-ade2-7e89bd313fea\" (UID: \"900b00e0-ab94-4293-ade2-7e89bd313fea\") "
Oct 30 13:20:08.749146 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:08.749112    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/900b00e0-ab94-4293-ade2-7e89bd313fea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/900b00e0-ab94-4293-ade2-7e89bd313fea-credential-socket\") pod \"900b00e0-ab94-4293-ade2-7e89bd313fea\" (UID: \"900b00e0-ab94-4293-ade2-7e89bd313fea\") "
Oct 30 13:20:08.749207 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:08.749151    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/900b00e0-ab94-4293-ade2-7e89bd313fea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/900b00e0-ab94-4293-ade2-7e89bd313fea-istiod-ca-cert\") pod \"900b00e0-ab94-4293-ade2-7e89bd313fea\" (UID: \"900b00e0-ab94-4293-ade2-7e89bd313fea\") "
Oct 30 13:20:08.749207 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:08.749188    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/900b00e0-ab94-4293-ade2-7e89bd313fea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/900b00e0-ab94-4293-ade2-7e89bd313fea-istio-envoy\") pod \"900b00e0-ab94-4293-ade2-7e89bd313fea\" (UID: \"900b00e0-ab94-4293-ade2-7e89bd313fea\") "
Oct 30 13:20:08.749315 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:08.749115    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/900b00e0-ab94-4293-ade2-7e89bd313fea-workload-certs" pod "900b00e0-ab94-4293-ade2-7e89bd313fea" (UID: "900b00e0-ab94-4293-ade2-7e89bd313fea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:20:08.749371 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:08.749307    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/900b00e0-ab94-4293-ade2-7e89bd313fea-workload-socket" pod "900b00e0-ab94-4293-ade2-7e89bd313fea" (UID: "900b00e0-ab94-4293-ade2-7e89bd313fea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:20:08.749371 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:08.749330    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/900b00e0-ab94-4293-ade2-7e89bd313fea-credential-socket" pod "900b00e0-ab94-4293-ade2-7e89bd313fea" (UID: "900b00e0-ab94-4293-ade2-7e89bd313fea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:20:08.749587 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:08.749557    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/900b00e0-ab94-4293-ade2-7e89bd313fea-istiod-ca-cert" pod "900b00e0-ab94-4293-ade2-7e89bd313fea" (UID: "900b00e0-ab94-4293-ade2-7e89bd313fea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:20:08.750161 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:08.750134    4364 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/900b00e0-ab94-4293-ade2-7e89bd313fea-workload-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:20:08.750282 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:08.750167    4364 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/900b00e0-ab94-4293-ade2-7e89bd313fea-workload-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:20:08.750282 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:08.750181    4364 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/900b00e0-ab94-4293-ade2-7e89bd313fea-credential-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:20:08.750282 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:08.750199    4364 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/900b00e0-ab94-4293-ade2-7e89bd313fea-istiod-ca-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:20:08.750282 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:08.750213    4364 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/900b00e0-ab94-4293-ade2-7e89bd313fea-istio-data\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:20:08.751044 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:08.751022    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/900b00e0-ab94-4293-ade2-7e89bd313fea-istio-podinfo" pod "900b00e0-ab94-4293-ade2-7e89bd313fea" (UID: "900b00e0-ab94-4293-ade2-7e89bd313fea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:20:08.751165 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:08.751129    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/900b00e0-ab94-4293-ade2-7e89bd313fea-istio-token" pod "900b00e0-ab94-4293-ade2-7e89bd313fea" (UID: "900b00e0-ab94-4293-ade2-7e89bd313fea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:20:08.751232 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:08.751191    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/900b00e0-ab94-4293-ade2-7e89bd313fea-kube-api-access-pdsjz" pod "900b00e0-ab94-4293-ade2-7e89bd313fea" (UID: "900b00e0-ab94-4293-ade2-7e89bd313fea"). InnerVolumeSpecName "kube-api-access-pdsjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:20:08.751294 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:08.751269    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/900b00e0-ab94-4293-ade2-7e89bd313fea-istio-envoy" pod "900b00e0-ab94-4293-ade2-7e89bd313fea" (UID: "900b00e0-ab94-4293-ade2-7e89bd313fea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:20:08.814090 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:08.813996    4364 generic.go:356] "Generic (PLEG): container finished" podID="900b00e0-ab94-4293-ade2-7e89bd313fea" containerID="7f8d4538497a7934ffdd59df1c4840ac99336ca49c873227643a5e716acd7885" exitCode=0
Oct 30 13:20:08.814090 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:08.814071    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-84942a-openshift-default-58974cd687-wtjlv"
Oct 30 13:20:08.814090 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:08.814073    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-84942a-openshift-default-58974cd687-wtjlv" event={"ID":"900b00e0-ab94-4293-ade2-7e89bd313fea","Type":"ContainerDied","Data":"7f8d4538497a7934ffdd59df1c4840ac99336ca49c873227643a5e716acd7885"}
Oct 30 13:20:08.814390 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:08.814171    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-84942a-openshift-default-58974cd687-wtjlv" event={"ID":"900b00e0-ab94-4293-ade2-7e89bd313fea","Type":"ContainerDied","Data":"27a334d3a761c8a7888c29a7a1c5d444fe3dec8fe120d281cc075c01d8545d3b"}
Oct 30 13:20:08.814390 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:08.814204    4364 scope.go:122] "RemoveContainer" containerID="7f8d4538497a7934ffdd59df1c4840ac99336ca49c873227643a5e716acd7885"
Oct 30 13:20:08.826260 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:08.826242    4364 scope.go:122] "RemoveContainer" containerID="d9ce283f4848e306aa5e2b66efbbc4b69e655759246703e2c231008f4c27143f"
Oct 30 13:20:08.836484 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:08.836468    4364 scope.go:122] "RemoveContainer" containerID="7f8d4538497a7934ffdd59df1c4840ac99336ca49c873227643a5e716acd7885"
Oct 30 13:20:08.836764 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:20:08.836747    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f8d4538497a7934ffdd59df1c4840ac99336ca49c873227643a5e716acd7885\": container with ID starting with 7f8d4538497a7934ffdd59df1c4840ac99336ca49c873227643a5e716acd7885 not found: ID does not exist" containerID="7f8d4538497a7934ffdd59df1c4840ac99336ca49c873227643a5e716acd7885"
Oct 30 13:20:08.836852 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:08.836777    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f8d4538497a7934ffdd59df1c4840ac99336ca49c873227643a5e716acd7885"} err="failed to get container status \"7f8d4538497a7934ffdd59df1c4840ac99336ca49c873227643a5e716acd7885\": rpc error: code = NotFound desc = could not find container \"7f8d4538497a7934ffdd59df1c4840ac99336ca49c873227643a5e716acd7885\": container with ID starting with 7f8d4538497a7934ffdd59df1c4840ac99336ca49c873227643a5e716acd7885 not found: ID does not exist"
Oct 30 13:20:08.836852 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:08.836799    4364 scope.go:122] "RemoveContainer" containerID="d9ce283f4848e306aa5e2b66efbbc4b69e655759246703e2c231008f4c27143f"
Oct 30 13:20:08.837061 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:20:08.837038    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9ce283f4848e306aa5e2b66efbbc4b69e655759246703e2c231008f4c27143f\": container with ID starting with d9ce283f4848e306aa5e2b66efbbc4b69e655759246703e2c231008f4c27143f not found: ID does not exist" containerID="d9ce283f4848e306aa5e2b66efbbc4b69e655759246703e2c231008f4c27143f"
Oct 30 13:20:08.837115 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:08.837071    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9ce283f4848e306aa5e2b66efbbc4b69e655759246703e2c231008f4c27143f"} err="failed to get container status \"d9ce283f4848e306aa5e2b66efbbc4b69e655759246703e2c231008f4c27143f\": rpc error: code = NotFound desc = could not find container \"d9ce283f4848e306aa5e2b66efbbc4b69e655759246703e2c231008f4c27143f\": container with ID starting with d9ce283f4848e306aa5e2b66efbbc4b69e655759246703e2c231008f4c27143f not found: ID does not exist"
Oct 30 13:20:08.851391 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:08.851368    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdsjz\" (UniqueName: \"kubernetes.io/projected/900b00e0-ab94-4293-ade2-7e89bd313fea-kube-api-access-pdsjz\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:20:08.851481 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:08.851393    4364 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/900b00e0-ab94-4293-ade2-7e89bd313fea-istio-envoy\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:20:08.851481 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:08.851403    4364 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/900b00e0-ab94-4293-ade2-7e89bd313fea-istio-token\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:20:08.851481 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:08.851412    4364 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/900b00e0-ab94-4293-ade2-7e89bd313fea-istio-podinfo\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:20:08.868505 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:08.868485    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-84942a-openshift-default-58974cd687-wtjlv"]
Oct 30 13:20:08.873590 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:08.873571    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-84942a-openshift-default-58974cd687-wtjlv"]
Oct 30 13:20:08.979618 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:08.977885    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="900b00e0-ab94-4293-ade2-7e89bd313fea" path="/var/lib/kubelet/pods/900b00e0-ab94-4293-ade2-7e89bd313fea/volumes"
Oct 30 13:20:09.590639 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:09.582931    4364 patch_prober.go:29] interesting pod/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Oct 30 13:20:09.590639 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:09.583004    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd" podUID="0f77c6bd-36f6-45cc-92ed-b22fa718700a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Oct 30 13:20:10.582622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:10.580560    4364 patch_prober.go:29] interesting pod/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Oct 30 13:20:10.582622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:10.580648    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd" podUID="0f77c6bd-36f6-45cc-92ed-b22fa718700a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Oct 30 13:20:10.994588 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:10.994557    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-47628d-668fb856c4-q5lp4"]
Oct 30 13:20:10.995346 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:10.995330    4364 cpu_manager.go:414] "containerMap: removing container" podUID="5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b" containerName="istio-proxy"
Oct 30 13:20:10.995389 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:10.995348    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b" containerName="istio-proxy"
Oct 30 13:20:10.995389 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:10.995370    4364 cpu_manager.go:414] "containerMap: removing container" podUID="086afc53-5c53-42d9-bd7b-b803cc41ed6d" containerName="istio-proxy"
Oct 30 13:20:10.995389 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:10.995376    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="086afc53-5c53-42d9-bd7b-b803cc41ed6d" containerName="istio-proxy"
Oct 30 13:20:10.995550 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:10.995389    4364 cpu_manager.go:414] "containerMap: removing container" podUID="900b00e0-ab94-4293-ade2-7e89bd313fea" containerName="istio-proxy"
Oct 30 13:20:10.995550 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:10.995395    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="900b00e0-ab94-4293-ade2-7e89bd313fea" containerName="istio-proxy"
Oct 30 13:20:10.995550 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:10.995407    4364 cpu_manager.go:414] "containerMap: removing container" podUID="900b00e0-ab94-4293-ade2-7e89bd313fea" containerName="istio-proxy"
Oct 30 13:20:10.995550 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:10.995412    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="900b00e0-ab94-4293-ade2-7e89bd313fea" containerName="istio-proxy"
Oct 30 13:20:10.995763 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:10.995590    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f4c0d4c-eaaf-45bd-b2e5-ea18da9fe89b" containerName="istio-proxy"
Oct 30 13:20:10.995763 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:10.995629    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="086afc53-5c53-42d9-bd7b-b803cc41ed6d" containerName="istio-proxy"
Oct 30 13:20:10.995763 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:10.995644    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="900b00e0-ab94-4293-ade2-7e89bd313fea" containerName="istio-proxy"
Oct 30 13:20:10.995763 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:10.995655    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="900b00e0-ab94-4293-ade2-7e89bd313fea" containerName="istio-proxy"
Oct 30 13:20:10.999730 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:10.999713    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-47628d-668fb856c4-q5lp4"
Oct 30 13:20:11.023203 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:11.023180    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-47628d-668fb856c4-q5lp4"]
Oct 30 13:20:11.026170 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:11.026148    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-47628d\""
Oct 30 13:20:11.084931 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:11.084903    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-47628d-6c75669467-c66k4"]
Oct 30 13:20:11.088460 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:11.088441    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-47628d-6c75669467-c66k4"
Oct 30 13:20:11.108375 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:11.108358    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-47628d-dockercfg-wkpcr\""
Oct 30 13:20:11.113543 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:11.113523    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-47628d-6c75669467-c66k4"]
Oct 30 13:20:11.154625 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:11.149965    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/981394e3-c7de-433d-a190-0ca90b59c64b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-47628d-668fb856c4-q5lp4\" (UID: \"981394e3-c7de-433d-a190-0ca90b59c64b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-47628d-668fb856c4-q5lp4"
Oct 30 13:20:11.154625 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:11.150218    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db25e66a-ea74-44b2-9d02-f1ece73cef2a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-47628d-6c75669467-c66k4\" (UID: \"db25e66a-ea74-44b2-9d02-f1ece73cef2a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-47628d-6c75669467-c66k4"
Oct 30 13:20:11.154625 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:11.150324    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfvnx\" (UniqueName: \"kubernetes.io/projected/981394e3-c7de-433d-a190-0ca90b59c64b-kube-api-access-sfvnx\") pod \"payload-processing-e2e-inf-a-47628d-668fb856c4-q5lp4\" (UID: \"981394e3-c7de-433d-a190-0ca90b59c64b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-47628d-668fb856c4-q5lp4"
Oct 30 13:20:11.154625 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:11.150359    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjnmd\" (UniqueName: \"kubernetes.io/projected/db25e66a-ea74-44b2-9d02-f1ece73cef2a-kube-api-access-jjnmd\") pod \"payload-pre-processing-e2e-inf-a-47628d-6c75669467-c66k4\" (UID: \"db25e66a-ea74-44b2-9d02-f1ece73cef2a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-47628d-6c75669467-c66k4"
Oct 30 13:20:11.337668 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:11.270504    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/981394e3-c7de-433d-a190-0ca90b59c64b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-47628d-668fb856c4-q5lp4\" (UID: \"981394e3-c7de-433d-a190-0ca90b59c64b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-47628d-668fb856c4-q5lp4"
Oct 30 13:20:11.337668 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:11.270717    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db25e66a-ea74-44b2-9d02-f1ece73cef2a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-47628d-6c75669467-c66k4\" (UID: \"db25e66a-ea74-44b2-9d02-f1ece73cef2a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-47628d-6c75669467-c66k4"
Oct 30 13:20:11.337668 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:11.270786    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfvnx\" (UniqueName: \"kubernetes.io/projected/981394e3-c7de-433d-a190-0ca90b59c64b-kube-api-access-sfvnx\") pod \"payload-processing-e2e-inf-a-47628d-668fb856c4-q5lp4\" (UID: \"981394e3-c7de-433d-a190-0ca90b59c64b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-47628d-668fb856c4-q5lp4"
Oct 30 13:20:11.337668 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:11.270814    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjnmd\" (UniqueName: \"kubernetes.io/projected/db25e66a-ea74-44b2-9d02-f1ece73cef2a-kube-api-access-jjnmd\") pod \"payload-pre-processing-e2e-inf-a-47628d-6c75669467-c66k4\" (UID: \"db25e66a-ea74-44b2-9d02-f1ece73cef2a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-47628d-6c75669467-c66k4"
Oct 30 13:20:11.337668 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:11.272046    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/981394e3-c7de-433d-a190-0ca90b59c64b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-47628d-668fb856c4-q5lp4\" (UID: \"981394e3-c7de-433d-a190-0ca90b59c64b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-47628d-668fb856c4-q5lp4"
Oct 30 13:20:11.337668 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:11.278959    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db25e66a-ea74-44b2-9d02-f1ece73cef2a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-47628d-6c75669467-c66k4\" (UID: \"db25e66a-ea74-44b2-9d02-f1ece73cef2a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-47628d-6c75669467-c66k4"
Oct 30 13:20:11.450664 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:11.450417    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjnmd\" (UniqueName: \"kubernetes.io/projected/db25e66a-ea74-44b2-9d02-f1ece73cef2a-kube-api-access-jjnmd\") pod \"payload-pre-processing-e2e-inf-a-47628d-6c75669467-c66k4\" (UID: \"db25e66a-ea74-44b2-9d02-f1ece73cef2a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-47628d-6c75669467-c66k4"
Oct 30 13:20:11.488212 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:11.481300    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfvnx\" (UniqueName: \"kubernetes.io/projected/981394e3-c7de-433d-a190-0ca90b59c64b-kube-api-access-sfvnx\") pod \"payload-processing-e2e-inf-a-47628d-668fb856c4-q5lp4\" (UID: \"981394e3-c7de-433d-a190-0ca90b59c64b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-47628d-668fb856c4-q5lp4"
Oct 30 13:20:11.652023 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:11.644026    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-47628d-668fb856c4-q5lp4"
Oct 30 13:20:11.734795 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:11.713098    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-47628d-6c75669467-c66k4"
Oct 30 13:20:11.799824 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:11.799793    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd"
Oct 30 13:20:11.800799 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:11.800771    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd"
Oct 30 13:20:11.801732 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:11.801709    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd"
Oct 30 13:20:11.802757 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:20:11.802727    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:20:12.118386 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:12.118359    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-47628d-668fb856c4-q5lp4"]
Oct 30 13:20:12.120658 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:20:12.120629    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod981394e3_c7de_433d_a190_0ca90b59c64b.slice/crio-e105f8e66c8bb0cb635cdcc41c4c84a6289c96e7b795c8083547cf0c12906e3d WatchSource:0}: Error finding container e105f8e66c8bb0cb635cdcc41c4c84a6289c96e7b795c8083547cf0c12906e3d: Status 404 returned error can't find the container with id e105f8e66c8bb0cb635cdcc41c4c84a6289c96e7b795c8083547cf0c12906e3d
Oct 30 13:20:12.130082 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:12.130059    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-47628d-6c75669467-c66k4"]
Oct 30 13:20:12.134427 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:20:12.134405    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb25e66a_ea74_44b2_9d02_f1ece73cef2a.slice/crio-b136513c9b38371e3907daaf6de353aa1f08502b5483bd92797652091cd22f40 WatchSource:0}: Error finding container b136513c9b38371e3907daaf6de353aa1f08502b5483bd92797652091cd22f40: Status 404 returned error can't find the container with id b136513c9b38371e3907daaf6de353aa1f08502b5483bd92797652091cd22f40
Oct 30 13:20:12.858230 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:12.858187    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-47628d-6c75669467-c66k4" event={"ID":"db25e66a-ea74-44b2-9d02-f1ece73cef2a","Type":"ContainerStarted","Data":"63a43f91ba1f819858c9e5f331ed8a9db3c3527a7476382379323b43102aa649"}
Oct 30 13:20:12.858735 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:12.858255    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-47628d-6c75669467-c66k4"
Oct 30 13:20:12.858735 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:12.858278    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-47628d-6c75669467-c66k4" event={"ID":"db25e66a-ea74-44b2-9d02-f1ece73cef2a","Type":"ContainerStarted","Data":"b136513c9b38371e3907daaf6de353aa1f08502b5483bd92797652091cd22f40"}
Oct 30 13:20:12.860423 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:12.860390    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-47628d-668fb856c4-q5lp4" event={"ID":"981394e3-c7de-433d-a190-0ca90b59c64b","Type":"ContainerStarted","Data":"94b7100e50a0976a08db439bd7595920a8f47523d9407abec263109f5c967917"}
Oct 30 13:20:12.860556 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:12.860452    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-47628d-668fb856c4-q5lp4"
Oct 30 13:20:12.860556 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:12.860479    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-47628d-668fb856c4-q5lp4" event={"ID":"981394e3-c7de-433d-a190-0ca90b59c64b","Type":"ContainerStarted","Data":"e105f8e66c8bb0cb635cdcc41c4c84a6289c96e7b795c8083547cf0c12906e3d"}
Oct 30 13:20:12.901180 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:12.901134    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-47628d-6c75669467-c66k4" podStartSLOduration=1.9011231990000002 podStartE2EDuration="1.901123199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:20:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:20:12.899474742 +0000 UTC m=+1712.065839391" watchObservedRunningTime="2026-10-30 13:20:12.901123199 +0000 UTC m=+1712.067487846"
Oct 30 13:20:12.945268 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:12.945217    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-47628d-668fb856c4-q5lp4" podStartSLOduration=2.945205416 podStartE2EDuration="2.945205416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:20:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:20:12.940228596 +0000 UTC m=+1712.106593241" watchObservedRunningTime="2026-10-30 13:20:12.945205416 +0000 UTC m=+1712.111570064"
Oct 30 13:20:13.674340 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:13.674305    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj"]
Oct 30 13:20:13.674576 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:13.674548    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj" podUID="dc3dc010-a693-406c-ba1e-55593789e70a" containerName="istio-proxy" containerID="cri-o://b146f090c42801191e551e76cfefed623e03a5676c488624db81f6a54666589d" gracePeriod=30
Oct 30 13:20:13.711649 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:13.711582    4364 patch_prober.go:29] interesting pod/e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:20:13.711810 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:13.711688    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj" podUID="dc3dc010-a693-406c-ba1e-55593789e70a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:20:14.927300 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:14.927268    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks"]
Oct 30 13:20:14.932565 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:14.932541    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks"
Oct 30 13:20:14.954920 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:14.954897    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-4651e8-openshift-default-dockercfg-v9dl8\""
Oct 30 13:20:14.982127 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:14.982095    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks"]
Oct 30 13:20:15.103628 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:15.102014    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb9e1069-efce-4243-b809-ad470e124bd1-istio-envoy\") pod \"e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks\" (UID: \"fb9e1069-efce-4243-b809-ad470e124bd1\") " pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks"
Oct 30 13:20:15.103628 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:15.102072    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb9e1069-efce-4243-b809-ad470e124bd1-istio-podinfo\") pod \"e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks\" (UID: \"fb9e1069-efce-4243-b809-ad470e124bd1\") " pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks"
Oct 30 13:20:15.103628 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:15.102134    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb9e1069-efce-4243-b809-ad470e124bd1-workload-socket\") pod \"e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks\" (UID: \"fb9e1069-efce-4243-b809-ad470e124bd1\") " pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks"
Oct 30 13:20:15.103628 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:15.102175    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb9e1069-efce-4243-b809-ad470e124bd1-istio-token\") pod \"e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks\" (UID: \"fb9e1069-efce-4243-b809-ad470e124bd1\") " pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks"
Oct 30 13:20:15.103628 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:15.102396    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp8zx\" (UniqueName: \"kubernetes.io/projected/fb9e1069-efce-4243-b809-ad470e124bd1-kube-api-access-hp8zx\") pod \"e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks\" (UID: \"fb9e1069-efce-4243-b809-ad470e124bd1\") " pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks"
Oct 30 13:20:15.103628 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:15.102455    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb9e1069-efce-4243-b809-ad470e124bd1-istiod-ca-cert\") pod \"e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks\" (UID: \"fb9e1069-efce-4243-b809-ad470e124bd1\") " pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks"
Oct 30 13:20:15.103628 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:15.102490    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb9e1069-efce-4243-b809-ad470e124bd1-istio-data\") pod \"e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks\" (UID: \"fb9e1069-efce-4243-b809-ad470e124bd1\") " pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks"
Oct 30 13:20:15.103628 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:15.102611    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb9e1069-efce-4243-b809-ad470e124bd1-workload-certs\") pod \"e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks\" (UID: \"fb9e1069-efce-4243-b809-ad470e124bd1\") " pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks"
Oct 30 13:20:15.103628 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:15.102651    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb9e1069-efce-4243-b809-ad470e124bd1-credential-socket\") pod \"e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks\" (UID: \"fb9e1069-efce-4243-b809-ad470e124bd1\") " pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks"
Oct 30 13:20:15.208673 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:15.208247    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp8zx\" (UniqueName: \"kubernetes.io/projected/fb9e1069-efce-4243-b809-ad470e124bd1-kube-api-access-hp8zx\") pod \"e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks\" (UID: \"fb9e1069-efce-4243-b809-ad470e124bd1\") " pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks"
Oct 30 13:20:15.221654 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:15.208827    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb9e1069-efce-4243-b809-ad470e124bd1-istiod-ca-cert\") pod \"e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks\" (UID: \"fb9e1069-efce-4243-b809-ad470e124bd1\") " pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks"
Oct 30 13:20:15.221654 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:15.209696    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb9e1069-efce-4243-b809-ad470e124bd1-istiod-ca-cert\") pod \"e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks\" (UID: \"fb9e1069-efce-4243-b809-ad470e124bd1\") " pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks"
Oct 30 13:20:15.221654 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:15.209779    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb9e1069-efce-4243-b809-ad470e124bd1-istio-data\") pod \"e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks\" (UID: \"fb9e1069-efce-4243-b809-ad470e124bd1\") " pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks"
Oct 30 13:20:15.221654 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:15.210063    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb9e1069-efce-4243-b809-ad470e124bd1-istio-data\") pod \"e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks\" (UID: \"fb9e1069-efce-4243-b809-ad470e124bd1\") " pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks"
Oct 30 13:20:15.221654 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:15.210262    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb9e1069-efce-4243-b809-ad470e124bd1-workload-certs\") pod \"e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks\" (UID: \"fb9e1069-efce-4243-b809-ad470e124bd1\") " pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks"
Oct 30 13:20:15.221654 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:15.210302    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb9e1069-efce-4243-b809-ad470e124bd1-credential-socket\") pod \"e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks\" (UID: \"fb9e1069-efce-4243-b809-ad470e124bd1\") " pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks"
Oct 30 13:20:15.221654 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:15.210558    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb9e1069-efce-4243-b809-ad470e124bd1-workload-certs\") pod \"e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks\" (UID: \"fb9e1069-efce-4243-b809-ad470e124bd1\") " pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks"
Oct 30 13:20:15.221654 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:15.210822    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb9e1069-efce-4243-b809-ad470e124bd1-credential-socket\") pod \"e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks\" (UID: \"fb9e1069-efce-4243-b809-ad470e124bd1\") " pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks"
Oct 30 13:20:15.221654 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:15.210936    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb9e1069-efce-4243-b809-ad470e124bd1-istio-envoy\") pod \"e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks\" (UID: \"fb9e1069-efce-4243-b809-ad470e124bd1\") " pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks"
Oct 30 13:20:15.221654 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:15.210971    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb9e1069-efce-4243-b809-ad470e124bd1-istio-podinfo\") pod \"e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks\" (UID: \"fb9e1069-efce-4243-b809-ad470e124bd1\") " pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks"
Oct 30 13:20:15.221654 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:15.211028    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb9e1069-efce-4243-b809-ad470e124bd1-workload-socket\") pod \"e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks\" (UID: \"fb9e1069-efce-4243-b809-ad470e124bd1\") " pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks"
Oct 30 13:20:15.221654 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:15.211068    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb9e1069-efce-4243-b809-ad470e124bd1-istio-token\") pod \"e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks\" (UID: \"fb9e1069-efce-4243-b809-ad470e124bd1\") " pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks"
Oct 30 13:20:15.221654 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:15.212279    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb9e1069-efce-4243-b809-ad470e124bd1-workload-socket\") pod \"e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks\" (UID: \"fb9e1069-efce-4243-b809-ad470e124bd1\") " pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks"
Oct 30 13:20:15.233263 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:15.233232    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb9e1069-efce-4243-b809-ad470e124bd1-istio-podinfo\") pod \"e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks\" (UID: \"fb9e1069-efce-4243-b809-ad470e124bd1\") " pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks"
Oct 30 13:20:15.275623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:15.255677    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp8zx\" (UniqueName: \"kubernetes.io/projected/fb9e1069-efce-4243-b809-ad470e124bd1-kube-api-access-hp8zx\") pod \"e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks\" (UID: \"fb9e1069-efce-4243-b809-ad470e124bd1\") " pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks"
Oct 30 13:20:15.278210 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:15.278175    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb9e1069-efce-4243-b809-ad470e124bd1-istio-token\") pod \"e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks\" (UID: \"fb9e1069-efce-4243-b809-ad470e124bd1\") " pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks"
Oct 30 13:20:15.282897 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:15.282861    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb9e1069-efce-4243-b809-ad470e124bd1-istio-envoy\") pod \"e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks\" (UID: \"fb9e1069-efce-4243-b809-ad470e124bd1\") " pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks"
Oct 30 13:20:15.567443 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:15.567361    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks"
Oct 30 13:20:15.962970 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:15.962937    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks"]
Oct 30 13:20:15.963336 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:20:15.963095    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb9e1069_efce_4243_b809_ad470e124bd1.slice/crio-ce4d88a199fcf696ed42f84620a82c65d2884df1718dc8f9107deaf0cca0314e WatchSource:0}: Error finding container ce4d88a199fcf696ed42f84620a82c65d2884df1718dc8f9107deaf0cca0314e: Status 404 returned error can't find the container with id ce4d88a199fcf696ed42f84620a82c65d2884df1718dc8f9107deaf0cca0314e
Oct 30 13:20:15.966260 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:15.966218    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:20:15.966352 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:15.966307    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:20:15.966392 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:15.966350    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:20:16.951945 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:16.951900    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks" event={"ID":"fb9e1069-efce-4243-b809-ad470e124bd1","Type":"ContainerStarted","Data":"cae3c76fb0576f5591a338bc3d5958baa7cc4110ba759c4d7f4e2419ce57be51"}
Oct 30 13:20:16.952121 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:16.951960    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks" event={"ID":"fb9e1069-efce-4243-b809-ad470e124bd1","Type":"ContainerStarted","Data":"ce4d88a199fcf696ed42f84620a82c65d2884df1718dc8f9107deaf0cca0314e"}
Oct 30 13:20:17.026024 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:17.025976    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks" podStartSLOduration=3.025961834 podStartE2EDuration="3.025961834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:20:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:20:17.012996052 +0000 UTC m=+1716.179360699" watchObservedRunningTime="2026-10-30 13:20:17.025961834 +0000 UTC m=+1716.192326482"
Oct 30 13:20:17.573621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:17.571774    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks"
Oct 30 13:20:17.602126 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:17.596077    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks"
Oct 30 13:20:17.960328 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:17.960296    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks"
Oct 30 13:20:17.961347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:17.961327    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks"
Oct 30 13:20:18.971707 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:18.971668    4364 generic.go:356] "Generic (PLEG): container finished" podID="dc3dc010-a693-406c-ba1e-55593789e70a" containerID="b146f090c42801191e551e76cfefed623e03a5676c488624db81f6a54666589d" exitCode=0
Oct 30 13:20:18.974074 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:18.974046    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj" event={"ID":"dc3dc010-a693-406c-ba1e-55593789e70a","Type":"ContainerDied","Data":"b146f090c42801191e551e76cfefed623e03a5676c488624db81f6a54666589d"}
Oct 30 13:20:18.974184 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:18.974100    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj" event={"ID":"dc3dc010-a693-406c-ba1e-55593789e70a","Type":"ContainerDied","Data":"3da49413b3f36822e9987a7d65e7e88c824c69a69d7d595e7e9ad5e9c288c1ad"}
Oct 30 13:20:18.974184 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:18.974117    4364 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3da49413b3f36822e9987a7d65e7e88c824c69a69d7d595e7e9ad5e9c288c1ad"
Oct 30 13:20:18.974184 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:18.974131    4364 scope.go:122] "RemoveContainer" containerID="f905015273e255c0722a45299627b4b2fb5bcd0811c8d44578957967701da44c"
Oct 30 13:20:18.985687 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:18.985665    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj"
Oct 30 13:20:19.037763 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:19.037733    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-774b5079-rnk2n"]
Oct 30 13:20:19.043208 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:19.043182    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-774b5079-rnk2n"]
Oct 30 13:20:19.052022 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:19.051997    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc3dc010-a693-406c-ba1e-55593789e70a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc3dc010-a693-406c-ba1e-55593789e70a-workload-certs\") pod \"dc3dc010-a693-406c-ba1e-55593789e70a\" (UID: \"dc3dc010-a693-406c-ba1e-55593789e70a\") "
Oct 30 13:20:19.052150 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:19.052046    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dc3dc010-a693-406c-ba1e-55593789e70a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc3dc010-a693-406c-ba1e-55593789e70a-istio-podinfo\") pod \"dc3dc010-a693-406c-ba1e-55593789e70a\" (UID: \"dc3dc010-a693-406c-ba1e-55593789e70a\") "
Oct 30 13:20:19.052150 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:19.052071    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc3dc010-a693-406c-ba1e-55593789e70a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc3dc010-a693-406c-ba1e-55593789e70a-istio-data\") pod \"dc3dc010-a693-406c-ba1e-55593789e70a\" (UID: \"dc3dc010-a693-406c-ba1e-55593789e70a\") "
Oct 30 13:20:19.056074 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:19.052333    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc3dc010-a693-406c-ba1e-55593789e70a-workload-certs" pod "dc3dc010-a693-406c-ba1e-55593789e70a" (UID: "dc3dc010-a693-406c-ba1e-55593789e70a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:20:19.056074 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:19.052339    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc3dc010-a693-406c-ba1e-55593789e70a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc3dc010-a693-406c-ba1e-55593789e70a-workload-socket\") pod \"dc3dc010-a693-406c-ba1e-55593789e70a\" (UID: \"dc3dc010-a693-406c-ba1e-55593789e70a\") "
Oct 30 13:20:19.056074 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:19.052394    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc3dc010-a693-406c-ba1e-55593789e70a-istio-data" pod "dc3dc010-a693-406c-ba1e-55593789e70a" (UID: "dc3dc010-a693-406c-ba1e-55593789e70a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:20:19.056074 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:19.052434    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc3dc010-a693-406c-ba1e-55593789e70a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc3dc010-a693-406c-ba1e-55593789e70a-credential-socket\") pod \"dc3dc010-a693-406c-ba1e-55593789e70a\" (UID: \"dc3dc010-a693-406c-ba1e-55593789e70a\") "
Oct 30 13:20:19.056074 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:19.052540    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc3dc010-a693-406c-ba1e-55593789e70a-workload-socket" pod "dc3dc010-a693-406c-ba1e-55593789e70a" (UID: "dc3dc010-a693-406c-ba1e-55593789e70a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:20:19.056074 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:19.052645    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc3dc010-a693-406c-ba1e-55593789e70a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc3dc010-a693-406c-ba1e-55593789e70a-istio-envoy\") pod \"dc3dc010-a693-406c-ba1e-55593789e70a\" (UID: \"dc3dc010-a693-406c-ba1e-55593789e70a\") "
Oct 30 13:20:19.056074 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:19.052705    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc3dc010-a693-406c-ba1e-55593789e70a-credential-socket" pod "dc3dc010-a693-406c-ba1e-55593789e70a" (UID: "dc3dc010-a693-406c-ba1e-55593789e70a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:20:19.056074 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:19.052781    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc3dc010-a693-406c-ba1e-55593789e70a-istio-token\" (UniqueName: \"kubernetes.io/projected/dc3dc010-a693-406c-ba1e-55593789e70a-istio-token\") pod \"dc3dc010-a693-406c-ba1e-55593789e70a\" (UID: \"dc3dc010-a693-406c-ba1e-55593789e70a\") "
Oct 30 13:20:19.056074 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:19.052913    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc3dc010-a693-406c-ba1e-55593789e70a-kube-api-access-62bl6\" (UniqueName: \"kubernetes.io/projected/dc3dc010-a693-406c-ba1e-55593789e70a-kube-api-access-62bl6\") pod \"dc3dc010-a693-406c-ba1e-55593789e70a\" (UID: \"dc3dc010-a693-406c-ba1e-55593789e70a\") "
Oct 30 13:20:19.056074 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:19.052966    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc3dc010-a693-406c-ba1e-55593789e70a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc3dc010-a693-406c-ba1e-55593789e70a-istiod-ca-cert\") pod \"dc3dc010-a693-406c-ba1e-55593789e70a\" (UID: \"dc3dc010-a693-406c-ba1e-55593789e70a\") "
Oct 30 13:20:19.056074 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:19.053785    4364 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc3dc010-a693-406c-ba1e-55593789e70a-workload-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:20:19.056074 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:19.053810    4364 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc3dc010-a693-406c-ba1e-55593789e70a-istio-data\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:20:19.056074 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:19.053823    4364 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc3dc010-a693-406c-ba1e-55593789e70a-workload-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:20:19.056074 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:19.053836    4364 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc3dc010-a693-406c-ba1e-55593789e70a-credential-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:20:19.056074 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:19.054752    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc3dc010-a693-406c-ba1e-55593789e70a-istio-podinfo" pod "dc3dc010-a693-406c-ba1e-55593789e70a" (UID: "dc3dc010-a693-406c-ba1e-55593789e70a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:20:19.056074 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:19.055282    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc3dc010-a693-406c-ba1e-55593789e70a-istiod-ca-cert" pod "dc3dc010-a693-406c-ba1e-55593789e70a" (UID: "dc3dc010-a693-406c-ba1e-55593789e70a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:20:19.059031 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:19.059002    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc3dc010-a693-406c-ba1e-55593789e70a-istio-envoy" pod "dc3dc010-a693-406c-ba1e-55593789e70a" (UID: "dc3dc010-a693-406c-ba1e-55593789e70a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:20:19.059143 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:19.059060    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc3dc010-a693-406c-ba1e-55593789e70a-kube-api-access-62bl6" pod "dc3dc010-a693-406c-ba1e-55593789e70a" (UID: "dc3dc010-a693-406c-ba1e-55593789e70a"). InnerVolumeSpecName "kube-api-access-62bl6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:20:19.059143 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:19.059111    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc3dc010-a693-406c-ba1e-55593789e70a-istio-token" pod "dc3dc010-a693-406c-ba1e-55593789e70a" (UID: "dc3dc010-a693-406c-ba1e-55593789e70a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:20:19.155830 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:19.155793    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62bl6\" (UniqueName: \"kubernetes.io/projected/dc3dc010-a693-406c-ba1e-55593789e70a-kube-api-access-62bl6\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:20:19.155830 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:19.155825    4364 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc3dc010-a693-406c-ba1e-55593789e70a-istiod-ca-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:20:19.156046 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:19.155842    4364 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc3dc010-a693-406c-ba1e-55593789e70a-istio-podinfo\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:20:19.156046 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:19.155852    4364 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc3dc010-a693-406c-ba1e-55593789e70a-istio-envoy\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:20:19.156046 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:19.155863    4364 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc3dc010-a693-406c-ba1e-55593789e70a-istio-token\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:20:19.491572 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:19.491536    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-77254d/test-model-77254d-kserve-5476b887bf-2zjpk"]
Oct 30 13:20:19.492661 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:19.492515    4364 cpu_manager.go:414] "containerMap: removing container" podUID="dc3dc010-a693-406c-ba1e-55593789e70a" containerName="istio-proxy"
Oct 30 13:20:19.492661 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:19.492539    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc3dc010-a693-406c-ba1e-55593789e70a" containerName="istio-proxy"
Oct 30 13:20:19.492856 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:19.492798    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc3dc010-a693-406c-ba1e-55593789e70a" containerName="istio-proxy"
Oct 30 13:20:19.492856 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:19.492815    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc3dc010-a693-406c-ba1e-55593789e70a" containerName="istio-proxy"
Oct 30 13:20:19.498157 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:19.498123    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-77254d/test-model-77254d-kserve-5476b887bf-2zjpk"
Oct 30 13:20:19.514270 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:19.514220    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-77254d\"/\"openshift-service-ca.crt\""
Oct 30 13:20:19.514903 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:19.514868    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-77254d\"/\"kube-root-ca.crt\""
Oct 30 13:20:19.515244 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:19.515143    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-77254d\"/\"default-dockercfg-9pk6w\""
Oct 30 13:20:19.524395 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:19.524345    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-77254d/test-model-77254d-kserve-5476b887bf-2zjpk"]
Oct 30 13:20:19.524814 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:19.524792    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-77254d\"/\"test-model-77254d-kserve-self-signed-certs\""
Oct 30 13:20:19.665953 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:19.665916    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a-home\") pod \"test-model-77254d-kserve-5476b887bf-2zjpk\" (UID: \"e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a\") " pod="ai-tenant-e2e-inf-a-77254d/test-model-77254d-kserve-5476b887bf-2zjpk"
Oct 30 13:20:19.666103 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:19.665972    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a-dshm\") pod \"test-model-77254d-kserve-5476b887bf-2zjpk\" (UID: \"e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a\") " pod="ai-tenant-e2e-inf-a-77254d/test-model-77254d-kserve-5476b887bf-2zjpk"
Oct 30 13:20:19.666103 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:19.666059    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a-model-cache\") pod \"test-model-77254d-kserve-5476b887bf-2zjpk\" (UID: \"e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a\") " pod="ai-tenant-e2e-inf-a-77254d/test-model-77254d-kserve-5476b887bf-2zjpk"
Oct 30 13:20:19.666265 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:19.666194    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a-tmp-dir\") pod \"test-model-77254d-kserve-5476b887bf-2zjpk\" (UID: \"e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a\") " pod="ai-tenant-e2e-inf-a-77254d/test-model-77254d-kserve-5476b887bf-2zjpk"
Oct 30 13:20:19.666347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:19.666327    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64hx5\" (UniqueName: \"kubernetes.io/projected/e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a-kube-api-access-64hx5\") pod \"test-model-77254d-kserve-5476b887bf-2zjpk\" (UID: \"e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a\") " pod="ai-tenant-e2e-inf-a-77254d/test-model-77254d-kserve-5476b887bf-2zjpk"
Oct 30 13:20:19.666409 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:19.666394    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a-tls-certs\") pod \"test-model-77254d-kserve-5476b887bf-2zjpk\" (UID: \"e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a\") " pod="ai-tenant-e2e-inf-a-77254d/test-model-77254d-kserve-5476b887bf-2zjpk"
Oct 30 13:20:19.767536 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:19.767462    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a-home\") pod \"test-model-77254d-kserve-5476b887bf-2zjpk\" (UID: \"e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a\") " pod="ai-tenant-e2e-inf-a-77254d/test-model-77254d-kserve-5476b887bf-2zjpk"
Oct 30 13:20:19.767536 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:19.767503    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a-dshm\") pod \"test-model-77254d-kserve-5476b887bf-2zjpk\" (UID: \"e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a\") " pod="ai-tenant-e2e-inf-a-77254d/test-model-77254d-kserve-5476b887bf-2zjpk"
Oct 30 13:20:19.767536 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:19.767525    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a-model-cache\") pod \"test-model-77254d-kserve-5476b887bf-2zjpk\" (UID: \"e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a\") " pod="ai-tenant-e2e-inf-a-77254d/test-model-77254d-kserve-5476b887bf-2zjpk"
Oct 30 13:20:19.767822 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:19.767644    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a-tmp-dir\") pod \"test-model-77254d-kserve-5476b887bf-2zjpk\" (UID: \"e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a\") " pod="ai-tenant-e2e-inf-a-77254d/test-model-77254d-kserve-5476b887bf-2zjpk"
Oct 30 13:20:19.767822 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:19.767745    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64hx5\" (UniqueName: \"kubernetes.io/projected/e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a-kube-api-access-64hx5\") pod \"test-model-77254d-kserve-5476b887bf-2zjpk\" (UID: \"e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a\") " pod="ai-tenant-e2e-inf-a-77254d/test-model-77254d-kserve-5476b887bf-2zjpk"
Oct 30 13:20:19.767822 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:19.767805    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a-tls-certs\") pod \"test-model-77254d-kserve-5476b887bf-2zjpk\" (UID: \"e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a\") " pod="ai-tenant-e2e-inf-a-77254d/test-model-77254d-kserve-5476b887bf-2zjpk"
Oct 30 13:20:19.767993 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:19.767893    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a-home\") pod \"test-model-77254d-kserve-5476b887bf-2zjpk\" (UID: \"e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a\") " pod="ai-tenant-e2e-inf-a-77254d/test-model-77254d-kserve-5476b887bf-2zjpk"
Oct 30 13:20:19.768054 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:19.768007    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a-model-cache\") pod \"test-model-77254d-kserve-5476b887bf-2zjpk\" (UID: \"e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a\") " pod="ai-tenant-e2e-inf-a-77254d/test-model-77254d-kserve-5476b887bf-2zjpk"
Oct 30 13:20:19.768054 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:19.768019    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a-tmp-dir\") pod \"test-model-77254d-kserve-5476b887bf-2zjpk\" (UID: \"e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a\") " pod="ai-tenant-e2e-inf-a-77254d/test-model-77254d-kserve-5476b887bf-2zjpk"
Oct 30 13:20:19.769801 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:19.769778    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a-dshm\") pod \"test-model-77254d-kserve-5476b887bf-2zjpk\" (UID: \"e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a\") " pod="ai-tenant-e2e-inf-a-77254d/test-model-77254d-kserve-5476b887bf-2zjpk"
Oct 30 13:20:19.770259 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:19.770240    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a-tls-certs\") pod \"test-model-77254d-kserve-5476b887bf-2zjpk\" (UID: \"e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a\") " pod="ai-tenant-e2e-inf-a-77254d/test-model-77254d-kserve-5476b887bf-2zjpk"
Oct 30 13:20:19.833621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:19.825775    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64hx5\" (UniqueName: \"kubernetes.io/projected/e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a-kube-api-access-64hx5\") pod \"test-model-77254d-kserve-5476b887bf-2zjpk\" (UID: \"e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a\") " pod="ai-tenant-e2e-inf-a-77254d/test-model-77254d-kserve-5476b887bf-2zjpk"
Oct 30 13:20:20.027378 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:20.027294    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj"
Oct 30 13:20:20.133405 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:20.133368    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-77254d/test-model-77254d-kserve-5476b887bf-2zjpk"
Oct 30 13:20:20.181264 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:20.181230    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj"]
Oct 30 13:20:20.186297 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:20.186270    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-4644a1-openshift-default-75b77787f-vtlgj"]
Oct 30 13:20:20.459722 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:20.459682    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-77254d/test-model-77254d-kserve-5476b887bf-2zjpk"]
Oct 30 13:20:20.460498 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:20:20.460476    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode3b39cfd_9e3a_4b05_bb20_451a89f5ea6a.slice/crio-cdb3a0b33c5d6c79e303db1d51ce7c792f796db1350a39712f29eb35522bddaa WatchSource:0}: Error finding container cdb3a0b33c5d6c79e303db1d51ce7c792f796db1350a39712f29eb35522bddaa: Status 404 returned error can't find the container with id cdb3a0b33c5d6c79e303db1d51ce7c792f796db1350a39712f29eb35522bddaa
Oct 30 13:20:20.973072 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:20.973032    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ada2dea-42f8-4811-8109-87b14971d203" path="/var/lib/kubelet/pods/2ada2dea-42f8-4811-8109-87b14971d203/volumes"
Oct 30 13:20:20.973576 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:20.973558    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc3dc010-a693-406c-ba1e-55593789e70a" path="/var/lib/kubelet/pods/dc3dc010-a693-406c-ba1e-55593789e70a/volumes"
Oct 30 13:20:21.028675 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:21.028638    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-77254d/test-model-77254d-kserve-5476b887bf-2zjpk" event={"ID":"e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a","Type":"ContainerStarted","Data":"dbddda331f04a98ef97bd01673922d0add5831dd85f39411a913f537deaee3e7"}
Oct 30 13:20:21.029131 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:21.028691    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-77254d/test-model-77254d-kserve-5476b887bf-2zjpk" event={"ID":"e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a","Type":"ContainerStarted","Data":"cdb3a0b33c5d6c79e303db1d51ce7c792f796db1350a39712f29eb35522bddaa"}
Oct 30 13:20:21.072963 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:21.072904    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-77254d/test-model-77254d-kserve-5476b887bf-2zjpk" podStartSLOduration=2.072890701 podStartE2EDuration="2.072890701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:20:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:20:21.069522664 +0000 UTC m=+1720.235887312" watchObservedRunningTime="2026-10-30 13:20:21.072890701 +0000 UTC m=+1720.239255348"
Oct 30 13:20:21.136205 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:21.136177    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-77254d/test-model-77254d-kserve-5476b887bf-2zjpk"
Oct 30 13:20:21.477287 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:21.477256    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4651e8-67665669fd-qdsnz"]
Oct 30 13:20:21.478204 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:21.478162    4364 cpu_manager.go:414] "containerMap: removing container" podUID="dc3dc010-a693-406c-ba1e-55593789e70a" containerName="istio-proxy"
Oct 30 13:20:21.478204 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:21.478204    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc3dc010-a693-406c-ba1e-55593789e70a" containerName="istio-proxy"
Oct 30 13:20:21.482139 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:21.482117    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4651e8-67665669fd-qdsnz"
Oct 30 13:20:21.500669 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:21.500644    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4651e8-67665669fd-qdsnz"]
Oct 30 13:20:21.507226 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:21.507202    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-4651e8\""
Oct 30 13:20:21.507226 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:21.507202    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-4651e8-dockercfg-gfcwt\""
Oct 30 13:20:21.544988 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:21.544957    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4651e8-8f64b59bc-2xng9"]
Oct 30 13:20:21.550947 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:21.550920    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4651e8-8f64b59bc-2xng9"
Oct 30 13:20:21.567167 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:21.567138    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4651e8-8f64b59bc-2xng9"]
Oct 30 13:20:21.624970 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:21.624932    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctcbw\" (UniqueName: \"kubernetes.io/projected/df0a589d-208c-4471-8626-07bc7b3ad709-kube-api-access-ctcbw\") pod \"payload-processing-e2e-inf-a-4651e8-67665669fd-qdsnz\" (UID: \"df0a589d-208c-4471-8626-07bc7b3ad709\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4651e8-67665669fd-qdsnz"
Oct 30 13:20:21.625110 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:21.625094    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df0a589d-208c-4471-8626-07bc7b3ad709-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4651e8-67665669fd-qdsnz\" (UID: \"df0a589d-208c-4471-8626-07bc7b3ad709\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4651e8-67665669fd-qdsnz"
Oct 30 13:20:21.742677 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:21.731775    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctcbw\" (UniqueName: \"kubernetes.io/projected/df0a589d-208c-4471-8626-07bc7b3ad709-kube-api-access-ctcbw\") pod \"payload-processing-e2e-inf-a-4651e8-67665669fd-qdsnz\" (UID: \"df0a589d-208c-4471-8626-07bc7b3ad709\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4651e8-67665669fd-qdsnz"
Oct 30 13:20:21.742677 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:21.731883    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfntw\" (UniqueName: \"kubernetes.io/projected/25395e75-3362-4355-a31d-de00a81b32c8-kube-api-access-rfntw\") pod \"payload-pre-processing-e2e-inf-a-4651e8-8f64b59bc-2xng9\" (UID: \"25395e75-3362-4355-a31d-de00a81b32c8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4651e8-8f64b59bc-2xng9"
Oct 30 13:20:21.742677 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:21.731959    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df0a589d-208c-4471-8626-07bc7b3ad709-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4651e8-67665669fd-qdsnz\" (UID: \"df0a589d-208c-4471-8626-07bc7b3ad709\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4651e8-67665669fd-qdsnz"
Oct 30 13:20:21.742677 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:21.731984    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25395e75-3362-4355-a31d-de00a81b32c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4651e8-8f64b59bc-2xng9\" (UID: \"25395e75-3362-4355-a31d-de00a81b32c8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4651e8-8f64b59bc-2xng9"
Oct 30 13:20:21.742677 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:21.733420    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df0a589d-208c-4471-8626-07bc7b3ad709-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4651e8-67665669fd-qdsnz\" (UID: \"df0a589d-208c-4471-8626-07bc7b3ad709\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4651e8-67665669fd-qdsnz"
Oct 30 13:20:21.780621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:21.772658    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctcbw\" (UniqueName: \"kubernetes.io/projected/df0a589d-208c-4471-8626-07bc7b3ad709-kube-api-access-ctcbw\") pod \"payload-processing-e2e-inf-a-4651e8-67665669fd-qdsnz\" (UID: \"df0a589d-208c-4471-8626-07bc7b3ad709\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4651e8-67665669fd-qdsnz"
Oct 30 13:20:21.803624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:21.797897    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4651e8-67665669fd-qdsnz"
Oct 30 13:20:21.886667 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:21.884465    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfntw\" (UniqueName: \"kubernetes.io/projected/25395e75-3362-4355-a31d-de00a81b32c8-kube-api-access-rfntw\") pod \"payload-pre-processing-e2e-inf-a-4651e8-8f64b59bc-2xng9\" (UID: \"25395e75-3362-4355-a31d-de00a81b32c8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4651e8-8f64b59bc-2xng9"
Oct 30 13:20:21.886667 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:21.884550    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25395e75-3362-4355-a31d-de00a81b32c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4651e8-8f64b59bc-2xng9\" (UID: \"25395e75-3362-4355-a31d-de00a81b32c8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4651e8-8f64b59bc-2xng9"
Oct 30 13:20:21.910621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:21.905393    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25395e75-3362-4355-a31d-de00a81b32c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4651e8-8f64b59bc-2xng9\" (UID: \"25395e75-3362-4355-a31d-de00a81b32c8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4651e8-8f64b59bc-2xng9"
Oct 30 13:20:21.942332 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:21.937265    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfntw\" (UniqueName: \"kubernetes.io/projected/25395e75-3362-4355-a31d-de00a81b32c8-kube-api-access-rfntw\") pod \"payload-pre-processing-e2e-inf-a-4651e8-8f64b59bc-2xng9\" (UID: \"25395e75-3362-4355-a31d-de00a81b32c8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4651e8-8f64b59bc-2xng9"
Oct 30 13:20:22.169497 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:22.169465    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4651e8-8f64b59bc-2xng9"
Oct 30 13:20:22.358997 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:22.358972    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4651e8-67665669fd-qdsnz"]
Oct 30 13:20:22.361096 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:20:22.361068    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf0a589d_208c_4471_8626_07bc7b3ad709.slice/crio-5728499dd201d7f3bbe523fc10470544764c73ce360832f1f5a0e85bd081a191 WatchSource:0}: Error finding container 5728499dd201d7f3bbe523fc10470544764c73ce360832f1f5a0e85bd081a191: Status 404 returned error can't find the container with id 5728499dd201d7f3bbe523fc10470544764c73ce360832f1f5a0e85bd081a191
Oct 30 13:20:22.484527 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:22.484504    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4651e8-8f64b59bc-2xng9"]
Oct 30 13:20:22.485246 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:20:22.485225    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25395e75_3362_4355_a31d_de00a81b32c8.slice/crio-92b762214b637a42830807ea099eff878a92f380bbf945cffe4d9c52a0e299f0 WatchSource:0}: Error finding container 92b762214b637a42830807ea099eff878a92f380bbf945cffe4d9c52a0e299f0: Status 404 returned error can't find the container with id 92b762214b637a42830807ea099eff878a92f380bbf945cffe4d9c52a0e299f0
Oct 30 13:20:23.098848 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:23.098814    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4651e8-8f64b59bc-2xng9" event={"ID":"25395e75-3362-4355-a31d-de00a81b32c8","Type":"ContainerStarted","Data":"ef0824db0e869daf55acff3879db70c5cd757db63fdb01ae114316bb80af7d6d"}
Oct 30 13:20:23.099015 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:23.098864    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4651e8-8f64b59bc-2xng9" event={"ID":"25395e75-3362-4355-a31d-de00a81b32c8","Type":"ContainerStarted","Data":"92b762214b637a42830807ea099eff878a92f380bbf945cffe4d9c52a0e299f0"}
Oct 30 13:20:23.099015 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:23.098889    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4651e8-8f64b59bc-2xng9"
Oct 30 13:20:23.100518 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:23.100489    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4651e8-67665669fd-qdsnz" event={"ID":"df0a589d-208c-4471-8626-07bc7b3ad709","Type":"ContainerStarted","Data":"0f9f324765eca0c3b27095b8a1b4f694a5c229573ae25e9db526209020a39b04"}
Oct 30 13:20:23.100659 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:23.100620    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-4651e8-67665669fd-qdsnz"
Oct 30 13:20:23.100659 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:23.100646    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4651e8-67665669fd-qdsnz" event={"ID":"df0a589d-208c-4471-8626-07bc7b3ad709","Type":"ContainerStarted","Data":"5728499dd201d7f3bbe523fc10470544764c73ce360832f1f5a0e85bd081a191"}
Oct 30 13:20:23.143129 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:23.143088    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4651e8-8f64b59bc-2xng9" podStartSLOduration=2.143076089 podStartE2EDuration="2.143076089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:20:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:20:23.137999877 +0000 UTC m=+1722.304364525" watchObservedRunningTime="2026-10-30 13:20:23.143076089 +0000 UTC m=+1722.309440775"
Oct 30 13:20:23.180112 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:23.180069    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-4651e8-67665669fd-qdsnz" podStartSLOduration=2.18005669 podStartE2EDuration="2.18005669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:20:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:20:23.177203845 +0000 UTC m=+1722.343568495" watchObservedRunningTime="2026-10-30 13:20:23.18005669 +0000 UTC m=+1722.346421337"
Oct 30 13:20:23.941405 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:23.941372    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-47628d-6c75669467-c66k4"
Oct 30 13:20:23.941631 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:23.941495    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-47628d-668fb856c4-q5lp4"
Oct 30 13:20:29.606888 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:29.606855    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8467ed-76766c9c88-cbg5d"]
Oct 30 13:20:29.611388 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:29.611368    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8467ed-76766c9c88-cbg5d"
Oct 30 13:20:29.628634 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:29.628592    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-8467ed-serving-cert\""
Oct 30 13:20:29.647891 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:29.647859    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8467ed-76766c9c88-cbg5d"]
Oct 30 13:20:29.660964 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:29.660924    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdwrt\" (UniqueName: \"kubernetes.io/projected/fc880b1c-4413-4369-950b-a4bb3eb62568-kube-api-access-mdwrt\") pod \"maas-api-e2e-inf-b-8467ed-76766c9c88-cbg5d\" (UID: \"fc880b1c-4413-4369-950b-a4bb3eb62568\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8467ed-76766c9c88-cbg5d"
Oct 30 13:20:29.661192 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:29.661059    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc880b1c-4413-4369-950b-a4bb3eb62568-metrics-certs\") pod \"maas-api-e2e-inf-b-8467ed-76766c9c88-cbg5d\" (UID: \"fc880b1c-4413-4369-950b-a4bb3eb62568\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8467ed-76766c9c88-cbg5d"
Oct 30 13:20:29.661347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:29.661327    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc880b1c-4413-4369-950b-a4bb3eb62568-maas-api-tls\") pod \"maas-api-e2e-inf-b-8467ed-76766c9c88-cbg5d\" (UID: \"fc880b1c-4413-4369-950b-a4bb3eb62568\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8467ed-76766c9c88-cbg5d"
Oct 30 13:20:29.661435 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:29.661384    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc880b1c-4413-4369-950b-a4bb3eb62568-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8467ed-76766c9c88-cbg5d\" (UID: \"fc880b1c-4413-4369-950b-a4bb3eb62568\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8467ed-76766c9c88-cbg5d"
Oct 30 13:20:29.762858 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:29.762826    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc880b1c-4413-4369-950b-a4bb3eb62568-metrics-certs\") pod \"maas-api-e2e-inf-b-8467ed-76766c9c88-cbg5d\" (UID: \"fc880b1c-4413-4369-950b-a4bb3eb62568\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8467ed-76766c9c88-cbg5d"
Oct 30 13:20:29.763057 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:29.762990    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc880b1c-4413-4369-950b-a4bb3eb62568-maas-api-tls\") pod \"maas-api-e2e-inf-b-8467ed-76766c9c88-cbg5d\" (UID: \"fc880b1c-4413-4369-950b-a4bb3eb62568\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8467ed-76766c9c88-cbg5d"
Oct 30 13:20:29.763057 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:29.763026    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc880b1c-4413-4369-950b-a4bb3eb62568-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8467ed-76766c9c88-cbg5d\" (UID: \"fc880b1c-4413-4369-950b-a4bb3eb62568\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8467ed-76766c9c88-cbg5d"
Oct 30 13:20:29.763219 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:29.763199    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdwrt\" (UniqueName: \"kubernetes.io/projected/fc880b1c-4413-4369-950b-a4bb3eb62568-kube-api-access-mdwrt\") pod \"maas-api-e2e-inf-b-8467ed-76766c9c88-cbg5d\" (UID: \"fc880b1c-4413-4369-950b-a4bb3eb62568\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8467ed-76766c9c88-cbg5d"
Oct 30 13:20:29.763846 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:29.763819    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc880b1c-4413-4369-950b-a4bb3eb62568-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8467ed-76766c9c88-cbg5d\" (UID: \"fc880b1c-4413-4369-950b-a4bb3eb62568\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8467ed-76766c9c88-cbg5d"
Oct 30 13:20:29.765430 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:29.765407    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc880b1c-4413-4369-950b-a4bb3eb62568-maas-api-tls\") pod \"maas-api-e2e-inf-b-8467ed-76766c9c88-cbg5d\" (UID: \"fc880b1c-4413-4369-950b-a4bb3eb62568\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8467ed-76766c9c88-cbg5d"
Oct 30 13:20:29.765561 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:29.765452    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc880b1c-4413-4369-950b-a4bb3eb62568-metrics-certs\") pod \"maas-api-e2e-inf-b-8467ed-76766c9c88-cbg5d\" (UID: \"fc880b1c-4413-4369-950b-a4bb3eb62568\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8467ed-76766c9c88-cbg5d"
Oct 30 13:20:29.791823 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:29.791792    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdwrt\" (UniqueName: \"kubernetes.io/projected/fc880b1c-4413-4369-950b-a4bb3eb62568-kube-api-access-mdwrt\") pod \"maas-api-e2e-inf-b-8467ed-76766c9c88-cbg5d\" (UID: \"fc880b1c-4413-4369-950b-a4bb3eb62568\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8467ed-76766c9c88-cbg5d"
Oct 30 13:20:29.932925 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:29.926520    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8467ed-76766c9c88-cbg5d"
Oct 30 13:20:30.136365 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:30.136334    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-77254d/test-model-77254d-kserve-5476b887bf-2zjpk"
Oct 30 13:20:30.149290 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:30.149255    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-77254d/test-model-77254d-kserve-5476b887bf-2zjpk"
Oct 30 13:20:30.243543 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:30.243475    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-77254d/test-model-77254d-kserve-5476b887bf-2zjpk"
Oct 30 13:20:30.319397 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:30.319371    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8467ed-76766c9c88-cbg5d"]
Oct 30 13:20:30.320973 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:20:30.320952    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc880b1c_4413_4369_950b_a4bb3eb62568.slice/crio-e69ee59df6e7dfe30e0f81594c17c23a57e1a4c6ee3211462559a746dac22499 WatchSource:0}: Error finding container e69ee59df6e7dfe30e0f81594c17c23a57e1a4c6ee3211462559a746dac22499: Status 404 returned error can't find the container with id e69ee59df6e7dfe30e0f81594c17c23a57e1a4c6ee3211462559a746dac22499
Oct 30 13:20:30.322693 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:30.322669    4364 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 13:20:31.240916 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:31.240883    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5dd8cc-6c7f8464c9-2jtlq"]
Oct 30 13:20:31.245332 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:31.245304    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8467ed-76766c9c88-cbg5d" event={"ID":"fc880b1c-4413-4369-950b-a4bb3eb62568","Type":"ContainerStarted","Data":"346618995aa7919ab0f0c3800c4c664682b81aa127d076025b041df3533d7b04"}
Oct 30 13:20:31.245475 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:31.245341    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8467ed-76766c9c88-cbg5d" event={"ID":"fc880b1c-4413-4369-950b-a4bb3eb62568","Type":"ContainerStarted","Data":"e69ee59df6e7dfe30e0f81594c17c23a57e1a4c6ee3211462559a746dac22499"}
Oct 30 13:20:31.245475 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:31.245395    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5dd8cc-6c7f8464c9-2jtlq"
Oct 30 13:20:31.257504 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:31.257472    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:20:31.262965 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:31.262941    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-5dd8cc\""
Oct 30 13:20:31.263200 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:31.263175    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-5dd8cc-dockercfg-kkm24\""
Oct 30 13:20:31.265431 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:31.265398    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5dd8cc-6c7f8464c9-2jtlq"]
Oct 30 13:20:31.294632 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:31.294562    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8467ed-76766c9c88-cbg5d" podStartSLOduration=1.9192631759999998 podStartE2EDuration="2.294548492s" totalImagesPullingTime="375.285316ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:20:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:20:31.289352184 +0000 UTC m=+1730.455716834" watchObservedRunningTime="2026-10-30 13:20:31.294548492 +0000 UTC m=+1730.460913140"
Oct 30 13:20:31.402224 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:31.402176    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggqjs\" (UniqueName: \"kubernetes.io/projected/a9c7d05a-ebeb-4506-88f9-5932b1730822-kube-api-access-ggqjs\") pod \"payload-pre-processing-e2e-inf-b-5dd8cc-6c7f8464c9-2jtlq\" (UID: \"a9c7d05a-ebeb-4506-88f9-5932b1730822\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5dd8cc-6c7f8464c9-2jtlq"
Oct 30 13:20:31.402420 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:31.402330    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9c7d05a-ebeb-4506-88f9-5932b1730822-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-5dd8cc-6c7f8464c9-2jtlq\" (UID: \"a9c7d05a-ebeb-4506-88f9-5932b1730822\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5dd8cc-6c7f8464c9-2jtlq"
Oct 30 13:20:31.578697 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:31.505087    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggqjs\" (UniqueName: \"kubernetes.io/projected/a9c7d05a-ebeb-4506-88f9-5932b1730822-kube-api-access-ggqjs\") pod \"payload-pre-processing-e2e-inf-b-5dd8cc-6c7f8464c9-2jtlq\" (UID: \"a9c7d05a-ebeb-4506-88f9-5932b1730822\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5dd8cc-6c7f8464c9-2jtlq"
Oct 30 13:20:31.612622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:31.601734    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9c7d05a-ebeb-4506-88f9-5932b1730822-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-5dd8cc-6c7f8464c9-2jtlq\" (UID: \"a9c7d05a-ebeb-4506-88f9-5932b1730822\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5dd8cc-6c7f8464c9-2jtlq"
Oct 30 13:20:31.674622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:31.662365    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9c7d05a-ebeb-4506-88f9-5932b1730822-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-5dd8cc-6c7f8464c9-2jtlq\" (UID: \"a9c7d05a-ebeb-4506-88f9-5932b1730822\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5dd8cc-6c7f8464c9-2jtlq"
Oct 30 13:20:31.825587 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:31.825410    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggqjs\" (UniqueName: \"kubernetes.io/projected/a9c7d05a-ebeb-4506-88f9-5932b1730822-kube-api-access-ggqjs\") pod \"payload-pre-processing-e2e-inf-b-5dd8cc-6c7f8464c9-2jtlq\" (UID: \"a9c7d05a-ebeb-4506-88f9-5932b1730822\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5dd8cc-6c7f8464c9-2jtlq"
Oct 30 13:20:31.862450 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:31.862360    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5dd8cc-6c7f8464c9-2jtlq"
Oct 30 13:20:32.201591 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:20:32.201558    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9c7d05a_ebeb_4506_88f9_5932b1730822.slice/crio-812b790e517fae219de8f3666a7635929a4f8265eb343af5ded6e0994bf9b4c3 WatchSource:0}: Error finding container 812b790e517fae219de8f3666a7635929a4f8265eb343af5ded6e0994bf9b4c3: Status 404 returned error can't find the container with id 812b790e517fae219de8f3666a7635929a4f8265eb343af5ded6e0994bf9b4c3
Oct 30 13:20:32.201732 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:32.201652    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5dd8cc-6c7f8464c9-2jtlq"]
Oct 30 13:20:32.255188 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:32.255155    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5dd8cc-6c7f8464c9-2jtlq" event={"ID":"a9c7d05a-ebeb-4506-88f9-5932b1730822","Type":"ContainerStarted","Data":"812b790e517fae219de8f3666a7635929a4f8265eb343af5ded6e0994bf9b4c3"}
Oct 30 13:20:32.255525 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:32.255401    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8467ed-76766c9c88-cbg5d"
Oct 30 13:20:33.349624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:33.347679    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5dd8cc-6c7f8464c9-2jtlq" event={"ID":"a9c7d05a-ebeb-4506-88f9-5932b1730822","Type":"ContainerStarted","Data":"60b8d619a5c60919a59a4ab2d505befec14fd56c434e06bcb233a7272ca88444"}
Oct 30 13:20:33.349624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:33.347777    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5dd8cc-6c7f8464c9-2jtlq"
Oct 30 13:20:33.455622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:33.440737    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5dd8cc-6c7f8464c9-2jtlq" podStartSLOduration=2.440717789 podStartE2EDuration="2.440717789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:20:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:20:33.402848128 +0000 UTC m=+1732.569212777" watchObservedRunningTime="2026-10-30 13:20:33.440717789 +0000 UTC m=+1732.607082440"
Oct 30 13:20:34.154652 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:34.146990    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-4651e8-67665669fd-qdsnz"
Oct 30 13:20:34.154652 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:34.148776    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4651e8-8f64b59bc-2xng9"
Oct 30 13:20:38.351512 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:38.351480    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8467ed-76766c9c88-cbg5d"
Oct 30 13:20:40.033975 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:40.033943    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3d3ba8b-q2sw5"]
Oct 30 13:20:40.045743 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:40.045706    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3d3ba8b-q2sw5"]
Oct 30 13:20:40.982014 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:40.980415    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a16d3810-dcc1-480d-b421-1f5ee1ac3da6" path="/var/lib/kubelet/pods/a16d3810-dcc1-480d-b421-1f5ee1ac3da6/volumes"
Oct 30 13:20:41.516954 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:20:41.516912    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:20:42.203761 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:42.203727    4364 scope.go:122] "RemoveContainer" containerID="898750e5cc78fdbee11c9a70b19b7b9ab00516e1d8178dcf98fbb6b6fd36ad71"
Oct 30 13:20:43.503661 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:43.503629    4364 generic.go:356] "Generic (PLEG): container finished" podID="945bfd4e-0d0d-499f-8457-5f37a0c48e28" containerID="632629d92e8dc7487e27cd045c846c6f44737ecfe34b41e6779ba9ddb88775af" exitCode=0
Oct 30 13:20:43.504062 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:43.503697    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx" event={"ID":"945bfd4e-0d0d-499f-8457-5f37a0c48e28","Type":"ContainerDied","Data":"632629d92e8dc7487e27cd045c846c6f44737ecfe34b41e6779ba9ddb88775af"}
Oct 30 13:20:43.504268 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:43.504249    4364 scope.go:122] "RemoveContainer" containerID="632629d92e8dc7487e27cd045c846c6f44737ecfe34b41e6779ba9ddb88775af"
Oct 30 13:20:43.506149 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:43.506120    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:20:43.506218 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:43.506190    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:20:43.506265 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:43.506221    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:20:44.515249 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:44.515219    4364 generic.go:356] "Generic (PLEG): container finished" podID="fd17aa36-e0ae-4652-a85f-20945233ccc0" containerID="b96ac07b11db5c0d57da34eada5d105abdfb0f1c9e05bd72d8f06f3357c2d022" exitCode=0
Oct 30 13:20:44.515765 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:44.515295    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2" event={"ID":"fd17aa36-e0ae-4652-a85f-20945233ccc0","Type":"ContainerDied","Data":"b96ac07b11db5c0d57da34eada5d105abdfb0f1c9e05bd72d8f06f3357c2d022"}
Oct 30 13:20:44.515875 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:44.515857    4364 scope.go:122] "RemoveContainer" containerID="b96ac07b11db5c0d57da34eada5d105abdfb0f1c9e05bd72d8f06f3357c2d022"
Oct 30 13:20:44.517844 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:44.517817    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:20:44.517906 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:44.517886    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:20:44.517947 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:44.517918    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:20:44.518943 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:44.518922    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx" event={"ID":"945bfd4e-0d0d-499f-8457-5f37a0c48e28","Type":"ContainerStarted","Data":"ddfca0d35579eff6abf7a1ce3551a4bfc65fbcb296aa7d88317e07b21358ea88"}
Oct 30 13:20:44.616705 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:44.616664    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx"
Oct 30 13:20:44.616876 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:44.616746    4364 patch_prober.go:29] interesting pod/e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Oct 30 13:20:44.616876 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:44.616788    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx" podUID="945bfd4e-0d0d-499f-8457-5f37a0c48e28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Oct 30 13:20:45.368946 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:45.368914    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5dd8cc-6c7f8464c9-2jtlq"
Oct 30 13:20:45.573115 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:45.572227    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2" event={"ID":"fd17aa36-e0ae-4652-a85f-20945233ccc0","Type":"ContainerStarted","Data":"dde43c13d515a38bb07a29af73448f8a91f8d16989b8908d1f16060414f5b88d"}
Oct 30 13:20:45.609619 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:45.608663    4364 patch_prober.go:29] interesting pod/e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Oct 30 13:20:45.609619 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:45.608740    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx" podUID="945bfd4e-0d0d-499f-8457-5f37a0c48e28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Oct 30 13:20:45.749888 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:45.749384    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942"]
Oct 30 13:20:45.754776 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:45.754742    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942"
Oct 30 13:20:45.787138 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:45.783018    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-8d82d5-openshift-default-dockercfg-w5n4b\""
Oct 30 13:20:45.833623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:45.827859    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942"]
Oct 30 13:20:45.875622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:45.858874    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k242\" (UniqueName: \"kubernetes.io/projected/654161bd-9cf8-4f63-9a86-b312763e75cc-kube-api-access-8k242\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942\" (UID: \"654161bd-9cf8-4f63-9a86-b312763e75cc\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942"
Oct 30 13:20:45.875622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:45.858978    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/654161bd-9cf8-4f63-9a86-b312763e75cc-istiod-ca-cert\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942\" (UID: \"654161bd-9cf8-4f63-9a86-b312763e75cc\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942"
Oct 30 13:20:45.875622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:45.859037    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/654161bd-9cf8-4f63-9a86-b312763e75cc-credential-socket\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942\" (UID: \"654161bd-9cf8-4f63-9a86-b312763e75cc\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942"
Oct 30 13:20:45.875622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:45.859069    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/654161bd-9cf8-4f63-9a86-b312763e75cc-workload-certs\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942\" (UID: \"654161bd-9cf8-4f63-9a86-b312763e75cc\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942"
Oct 30 13:20:45.875622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:45.859239    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/654161bd-9cf8-4f63-9a86-b312763e75cc-istio-envoy\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942\" (UID: \"654161bd-9cf8-4f63-9a86-b312763e75cc\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942"
Oct 30 13:20:45.875622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:45.859280    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/654161bd-9cf8-4f63-9a86-b312763e75cc-istio-token\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942\" (UID: \"654161bd-9cf8-4f63-9a86-b312763e75cc\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942"
Oct 30 13:20:45.875622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:45.859384    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/654161bd-9cf8-4f63-9a86-b312763e75cc-workload-socket\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942\" (UID: \"654161bd-9cf8-4f63-9a86-b312763e75cc\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942"
Oct 30 13:20:45.875622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:45.859446    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/654161bd-9cf8-4f63-9a86-b312763e75cc-istio-data\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942\" (UID: \"654161bd-9cf8-4f63-9a86-b312763e75cc\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942"
Oct 30 13:20:45.875622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:45.859582    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/654161bd-9cf8-4f63-9a86-b312763e75cc-istio-podinfo\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942\" (UID: \"654161bd-9cf8-4f63-9a86-b312763e75cc\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942"
Oct 30 13:20:45.957618 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:45.943438    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2"
Oct 30 13:20:45.957618 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:45.943552    4364 patch_prober.go:29] interesting pod/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 30 13:20:45.957618 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:45.943625    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2" podUID="fd17aa36-e0ae-4652-a85f-20945233ccc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 30 13:20:45.979714 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:45.978197    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/654161bd-9cf8-4f63-9a86-b312763e75cc-istio-podinfo\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942\" (UID: \"654161bd-9cf8-4f63-9a86-b312763e75cc\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942"
Oct 30 13:20:45.979714 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:45.978267    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k242\" (UniqueName: \"kubernetes.io/projected/654161bd-9cf8-4f63-9a86-b312763e75cc-kube-api-access-8k242\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942\" (UID: \"654161bd-9cf8-4f63-9a86-b312763e75cc\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942"
Oct 30 13:20:45.979714 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:45.978339    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/654161bd-9cf8-4f63-9a86-b312763e75cc-istiod-ca-cert\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942\" (UID: \"654161bd-9cf8-4f63-9a86-b312763e75cc\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942"
Oct 30 13:20:45.979714 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:45.978395    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/654161bd-9cf8-4f63-9a86-b312763e75cc-credential-socket\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942\" (UID: \"654161bd-9cf8-4f63-9a86-b312763e75cc\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942"
Oct 30 13:20:45.979714 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:45.978428    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/654161bd-9cf8-4f63-9a86-b312763e75cc-workload-certs\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942\" (UID: \"654161bd-9cf8-4f63-9a86-b312763e75cc\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942"
Oct 30 13:20:45.979714 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:45.978592    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/654161bd-9cf8-4f63-9a86-b312763e75cc-istio-envoy\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942\" (UID: \"654161bd-9cf8-4f63-9a86-b312763e75cc\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942"
Oct 30 13:20:45.979714 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:45.978655    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/654161bd-9cf8-4f63-9a86-b312763e75cc-istio-token\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942\" (UID: \"654161bd-9cf8-4f63-9a86-b312763e75cc\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942"
Oct 30 13:20:45.979714 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:45.978761    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/654161bd-9cf8-4f63-9a86-b312763e75cc-workload-socket\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942\" (UID: \"654161bd-9cf8-4f63-9a86-b312763e75cc\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942"
Oct 30 13:20:45.979714 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:45.978817    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/654161bd-9cf8-4f63-9a86-b312763e75cc-istio-data\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942\" (UID: \"654161bd-9cf8-4f63-9a86-b312763e75cc\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942"
Oct 30 13:20:45.979714 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:45.979297    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/654161bd-9cf8-4f63-9a86-b312763e75cc-istio-data\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942\" (UID: \"654161bd-9cf8-4f63-9a86-b312763e75cc\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942"
Oct 30 13:20:45.991622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:45.981242    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/654161bd-9cf8-4f63-9a86-b312763e75cc-workload-certs\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942\" (UID: \"654161bd-9cf8-4f63-9a86-b312763e75cc\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942"
Oct 30 13:20:46.025707 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:45.992489    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/654161bd-9cf8-4f63-9a86-b312763e75cc-istiod-ca-cert\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942\" (UID: \"654161bd-9cf8-4f63-9a86-b312763e75cc\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942"
Oct 30 13:20:46.025707 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:45.992805    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/654161bd-9cf8-4f63-9a86-b312763e75cc-credential-socket\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942\" (UID: \"654161bd-9cf8-4f63-9a86-b312763e75cc\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942"
Oct 30 13:20:46.025707 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:46.005385    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/654161bd-9cf8-4f63-9a86-b312763e75cc-workload-socket\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942\" (UID: \"654161bd-9cf8-4f63-9a86-b312763e75cc\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942"
Oct 30 13:20:46.160384 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:46.096240    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/654161bd-9cf8-4f63-9a86-b312763e75cc-istio-podinfo\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942\" (UID: \"654161bd-9cf8-4f63-9a86-b312763e75cc\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942"
Oct 30 13:20:46.268667 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:46.267157    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/654161bd-9cf8-4f63-9a86-b312763e75cc-istio-token\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942\" (UID: \"654161bd-9cf8-4f63-9a86-b312763e75cc\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942"
Oct 30 13:20:46.496671 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:46.486710    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k242\" (UniqueName: \"kubernetes.io/projected/654161bd-9cf8-4f63-9a86-b312763e75cc-kube-api-access-8k242\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942\" (UID: \"654161bd-9cf8-4f63-9a86-b312763e75cc\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942"
Oct 30 13:20:46.519650 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:46.508613    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/654161bd-9cf8-4f63-9a86-b312763e75cc-istio-envoy\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942\" (UID: \"654161bd-9cf8-4f63-9a86-b312763e75cc\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942"
Oct 30 13:20:46.634383 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:46.632791    4364 patch_prober.go:29] interesting pod/e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Oct 30 13:20:46.634383 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:46.632869    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx" podUID="945bfd4e-0d0d-499f-8457-5f37a0c48e28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Oct 30 13:20:46.671220 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:46.671190    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942"
Oct 30 13:20:46.873774 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:46.872864    4364 patch_prober.go:29] interesting pod/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 30 13:20:46.873774 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:46.872927    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2" podUID="fd17aa36-e0ae-4652-a85f-20945233ccc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 30 13:20:47.075211 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:20:47.075176    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod654161bd_9cf8_4f63_9a86_b312763e75cc.slice/crio-8a700846e9fc29ee39c5a14736c18535ef62b64609cec205a586f7c4800af125 WatchSource:0}: Error finding container 8a700846e9fc29ee39c5a14736c18535ef62b64609cec205a586f7c4800af125: Status 404 returned error can't find the container with id 8a700846e9fc29ee39c5a14736c18535ef62b64609cec205a586f7c4800af125
Oct 30 13:20:47.076536 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:47.076508    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942"]
Oct 30 13:20:47.078565 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:47.078533    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:20:47.090725 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:47.078633    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:20:47.090725 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:47.078681    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:20:47.628095 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:47.609985    4364 patch_prober.go:29] interesting pod/e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Oct 30 13:20:47.628095 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:47.610048    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx" podUID="945bfd4e-0d0d-499f-8457-5f37a0c48e28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Oct 30 13:20:47.711624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:47.698069    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942" event={"ID":"654161bd-9cf8-4f63-9a86-b312763e75cc","Type":"ContainerStarted","Data":"c61d09677f91d362767a518d53c66a58b0fac3495ef391ec0421b63a1c4a6c95"}
Oct 30 13:20:47.711624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:47.698153    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942" event={"ID":"654161bd-9cf8-4f63-9a86-b312763e75cc","Type":"ContainerStarted","Data":"8a700846e9fc29ee39c5a14736c18535ef62b64609cec205a586f7c4800af125"}
Oct 30 13:20:47.843625 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:47.843267    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942" podStartSLOduration=2.843246458 podStartE2EDuration="2.843246458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:20:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:20:47.791395825 +0000 UTC m=+1746.957760466" watchObservedRunningTime="2026-10-30 13:20:47.843246458 +0000 UTC m=+1747.009611107"
Oct 30 13:20:47.860460 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:47.860428    4364 patch_prober.go:29] interesting pod/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 30 13:20:47.860631 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:47.860486    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2" podUID="fd17aa36-e0ae-4652-a85f-20945233ccc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 30 13:20:48.610734 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:48.610696    4364 patch_prober.go:29] interesting pod/e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Oct 30 13:20:48.611023 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:48.610993    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx" podUID="945bfd4e-0d0d-499f-8457-5f37a0c48e28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Oct 30 13:20:48.671591 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:48.671567    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942"
Oct 30 13:20:48.676923 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:48.676902    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942"
Oct 30 13:20:48.733621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:48.732785    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942"
Oct 30 13:20:48.734795 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:48.734588    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942"
Oct 30 13:20:48.860913 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:48.860831    4364 patch_prober.go:29] interesting pod/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 30 13:20:48.860913 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:48.860883    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2" podUID="fd17aa36-e0ae-4652-a85f-20945233ccc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 30 13:20:49.617622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:49.617152    4364 patch_prober.go:29] interesting pod/e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Oct 30 13:20:49.617622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:49.617208    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx" podUID="945bfd4e-0d0d-499f-8457-5f37a0c48e28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Oct 30 13:20:49.859921 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:49.859886    4364 patch_prober.go:29] interesting pod/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 30 13:20:49.860415 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:49.859937    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2" podUID="fd17aa36-e0ae-4652-a85f-20945233ccc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 30 13:20:50.692968 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:50.692932    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx"
Oct 30 13:20:50.699015 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:50.698983    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx"
Oct 30 13:20:50.709898 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:50.709874    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx"
Oct 30 13:20:50.860290 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:50.860259    4364 patch_prober.go:29] interesting pod/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 30 13:20:50.860776 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:50.860313    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2" podUID="fd17aa36-e0ae-4652-a85f-20945233ccc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 30 13:20:51.320821 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:51.320790    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2"
Oct 30 13:20:51.862531 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:51.861701    4364 patch_prober.go:29] interesting pod/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 30 13:20:51.862531 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:51.861759    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2" podUID="fd17aa36-e0ae-4652-a85f-20945233ccc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 30 13:20:52.865456 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:52.865425    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2"
Oct 30 13:20:52.866495 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:52.866472    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2"
Oct 30 13:20:59.085950 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:59.085879    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-09e4c0-5454d677c9-bqbgj"]
Oct 30 13:20:59.090270 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:59.090251    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-09e4c0-5454d677c9-bqbgj"
Oct 30 13:20:59.113165 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:59.113137    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-09e4c0-5454d677c9-bqbgj"]
Oct 30 13:20:59.117750 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:59.117727    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-09e4c0-dockercfg-scsgn\""
Oct 30 13:20:59.117915 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:59.117760    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-09e4c0\""
Oct 30 13:20:59.230622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:59.207061    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79263302-817d-49f0-91a0-43281bff1ebd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-09e4c0-5454d677c9-bqbgj\" (UID: \"79263302-817d-49f0-91a0-43281bff1ebd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-09e4c0-5454d677c9-bqbgj"
Oct 30 13:20:59.230958 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:59.230926    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npf2g\" (UniqueName: \"kubernetes.io/projected/79263302-817d-49f0-91a0-43281bff1ebd-kube-api-access-npf2g\") pod \"payload-pre-processing-e2e-inf-b-09e4c0-5454d677c9-bqbgj\" (UID: \"79263302-817d-49f0-91a0-43281bff1ebd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-09e4c0-5454d677c9-bqbgj"
Oct 30 13:20:59.429388 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:59.417852    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79263302-817d-49f0-91a0-43281bff1ebd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-09e4c0-5454d677c9-bqbgj\" (UID: \"79263302-817d-49f0-91a0-43281bff1ebd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-09e4c0-5454d677c9-bqbgj"
Oct 30 13:20:59.429815 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:59.429787    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npf2g\" (UniqueName: \"kubernetes.io/projected/79263302-817d-49f0-91a0-43281bff1ebd-kube-api-access-npf2g\") pod \"payload-pre-processing-e2e-inf-b-09e4c0-5454d677c9-bqbgj\" (UID: \"79263302-817d-49f0-91a0-43281bff1ebd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-09e4c0-5454d677c9-bqbgj"
Oct 30 13:20:59.430849 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:59.425475    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79263302-817d-49f0-91a0-43281bff1ebd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-09e4c0-5454d677c9-bqbgj\" (UID: \"79263302-817d-49f0-91a0-43281bff1ebd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-09e4c0-5454d677c9-bqbgj"
Oct 30 13:20:59.508479 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:59.508445    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npf2g\" (UniqueName: \"kubernetes.io/projected/79263302-817d-49f0-91a0-43281bff1ebd-kube-api-access-npf2g\") pod \"payload-pre-processing-e2e-inf-b-09e4c0-5454d677c9-bqbgj\" (UID: \"79263302-817d-49f0-91a0-43281bff1ebd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-09e4c0-5454d677c9-bqbgj"
Oct 30 13:20:59.704257 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:20:59.704181    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-09e4c0-5454d677c9-bqbgj"
Oct 30 13:21:00.023465 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:00.023441    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-09e4c0-5454d677c9-bqbgj"]
Oct 30 13:21:00.024021 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:21:00.023999    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79263302_817d_49f0_91a0_43281bff1ebd.slice/crio-ba63d80bc5b3b524d38e796b3aeaa1c95ab55e8ac4054268e405c9351336903b WatchSource:0}: Error finding container ba63d80bc5b3b524d38e796b3aeaa1c95ab55e8ac4054268e405c9351336903b: Status 404 returned error can't find the container with id ba63d80bc5b3b524d38e796b3aeaa1c95ab55e8ac4054268e405c9351336903b
Oct 30 13:21:01.030937 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:01.030891    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-09e4c0-5454d677c9-bqbgj" event={"ID":"79263302-817d-49f0-91a0-43281bff1ebd","Type":"ContainerStarted","Data":"a984316c0e942feb36207be82bcfc69f877d2610eb7a7ae31d1dcbd19d381e16"}
Oct 30 13:21:01.031332 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:01.030953    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-09e4c0-5454d677c9-bqbgj" event={"ID":"79263302-817d-49f0-91a0-43281bff1ebd","Type":"ContainerStarted","Data":"ba63d80bc5b3b524d38e796b3aeaa1c95ab55e8ac4054268e405c9351336903b"}
Oct 30 13:21:01.031332 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:01.031066    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-09e4c0-5454d677c9-bqbgj"
Oct 30 13:21:01.072012 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:01.071962    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-09e4c0-5454d677c9-bqbgj" podStartSLOduration=2.0719493350000002 podStartE2EDuration="2.071949335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:20:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:21:01.070651537 +0000 UTC m=+1760.237016182" watchObservedRunningTime="2026-10-30 13:21:01.071949335 +0000 UTC m=+1760.238313983"
Oct 30 13:21:11.090635 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:11.090570    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g"]
Oct 30 13:21:11.091106 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:11.090834    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g" podUID="370ee2a9-2643-4e24-81e4-ab56f3caefb3" containerName="istio-proxy" containerID="cri-o://8d6e9592a05c1a4f1a42b58fc2bb8fadb0c66eef23ed01c3bbb6e7b3b0fb2317" gracePeriod=30
Oct 30 13:21:11.334784 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:11.334745    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7"]
Oct 30 13:21:11.335097 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:11.335044    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7" podUID="134eb93c-b6ee-43e9-8977-950cb0188887" containerName="istio-proxy" containerID="cri-o://97cbae683a7c99fa36188a38ee1fb80a181bdd1f0bffcfdf8f0a4e5dd8940371" gracePeriod=30
Oct 30 13:21:11.507803 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:21:11.507760    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:21:12.042695 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:12.042663    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-09e4c0-5454d677c9-bqbgj"
Oct 30 13:21:15.702499 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:15.702462    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5dd8cc-657cd56d9c-wps6v"]
Oct 30 13:21:15.707536 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:15.707515    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5dd8cc-657cd56d9c-wps6v"
Oct 30 13:21:15.734362 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:15.734318    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-5dd8cc-serving-cert\""
Oct 30 13:21:15.755642 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:15.755580    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5dd8cc-657cd56d9c-wps6v"]
Oct 30 13:21:15.824153 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:15.821719    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/873a9c3f-d2a1-4eb8-a413-216e4d5cb48c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5dd8cc-657cd56d9c-wps6v\" (UID: \"873a9c3f-d2a1-4eb8-a413-216e4d5cb48c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5dd8cc-657cd56d9c-wps6v"
Oct 30 13:21:15.824153 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:15.821890    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/873a9c3f-d2a1-4eb8-a413-216e4d5cb48c-maas-api-tls\") pod \"maas-api-e2e-inf-b-5dd8cc-657cd56d9c-wps6v\" (UID: \"873a9c3f-d2a1-4eb8-a413-216e4d5cb48c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5dd8cc-657cd56d9c-wps6v"
Oct 30 13:21:15.824153 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:15.822009    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/873a9c3f-d2a1-4eb8-a413-216e4d5cb48c-metrics-certs\") pod \"maas-api-e2e-inf-b-5dd8cc-657cd56d9c-wps6v\" (UID: \"873a9c3f-d2a1-4eb8-a413-216e4d5cb48c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5dd8cc-657cd56d9c-wps6v"
Oct 30 13:21:15.824153 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:15.822153    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8p6k\" (UniqueName: \"kubernetes.io/projected/873a9c3f-d2a1-4eb8-a413-216e4d5cb48c-kube-api-access-r8p6k\") pod \"maas-api-e2e-inf-b-5dd8cc-657cd56d9c-wps6v\" (UID: \"873a9c3f-d2a1-4eb8-a413-216e4d5cb48c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5dd8cc-657cd56d9c-wps6v"
Oct 30 13:21:15.974964 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:15.974871    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/873a9c3f-d2a1-4eb8-a413-216e4d5cb48c-maas-api-tls\") pod \"maas-api-e2e-inf-b-5dd8cc-657cd56d9c-wps6v\" (UID: \"873a9c3f-d2a1-4eb8-a413-216e4d5cb48c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5dd8cc-657cd56d9c-wps6v"
Oct 30 13:21:16.004649 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:16.003805    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/873a9c3f-d2a1-4eb8-a413-216e4d5cb48c-metrics-certs\") pod \"maas-api-e2e-inf-b-5dd8cc-657cd56d9c-wps6v\" (UID: \"873a9c3f-d2a1-4eb8-a413-216e4d5cb48c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5dd8cc-657cd56d9c-wps6v"
Oct 30 13:21:16.004649 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:16.004040    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8p6k\" (UniqueName: \"kubernetes.io/projected/873a9c3f-d2a1-4eb8-a413-216e4d5cb48c-kube-api-access-r8p6k\") pod \"maas-api-e2e-inf-b-5dd8cc-657cd56d9c-wps6v\" (UID: \"873a9c3f-d2a1-4eb8-a413-216e4d5cb48c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5dd8cc-657cd56d9c-wps6v"
Oct 30 13:21:16.004649 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:16.004129    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/873a9c3f-d2a1-4eb8-a413-216e4d5cb48c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5dd8cc-657cd56d9c-wps6v\" (UID: \"873a9c3f-d2a1-4eb8-a413-216e4d5cb48c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5dd8cc-657cd56d9c-wps6v"
Oct 30 13:21:16.005095 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:16.005066    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/873a9c3f-d2a1-4eb8-a413-216e4d5cb48c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5dd8cc-657cd56d9c-wps6v\" (UID: \"873a9c3f-d2a1-4eb8-a413-216e4d5cb48c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5dd8cc-657cd56d9c-wps6v"
Oct 30 13:21:16.014429 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:16.014399    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/873a9c3f-d2a1-4eb8-a413-216e4d5cb48c-maas-api-tls\") pod \"maas-api-e2e-inf-b-5dd8cc-657cd56d9c-wps6v\" (UID: \"873a9c3f-d2a1-4eb8-a413-216e4d5cb48c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5dd8cc-657cd56d9c-wps6v"
Oct 30 13:21:16.057361 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:16.057322    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8p6k\" (UniqueName: \"kubernetes.io/projected/873a9c3f-d2a1-4eb8-a413-216e4d5cb48c-kube-api-access-r8p6k\") pod \"maas-api-e2e-inf-b-5dd8cc-657cd56d9c-wps6v\" (UID: \"873a9c3f-d2a1-4eb8-a413-216e4d5cb48c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5dd8cc-657cd56d9c-wps6v"
Oct 30 13:21:16.072634 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:16.067310    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/873a9c3f-d2a1-4eb8-a413-216e4d5cb48c-metrics-certs\") pod \"maas-api-e2e-inf-b-5dd8cc-657cd56d9c-wps6v\" (UID: \"873a9c3f-d2a1-4eb8-a413-216e4d5cb48c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5dd8cc-657cd56d9c-wps6v"
Oct 30 13:21:16.223533 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:16.223505    4364 generic.go:356] "Generic (PLEG): container finished" podID="370ee2a9-2643-4e24-81e4-ab56f3caefb3" containerID="8d6e9592a05c1a4f1a42b58fc2bb8fadb0c66eef23ed01c3bbb6e7b3b0fb2317" exitCode=0
Oct 30 13:21:16.223711 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:16.223553    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g" event={"ID":"370ee2a9-2643-4e24-81e4-ab56f3caefb3","Type":"ContainerDied","Data":"8d6e9592a05c1a4f1a42b58fc2bb8fadb0c66eef23ed01c3bbb6e7b3b0fb2317"}
Oct 30 13:21:16.223711 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:16.223626    4364 scope.go:122] "RemoveContainer" containerID="294b3d510361d2ce3da926630f9b2f86e2f8539a6c5aac27ae90f6e2b0b5b2bc"
Oct 30 13:21:16.330099 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:16.330022    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5dd8cc-657cd56d9c-wps6v"
Oct 30 13:21:16.462338 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:16.461948    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g"
Oct 30 13:21:16.618465 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:16.618431    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/370ee2a9-2643-4e24-81e4-ab56f3caefb3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/370ee2a9-2643-4e24-81e4-ab56f3caefb3-istiod-ca-cert\") pod \"370ee2a9-2643-4e24-81e4-ab56f3caefb3\" (UID: \"370ee2a9-2643-4e24-81e4-ab56f3caefb3\") "
Oct 30 13:21:16.618689 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:16.618470    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/370ee2a9-2643-4e24-81e4-ab56f3caefb3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/370ee2a9-2643-4e24-81e4-ab56f3caefb3-istio-data\") pod \"370ee2a9-2643-4e24-81e4-ab56f3caefb3\" (UID: \"370ee2a9-2643-4e24-81e4-ab56f3caefb3\") "
Oct 30 13:21:16.618789 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:16.618721    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/370ee2a9-2643-4e24-81e4-ab56f3caefb3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/370ee2a9-2643-4e24-81e4-ab56f3caefb3-workload-socket\") pod \"370ee2a9-2643-4e24-81e4-ab56f3caefb3\" (UID: \"370ee2a9-2643-4e24-81e4-ab56f3caefb3\") "
Oct 30 13:21:16.618946 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:16.618774    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/370ee2a9-2643-4e24-81e4-ab56f3caefb3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/370ee2a9-2643-4e24-81e4-ab56f3caefb3-credential-socket\") pod \"370ee2a9-2643-4e24-81e4-ab56f3caefb3\" (UID: \"370ee2a9-2643-4e24-81e4-ab56f3caefb3\") "
Oct 30 13:21:16.618946 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:16.618833    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/370ee2a9-2643-4e24-81e4-ab56f3caefb3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/370ee2a9-2643-4e24-81e4-ab56f3caefb3-workload-certs\") pod \"370ee2a9-2643-4e24-81e4-ab56f3caefb3\" (UID: \"370ee2a9-2643-4e24-81e4-ab56f3caefb3\") "
Oct 30 13:21:16.619177 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:16.618970    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/370ee2a9-2643-4e24-81e4-ab56f3caefb3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/370ee2a9-2643-4e24-81e4-ab56f3caefb3-istio-podinfo\") pod \"370ee2a9-2643-4e24-81e4-ab56f3caefb3\" (UID: \"370ee2a9-2643-4e24-81e4-ab56f3caefb3\") "
Oct 30 13:21:16.619264 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:16.619238    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/370ee2a9-2643-4e24-81e4-ab56f3caefb3-kube-api-access-26z2l\" (UniqueName: \"kubernetes.io/projected/370ee2a9-2643-4e24-81e4-ab56f3caefb3-kube-api-access-26z2l\") pod \"370ee2a9-2643-4e24-81e4-ab56f3caefb3\" (UID: \"370ee2a9-2643-4e24-81e4-ab56f3caefb3\") "
Oct 30 13:21:16.619329 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:16.619276    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/370ee2a9-2643-4e24-81e4-ab56f3caefb3-istio-token\" (UniqueName: \"kubernetes.io/projected/370ee2a9-2643-4e24-81e4-ab56f3caefb3-istio-token\") pod \"370ee2a9-2643-4e24-81e4-ab56f3caefb3\" (UID: \"370ee2a9-2643-4e24-81e4-ab56f3caefb3\") "
Oct 30 13:21:16.619390 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:16.619332    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/370ee2a9-2643-4e24-81e4-ab56f3caefb3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/370ee2a9-2643-4e24-81e4-ab56f3caefb3-istio-envoy\") pod \"370ee2a9-2643-4e24-81e4-ab56f3caefb3\" (UID: \"370ee2a9-2643-4e24-81e4-ab56f3caefb3\") "
Oct 30 13:21:16.621004 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:16.619674    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/370ee2a9-2643-4e24-81e4-ab56f3caefb3-workload-certs" pod "370ee2a9-2643-4e24-81e4-ab56f3caefb3" (UID: "370ee2a9-2643-4e24-81e4-ab56f3caefb3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:21:16.621004 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:16.619797    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/370ee2a9-2643-4e24-81e4-ab56f3caefb3-workload-socket" pod "370ee2a9-2643-4e24-81e4-ab56f3caefb3" (UID: "370ee2a9-2643-4e24-81e4-ab56f3caefb3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:21:16.621004 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:16.619888    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/370ee2a9-2643-4e24-81e4-ab56f3caefb3-istiod-ca-cert" pod "370ee2a9-2643-4e24-81e4-ab56f3caefb3" (UID: "370ee2a9-2643-4e24-81e4-ab56f3caefb3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:21:16.621004 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:16.620059    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/370ee2a9-2643-4e24-81e4-ab56f3caefb3-credential-socket" pod "370ee2a9-2643-4e24-81e4-ab56f3caefb3" (UID: "370ee2a9-2643-4e24-81e4-ab56f3caefb3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:21:16.621004 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:16.620688    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/370ee2a9-2643-4e24-81e4-ab56f3caefb3-istio-data" pod "370ee2a9-2643-4e24-81e4-ab56f3caefb3" (UID: "370ee2a9-2643-4e24-81e4-ab56f3caefb3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:21:16.621004 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:16.620840    4364 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/370ee2a9-2643-4e24-81e4-ab56f3caefb3-istiod-ca-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:21:16.621004 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:16.620854    4364 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/370ee2a9-2643-4e24-81e4-ab56f3caefb3-istio-data\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:21:16.621004 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:16.620862    4364 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/370ee2a9-2643-4e24-81e4-ab56f3caefb3-workload-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:21:16.621004 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:16.620870    4364 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/370ee2a9-2643-4e24-81e4-ab56f3caefb3-credential-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:21:16.621004 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:16.620878    4364 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/370ee2a9-2643-4e24-81e4-ab56f3caefb3-workload-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:21:16.622794 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:16.622766    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/370ee2a9-2643-4e24-81e4-ab56f3caefb3-kube-api-access-26z2l" pod "370ee2a9-2643-4e24-81e4-ab56f3caefb3" (UID: "370ee2a9-2643-4e24-81e4-ab56f3caefb3"). InnerVolumeSpecName "kube-api-access-26z2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:21:16.623736 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:16.623695    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/370ee2a9-2643-4e24-81e4-ab56f3caefb3-istio-podinfo" pod "370ee2a9-2643-4e24-81e4-ab56f3caefb3" (UID: "370ee2a9-2643-4e24-81e4-ab56f3caefb3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:21:16.624504 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:16.624426    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/370ee2a9-2643-4e24-81e4-ab56f3caefb3-istio-token" pod "370ee2a9-2643-4e24-81e4-ab56f3caefb3" (UID: "370ee2a9-2643-4e24-81e4-ab56f3caefb3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:21:16.624943 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:16.624917    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/370ee2a9-2643-4e24-81e4-ab56f3caefb3-istio-envoy" pod "370ee2a9-2643-4e24-81e4-ab56f3caefb3" (UID: "370ee2a9-2643-4e24-81e4-ab56f3caefb3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:21:16.763843 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:16.723369    4364 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/370ee2a9-2643-4e24-81e4-ab56f3caefb3-istio-podinfo\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:21:16.763843 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:16.723405    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26z2l\" (UniqueName: \"kubernetes.io/projected/370ee2a9-2643-4e24-81e4-ab56f3caefb3-kube-api-access-26z2l\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:21:16.763843 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:16.723420    4364 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/370ee2a9-2643-4e24-81e4-ab56f3caefb3-istio-token\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:21:16.763843 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:16.723435    4364 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/370ee2a9-2643-4e24-81e4-ab56f3caefb3-istio-envoy\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:21:16.763843 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:16.745504    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5dd8cc-657cd56d9c-wps6v"]
Oct 30 13:21:16.904855 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:16.904777    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7"
Oct 30 13:21:17.038080 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:17.038050    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/134eb93c-b6ee-43e9-8977-950cb0188887-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/134eb93c-b6ee-43e9-8977-950cb0188887-istio-envoy\") pod \"134eb93c-b6ee-43e9-8977-950cb0188887\" (UID: \"134eb93c-b6ee-43e9-8977-950cb0188887\") "
Oct 30 13:21:17.038237 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:17.038091    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/134eb93c-b6ee-43e9-8977-950cb0188887-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/134eb93c-b6ee-43e9-8977-950cb0188887-workload-certs\") pod \"134eb93c-b6ee-43e9-8977-950cb0188887\" (UID: \"134eb93c-b6ee-43e9-8977-950cb0188887\") "
Oct 30 13:21:17.038237 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:17.038125    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/134eb93c-b6ee-43e9-8977-950cb0188887-kube-api-access-txpk9\" (UniqueName: \"kubernetes.io/projected/134eb93c-b6ee-43e9-8977-950cb0188887-kube-api-access-txpk9\") pod \"134eb93c-b6ee-43e9-8977-950cb0188887\" (UID: \"134eb93c-b6ee-43e9-8977-950cb0188887\") "
Oct 30 13:21:17.038237 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:17.038170    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/134eb93c-b6ee-43e9-8977-950cb0188887-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/134eb93c-b6ee-43e9-8977-950cb0188887-istio-podinfo\") pod \"134eb93c-b6ee-43e9-8977-950cb0188887\" (UID: \"134eb93c-b6ee-43e9-8977-950cb0188887\") "
Oct 30 13:21:17.038430 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:17.038354    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/134eb93c-b6ee-43e9-8977-950cb0188887-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/134eb93c-b6ee-43e9-8977-950cb0188887-workload-socket\") pod \"134eb93c-b6ee-43e9-8977-950cb0188887\" (UID: \"134eb93c-b6ee-43e9-8977-950cb0188887\") "
Oct 30 13:21:17.038430 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:17.038403    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/134eb93c-b6ee-43e9-8977-950cb0188887-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/134eb93c-b6ee-43e9-8977-950cb0188887-istiod-ca-cert\") pod \"134eb93c-b6ee-43e9-8977-950cb0188887\" (UID: \"134eb93c-b6ee-43e9-8977-950cb0188887\") "
Oct 30 13:21:17.038532 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:17.038431    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/134eb93c-b6ee-43e9-8977-950cb0188887-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/134eb93c-b6ee-43e9-8977-950cb0188887-credential-socket\") pod \"134eb93c-b6ee-43e9-8977-950cb0188887\" (UID: \"134eb93c-b6ee-43e9-8977-950cb0188887\") "
Oct 30 13:21:17.038532 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:17.038449    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/134eb93c-b6ee-43e9-8977-950cb0188887-istio-token\" (UniqueName: \"kubernetes.io/projected/134eb93c-b6ee-43e9-8977-950cb0188887-istio-token\") pod \"134eb93c-b6ee-43e9-8977-950cb0188887\" (UID: \"134eb93c-b6ee-43e9-8977-950cb0188887\") "
Oct 30 13:21:17.038532 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:17.038479    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/134eb93c-b6ee-43e9-8977-950cb0188887-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/134eb93c-b6ee-43e9-8977-950cb0188887-istio-data\") pod \"134eb93c-b6ee-43e9-8977-950cb0188887\" (UID: \"134eb93c-b6ee-43e9-8977-950cb0188887\") "
Oct 30 13:21:17.039229 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:17.039205    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/134eb93c-b6ee-43e9-8977-950cb0188887-istio-data" pod "134eb93c-b6ee-43e9-8977-950cb0188887" (UID: "134eb93c-b6ee-43e9-8977-950cb0188887"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:21:17.039805 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:17.039778    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/134eb93c-b6ee-43e9-8977-950cb0188887-workload-socket" pod "134eb93c-b6ee-43e9-8977-950cb0188887" (UID: "134eb93c-b6ee-43e9-8977-950cb0188887"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:21:17.040145 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:17.040118    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/134eb93c-b6ee-43e9-8977-950cb0188887-istiod-ca-cert" pod "134eb93c-b6ee-43e9-8977-950cb0188887" (UID: "134eb93c-b6ee-43e9-8977-950cb0188887"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:21:17.040238 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:17.040101    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/134eb93c-b6ee-43e9-8977-950cb0188887-workload-certs" pod "134eb93c-b6ee-43e9-8977-950cb0188887" (UID: "134eb93c-b6ee-43e9-8977-950cb0188887"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:21:17.040480 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:17.040459    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/134eb93c-b6ee-43e9-8977-950cb0188887-credential-socket" pod "134eb93c-b6ee-43e9-8977-950cb0188887" (UID: "134eb93c-b6ee-43e9-8977-950cb0188887"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:21:17.042338 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:17.042192    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/134eb93c-b6ee-43e9-8977-950cb0188887-istio-envoy" pod "134eb93c-b6ee-43e9-8977-950cb0188887" (UID: "134eb93c-b6ee-43e9-8977-950cb0188887"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:21:17.045886 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:17.045851    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/134eb93c-b6ee-43e9-8977-950cb0188887-istio-podinfo" pod "134eb93c-b6ee-43e9-8977-950cb0188887" (UID: "134eb93c-b6ee-43e9-8977-950cb0188887"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:21:17.045886 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:17.045868    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/134eb93c-b6ee-43e9-8977-950cb0188887-istio-token" pod "134eb93c-b6ee-43e9-8977-950cb0188887" (UID: "134eb93c-b6ee-43e9-8977-950cb0188887"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:21:17.047447 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:17.047428    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/134eb93c-b6ee-43e9-8977-950cb0188887-kube-api-access-txpk9" pod "134eb93c-b6ee-43e9-8977-950cb0188887" (UID: "134eb93c-b6ee-43e9-8977-950cb0188887"). InnerVolumeSpecName "kube-api-access-txpk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:21:17.165670 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:17.159801    4364 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/134eb93c-b6ee-43e9-8977-950cb0188887-workload-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:21:17.165670 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:17.159838    4364 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/134eb93c-b6ee-43e9-8977-950cb0188887-istiod-ca-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:21:17.165670 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:17.159852    4364 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/134eb93c-b6ee-43e9-8977-950cb0188887-credential-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:21:17.165670 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:17.159867    4364 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/134eb93c-b6ee-43e9-8977-950cb0188887-istio-token\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:21:17.165670 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:17.159880    4364 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/134eb93c-b6ee-43e9-8977-950cb0188887-istio-data\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:21:17.165670 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:17.159892    4364 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/134eb93c-b6ee-43e9-8977-950cb0188887-istio-envoy\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:21:17.165670 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:17.159904    4364 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/134eb93c-b6ee-43e9-8977-950cb0188887-workload-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:21:17.165670 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:17.159918    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txpk9\" (UniqueName: \"kubernetes.io/projected/134eb93c-b6ee-43e9-8977-950cb0188887-kube-api-access-txpk9\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:21:17.165670 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:17.159933    4364 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/134eb93c-b6ee-43e9-8977-950cb0188887-istio-podinfo\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:21:17.322618 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:17.311700    4364 generic.go:356] "Generic (PLEG): container finished" podID="134eb93c-b6ee-43e9-8977-950cb0188887" containerID="97cbae683a7c99fa36188a38ee1fb80a181bdd1f0bffcfdf8f0a4e5dd8940371" exitCode=0
Oct 30 13:21:17.322618 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:17.311858    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7" event={"ID":"134eb93c-b6ee-43e9-8977-950cb0188887","Type":"ContainerDied","Data":"97cbae683a7c99fa36188a38ee1fb80a181bdd1f0bffcfdf8f0a4e5dd8940371"}
Oct 30 13:21:17.322618 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:17.311907    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7" event={"ID":"134eb93c-b6ee-43e9-8977-950cb0188887","Type":"ContainerDied","Data":"aada12b6f818ef67b6a50fc035fda26b7614a1bd660367754cb1130c292c253f"}
Oct 30 13:21:17.322618 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:17.311935    4364 scope.go:122] "RemoveContainer" containerID="97cbae683a7c99fa36188a38ee1fb80a181bdd1f0bffcfdf8f0a4e5dd8940371"
Oct 30 13:21:17.322618 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:17.312095    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7"
Oct 30 13:21:17.325125 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:17.324883    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g"
Oct 30 13:21:17.326450 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:17.326160    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g" event={"ID":"370ee2a9-2643-4e24-81e4-ab56f3caefb3","Type":"ContainerDied","Data":"df1d24e41308066232260a33b84ffdea7ed5a6a30c9a1ea7dcb4f7f7cba210be"}
Oct 30 13:21:17.328211 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:17.328186    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5dd8cc-657cd56d9c-wps6v" event={"ID":"873a9c3f-d2a1-4eb8-a413-216e4d5cb48c","Type":"ContainerStarted","Data":"c4f20b8f9310ac26974a515521ebd80be28afe9a4fdaa0a42286494175ab5ac9"}
Oct 30 13:21:17.340111 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:17.340090    4364 scope.go:122] "RemoveContainer" containerID="8109b3a388ae1a87ab9161858a90b4118be61694758679b68e06332870a8a139"
Oct 30 13:21:17.361455 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:17.361434    4364 scope.go:122] "RemoveContainer" containerID="97cbae683a7c99fa36188a38ee1fb80a181bdd1f0bffcfdf8f0a4e5dd8940371"
Oct 30 13:21:17.361768 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:21:17.361740    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97cbae683a7c99fa36188a38ee1fb80a181bdd1f0bffcfdf8f0a4e5dd8940371\": container with ID starting with 97cbae683a7c99fa36188a38ee1fb80a181bdd1f0bffcfdf8f0a4e5dd8940371 not found: ID does not exist" containerID="97cbae683a7c99fa36188a38ee1fb80a181bdd1f0bffcfdf8f0a4e5dd8940371"
Oct 30 13:21:17.361829 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:17.361776    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97cbae683a7c99fa36188a38ee1fb80a181bdd1f0bffcfdf8f0a4e5dd8940371"} err="failed to get container status \"97cbae683a7c99fa36188a38ee1fb80a181bdd1f0bffcfdf8f0a4e5dd8940371\": rpc error: code = NotFound desc = could not find container \"97cbae683a7c99fa36188a38ee1fb80a181bdd1f0bffcfdf8f0a4e5dd8940371\": container with ID starting with 97cbae683a7c99fa36188a38ee1fb80a181bdd1f0bffcfdf8f0a4e5dd8940371 not found: ID does not exist"
Oct 30 13:21:17.361829 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:17.361792    4364 scope.go:122] "RemoveContainer" containerID="8109b3a388ae1a87ab9161858a90b4118be61694758679b68e06332870a8a139"
Oct 30 13:21:17.362072 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:21:17.362049    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8109b3a388ae1a87ab9161858a90b4118be61694758679b68e06332870a8a139\": container with ID starting with 8109b3a388ae1a87ab9161858a90b4118be61694758679b68e06332870a8a139 not found: ID does not exist" containerID="8109b3a388ae1a87ab9161858a90b4118be61694758679b68e06332870a8a139"
Oct 30 13:21:17.362183 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:17.362075    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8109b3a388ae1a87ab9161858a90b4118be61694758679b68e06332870a8a139"} err="failed to get container status \"8109b3a388ae1a87ab9161858a90b4118be61694758679b68e06332870a8a139\": rpc error: code = NotFound desc = could not find container \"8109b3a388ae1a87ab9161858a90b4118be61694758679b68e06332870a8a139\": container with ID starting with 8109b3a388ae1a87ab9161858a90b4118be61694758679b68e06332870a8a139 not found: ID does not exist"
Oct 30 13:21:17.362183 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:17.362092    4364 scope.go:122] "RemoveContainer" containerID="8d6e9592a05c1a4f1a42b58fc2bb8fadb0c66eef23ed01c3bbb6e7b3b0fb2317"
Oct 30 13:21:17.412046 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:17.412017    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7"]
Oct 30 13:21:17.421694 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:17.421633    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-5bbl7"]
Oct 30 13:21:17.455788 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:17.455765    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g"]
Oct 30 13:21:17.461846 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:17.461823    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-9sm9g"]
Oct 30 13:21:18.340507 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:18.340479    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5dd8cc-657cd56d9c-wps6v" event={"ID":"873a9c3f-d2a1-4eb8-a413-216e4d5cb48c","Type":"ContainerStarted","Data":"a67dfd83613ae6c7334c3475ab447b7ec84f0027de21b7a083084d4b0afe7a06"}
Oct 30 13:21:18.341061 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:18.340640    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5dd8cc-657cd56d9c-wps6v"
Oct 30 13:21:18.419206 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:18.419154    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5dd8cc-657cd56d9c-wps6v" podStartSLOduration=2.877086168 podStartE2EDuration="3.419139252s" totalImagesPullingTime="542.053084ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:21:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:21:18.414108723 +0000 UTC m=+1777.580473372" watchObservedRunningTime="2026-10-30 13:21:18.419139252 +0000 UTC m=+1777.585503899"
Oct 30 13:21:18.973479 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:18.973450    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="134eb93c-b6ee-43e9-8977-950cb0188887" path="/var/lib/kubelet/pods/134eb93c-b6ee-43e9-8977-950cb0188887/volumes"
Oct 30 13:21:18.973966 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:18.973950    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="370ee2a9-2643-4e24-81e4-ab56f3caefb3" path="/var/lib/kubelet/pods/370ee2a9-2643-4e24-81e4-ab56f3caefb3/volumes"
Oct 30 13:21:24.355756 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:24.355728    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5dd8cc-657cd56d9c-wps6v"
Oct 30 13:21:24.950192 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:24.950158    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5"]
Oct 30 13:21:24.950434 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:24.950394    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5" podUID="8ee5a4af-eb64-470e-a42e-7d334cedce2d" containerName="istio-proxy" containerID="cri-o://6eb1fee14b03910c6dca350f0c270f6ff32beab86fbd388efc10ab386fca21bd" gracePeriod=30
Oct 30 13:21:25.598262 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:25.598164    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-b5a369/test-model-b5a369-kserve-6bfb684495-gckcs"]
Oct 30 13:21:25.599229 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:25.599206    4364 cpu_manager.go:414] "containerMap: removing container" podUID="370ee2a9-2643-4e24-81e4-ab56f3caefb3" containerName="istio-proxy"
Oct 30 13:21:25.599229 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:25.599229    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="370ee2a9-2643-4e24-81e4-ab56f3caefb3" containerName="istio-proxy"
Oct 30 13:21:25.599391 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:25.599243    4364 cpu_manager.go:414] "containerMap: removing container" podUID="134eb93c-b6ee-43e9-8977-950cb0188887" containerName="istio-proxy"
Oct 30 13:21:25.599391 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:25.599250    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="134eb93c-b6ee-43e9-8977-950cb0188887" containerName="istio-proxy"
Oct 30 13:21:25.599391 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:25.599329    4364 cpu_manager.go:414] "containerMap: removing container" podUID="370ee2a9-2643-4e24-81e4-ab56f3caefb3" containerName="istio-proxy"
Oct 30 13:21:25.599391 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:25.599337    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="370ee2a9-2643-4e24-81e4-ab56f3caefb3" containerName="istio-proxy"
Oct 30 13:21:25.599644 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:25.599531    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="370ee2a9-2643-4e24-81e4-ab56f3caefb3" containerName="istio-proxy"
Oct 30 13:21:25.599644 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:25.599551    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="134eb93c-b6ee-43e9-8977-950cb0188887" containerName="istio-proxy"
Oct 30 13:21:25.599644 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:25.599569    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="134eb93c-b6ee-43e9-8977-950cb0188887" containerName="istio-proxy"
Oct 30 13:21:25.603465 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:25.603445    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b5a369/test-model-b5a369-kserve-6bfb684495-gckcs"
Oct 30 13:21:25.618137 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:25.618105    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d52aab51-7fdf-4be6-8893-b5fe88dbe341-tls-certs\") pod \"test-model-b5a369-kserve-6bfb684495-gckcs\" (UID: \"d52aab51-7fdf-4be6-8893-b5fe88dbe341\") " pod="ai-tenant-e2e-inf-a-b5a369/test-model-b5a369-kserve-6bfb684495-gckcs"
Oct 30 13:21:25.618367 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:25.618334    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d52aab51-7fdf-4be6-8893-b5fe88dbe341-model-cache\") pod \"test-model-b5a369-kserve-6bfb684495-gckcs\" (UID: \"d52aab51-7fdf-4be6-8893-b5fe88dbe341\") " pod="ai-tenant-e2e-inf-a-b5a369/test-model-b5a369-kserve-6bfb684495-gckcs"
Oct 30 13:21:25.618445 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:25.618393    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4w4g\" (UniqueName: \"kubernetes.io/projected/d52aab51-7fdf-4be6-8893-b5fe88dbe341-kube-api-access-f4w4g\") pod \"test-model-b5a369-kserve-6bfb684495-gckcs\" (UID: \"d52aab51-7fdf-4be6-8893-b5fe88dbe341\") " pod="ai-tenant-e2e-inf-a-b5a369/test-model-b5a369-kserve-6bfb684495-gckcs"
Oct 30 13:21:25.618510 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:25.618453    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d52aab51-7fdf-4be6-8893-b5fe88dbe341-dshm\") pod \"test-model-b5a369-kserve-6bfb684495-gckcs\" (UID: \"d52aab51-7fdf-4be6-8893-b5fe88dbe341\") " pod="ai-tenant-e2e-inf-a-b5a369/test-model-b5a369-kserve-6bfb684495-gckcs"
Oct 30 13:21:25.618510 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:25.618490    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d52aab51-7fdf-4be6-8893-b5fe88dbe341-home\") pod \"test-model-b5a369-kserve-6bfb684495-gckcs\" (UID: \"d52aab51-7fdf-4be6-8893-b5fe88dbe341\") " pod="ai-tenant-e2e-inf-a-b5a369/test-model-b5a369-kserve-6bfb684495-gckcs"
Oct 30 13:21:25.618666 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:25.618531    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d52aab51-7fdf-4be6-8893-b5fe88dbe341-tmp-dir\") pod \"test-model-b5a369-kserve-6bfb684495-gckcs\" (UID: \"d52aab51-7fdf-4be6-8893-b5fe88dbe341\") " pod="ai-tenant-e2e-inf-a-b5a369/test-model-b5a369-kserve-6bfb684495-gckcs"
Oct 30 13:21:25.652084 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:25.650958    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-b5a369\"/\"openshift-service-ca.crt\""
Oct 30 13:21:25.652084 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:25.651171    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-b5a369\"/\"kube-root-ca.crt\""
Oct 30 13:21:25.652482 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:25.652465    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-b5a369/test-model-b5a369-kserve-6bfb684495-gckcs"]
Oct 30 13:21:25.657732 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:25.657708    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-b5a369\"/\"test-model-b5a369-kserve-self-signed-certs\""
Oct 30 13:21:25.657996 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:25.657976    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-b5a369\"/\"default-dockercfg-qngv7\""
Oct 30 13:21:25.719655 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:25.719586    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d52aab51-7fdf-4be6-8893-b5fe88dbe341-tls-certs\") pod \"test-model-b5a369-kserve-6bfb684495-gckcs\" (UID: \"d52aab51-7fdf-4be6-8893-b5fe88dbe341\") " pod="ai-tenant-e2e-inf-a-b5a369/test-model-b5a369-kserve-6bfb684495-gckcs"
Oct 30 13:21:25.719849 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:25.719831    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d52aab51-7fdf-4be6-8893-b5fe88dbe341-model-cache\") pod \"test-model-b5a369-kserve-6bfb684495-gckcs\" (UID: \"d52aab51-7fdf-4be6-8893-b5fe88dbe341\") " pod="ai-tenant-e2e-inf-a-b5a369/test-model-b5a369-kserve-6bfb684495-gckcs"
Oct 30 13:21:25.719937 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:25.719893    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4w4g\" (UniqueName: \"kubernetes.io/projected/d52aab51-7fdf-4be6-8893-b5fe88dbe341-kube-api-access-f4w4g\") pod \"test-model-b5a369-kserve-6bfb684495-gckcs\" (UID: \"d52aab51-7fdf-4be6-8893-b5fe88dbe341\") " pod="ai-tenant-e2e-inf-a-b5a369/test-model-b5a369-kserve-6bfb684495-gckcs"
Oct 30 13:21:25.720001 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:25.719945    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d52aab51-7fdf-4be6-8893-b5fe88dbe341-dshm\") pod \"test-model-b5a369-kserve-6bfb684495-gckcs\" (UID: \"d52aab51-7fdf-4be6-8893-b5fe88dbe341\") " pod="ai-tenant-e2e-inf-a-b5a369/test-model-b5a369-kserve-6bfb684495-gckcs"
Oct 30 13:21:25.720001 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:25.719992    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d52aab51-7fdf-4be6-8893-b5fe88dbe341-home\") pod \"test-model-b5a369-kserve-6bfb684495-gckcs\" (UID: \"d52aab51-7fdf-4be6-8893-b5fe88dbe341\") " pod="ai-tenant-e2e-inf-a-b5a369/test-model-b5a369-kserve-6bfb684495-gckcs"
Oct 30 13:21:25.720111 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:25.720044    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d52aab51-7fdf-4be6-8893-b5fe88dbe341-tmp-dir\") pod \"test-model-b5a369-kserve-6bfb684495-gckcs\" (UID: \"d52aab51-7fdf-4be6-8893-b5fe88dbe341\") " pod="ai-tenant-e2e-inf-a-b5a369/test-model-b5a369-kserve-6bfb684495-gckcs"
Oct 30 13:21:25.720539 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:25.720375    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d52aab51-7fdf-4be6-8893-b5fe88dbe341-model-cache\") pod \"test-model-b5a369-kserve-6bfb684495-gckcs\" (UID: \"d52aab51-7fdf-4be6-8893-b5fe88dbe341\") " pod="ai-tenant-e2e-inf-a-b5a369/test-model-b5a369-kserve-6bfb684495-gckcs"
Oct 30 13:21:25.720539 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:25.720404    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d52aab51-7fdf-4be6-8893-b5fe88dbe341-home\") pod \"test-model-b5a369-kserve-6bfb684495-gckcs\" (UID: \"d52aab51-7fdf-4be6-8893-b5fe88dbe341\") " pod="ai-tenant-e2e-inf-a-b5a369/test-model-b5a369-kserve-6bfb684495-gckcs"
Oct 30 13:21:25.720539 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:25.720497    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d52aab51-7fdf-4be6-8893-b5fe88dbe341-tmp-dir\") pod \"test-model-b5a369-kserve-6bfb684495-gckcs\" (UID: \"d52aab51-7fdf-4be6-8893-b5fe88dbe341\") " pod="ai-tenant-e2e-inf-a-b5a369/test-model-b5a369-kserve-6bfb684495-gckcs"
Oct 30 13:21:25.722320 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:25.722300    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d52aab51-7fdf-4be6-8893-b5fe88dbe341-dshm\") pod \"test-model-b5a369-kserve-6bfb684495-gckcs\" (UID: \"d52aab51-7fdf-4be6-8893-b5fe88dbe341\") " pod="ai-tenant-e2e-inf-a-b5a369/test-model-b5a369-kserve-6bfb684495-gckcs"
Oct 30 13:21:25.722717 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:25.722698    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d52aab51-7fdf-4be6-8893-b5fe88dbe341-tls-certs\") pod \"test-model-b5a369-kserve-6bfb684495-gckcs\" (UID: \"d52aab51-7fdf-4be6-8893-b5fe88dbe341\") " pod="ai-tenant-e2e-inf-a-b5a369/test-model-b5a369-kserve-6bfb684495-gckcs"
Oct 30 13:21:25.756117 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:25.756005    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4w4g\" (UniqueName: \"kubernetes.io/projected/d52aab51-7fdf-4be6-8893-b5fe88dbe341-kube-api-access-f4w4g\") pod \"test-model-b5a369-kserve-6bfb684495-gckcs\" (UID: \"d52aab51-7fdf-4be6-8893-b5fe88dbe341\") " pod="ai-tenant-e2e-inf-a-b5a369/test-model-b5a369-kserve-6bfb684495-gckcs"
Oct 30 13:21:25.922870 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:25.922779    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b5a369/test-model-b5a369-kserve-6bfb684495-gckcs"
Oct 30 13:21:26.421027 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:26.420988    4364 generic.go:356] "Generic (PLEG): container finished" podID="ebad4af8-38a2-4da7-a1ec-b56ef4c48a79" containerID="b4e8057dbfc71e745c1ec7b36ba257b389a44a362756b1272c6f5253e3d046f1" exitCode=0
Oct 30 13:21:26.421320 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:26.421298    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn" event={"ID":"ebad4af8-38a2-4da7-a1ec-b56ef4c48a79","Type":"ContainerDied","Data":"b4e8057dbfc71e745c1ec7b36ba257b389a44a362756b1272c6f5253e3d046f1"}
Oct 30 13:21:26.421927 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:26.421904    4364 scope.go:122] "RemoveContainer" containerID="b4e8057dbfc71e745c1ec7b36ba257b389a44a362756b1272c6f5253e3d046f1"
Oct 30 13:21:26.424454 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:26.424417    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:21:26.424566 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:26.424513    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:21:26.424638 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:26.424562    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:21:26.446224 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:26.446177    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-b5a369/test-model-b5a369-kserve-6bfb684495-gckcs"]
Oct 30 13:21:26.451002 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:21:26.450963    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd52aab51_7fdf_4be6_8893_b5fe88dbe341.slice/crio-3942eae0bcda38852a2326dc5be15070eb404d2c68786b6681ad046d05b1a979 WatchSource:0}: Error finding container 3942eae0bcda38852a2326dc5be15070eb404d2c68786b6681ad046d05b1a979: Status 404 returned error can't find the container with id 3942eae0bcda38852a2326dc5be15070eb404d2c68786b6681ad046d05b1a979
Oct 30 13:21:27.431802 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:27.431761    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b5a369/test-model-b5a369-kserve-6bfb684495-gckcs" event={"ID":"d52aab51-7fdf-4be6-8893-b5fe88dbe341","Type":"ContainerStarted","Data":"fc806d441740c27c7d1faa15cca81d6050ee11934ff3d6733efdeccb24cadeef"}
Oct 30 13:21:27.432267 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:27.431814    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b5a369/test-model-b5a369-kserve-6bfb684495-gckcs" event={"ID":"d52aab51-7fdf-4be6-8893-b5fe88dbe341","Type":"ContainerStarted","Data":"3942eae0bcda38852a2326dc5be15070eb404d2c68786b6681ad046d05b1a979"}
Oct 30 13:21:27.434022 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:27.433995    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn" event={"ID":"ebad4af8-38a2-4da7-a1ec-b56ef4c48a79","Type":"ContainerStarted","Data":"213ce31fde9bfe9810183291dcd689c66acc715d9865f4382393cea5102b9f28"}
Oct 30 13:21:27.475991 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:27.475937    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-b5a369/test-model-b5a369-kserve-6bfb684495-gckcs" podStartSLOduration=2.475918214 podStartE2EDuration="2.475918214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:21:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:21:27.472161668 +0000 UTC m=+1786.638526317" watchObservedRunningTime="2026-10-30 13:21:27.475918214 +0000 UTC m=+1786.642282863"
Oct 30 13:21:27.699668 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:27.693950    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn"
Oct 30 13:21:27.699668 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:27.697662    4364 patch_prober.go:29] interesting pod/e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Oct 30 13:21:27.699668 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:27.697719    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn" podUID="ebad4af8-38a2-4da7-a1ec-b56ef4c48a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Oct 30 13:21:27.923546 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:27.923515    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-b5a369/test-model-b5a369-kserve-6bfb684495-gckcs"
Oct 30 13:21:28.703983 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:28.703697    4364 patch_prober.go:29] interesting pod/e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Oct 30 13:21:28.703983 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:28.703775    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn" podUID="ebad4af8-38a2-4da7-a1ec-b56ef4c48a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Oct 30 13:21:29.700631 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:29.699984    4364 patch_prober.go:29] interesting pod/e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Oct 30 13:21:29.700631 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:29.700044    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn" podUID="ebad4af8-38a2-4da7-a1ec-b56ef4c48a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Oct 30 13:21:30.314088 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:30.314063    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5"
Oct 30 13:21:30.347240 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:30.347211    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ee5a4af-eb64-470e-a42e-7d334cedce2d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ee5a4af-eb64-470e-a42e-7d334cedce2d-workload-certs\") pod \"8ee5a4af-eb64-470e-a42e-7d334cedce2d\" (UID: \"8ee5a4af-eb64-470e-a42e-7d334cedce2d\") "
Oct 30 13:21:30.347392 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:30.347301    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ee5a4af-eb64-470e-a42e-7d334cedce2d-istio-token\" (UniqueName: \"kubernetes.io/projected/8ee5a4af-eb64-470e-a42e-7d334cedce2d-istio-token\") pod \"8ee5a4af-eb64-470e-a42e-7d334cedce2d\" (UID: \"8ee5a4af-eb64-470e-a42e-7d334cedce2d\") "
Oct 30 13:21:30.347392 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:30.347382    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ee5a4af-eb64-470e-a42e-7d334cedce2d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ee5a4af-eb64-470e-a42e-7d334cedce2d-istiod-ca-cert\") pod \"8ee5a4af-eb64-470e-a42e-7d334cedce2d\" (UID: \"8ee5a4af-eb64-470e-a42e-7d334cedce2d\") "
Oct 30 13:21:30.347509 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:30.347403    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ee5a4af-eb64-470e-a42e-7d334cedce2d-kube-api-access-5xz6r\" (UniqueName: \"kubernetes.io/projected/8ee5a4af-eb64-470e-a42e-7d334cedce2d-kube-api-access-5xz6r\") pod \"8ee5a4af-eb64-470e-a42e-7d334cedce2d\" (UID: \"8ee5a4af-eb64-470e-a42e-7d334cedce2d\") "
Oct 30 13:21:30.347509 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:30.347440    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ee5a4af-eb64-470e-a42e-7d334cedce2d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ee5a4af-eb64-470e-a42e-7d334cedce2d-istio-envoy\") pod \"8ee5a4af-eb64-470e-a42e-7d334cedce2d\" (UID: \"8ee5a4af-eb64-470e-a42e-7d334cedce2d\") "
Oct 30 13:21:30.347509 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:30.347445    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ee5a4af-eb64-470e-a42e-7d334cedce2d-workload-certs" pod "8ee5a4af-eb64-470e-a42e-7d334cedce2d" (UID: "8ee5a4af-eb64-470e-a42e-7d334cedce2d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:21:30.347509 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:30.347476    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ee5a4af-eb64-470e-a42e-7d334cedce2d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ee5a4af-eb64-470e-a42e-7d334cedce2d-credential-socket\") pod \"8ee5a4af-eb64-470e-a42e-7d334cedce2d\" (UID: \"8ee5a4af-eb64-470e-a42e-7d334cedce2d\") "
Oct 30 13:21:30.347744 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:30.347609    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ee5a4af-eb64-470e-a42e-7d334cedce2d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ee5a4af-eb64-470e-a42e-7d334cedce2d-istio-data\") pod \"8ee5a4af-eb64-470e-a42e-7d334cedce2d\" (UID: \"8ee5a4af-eb64-470e-a42e-7d334cedce2d\") "
Oct 30 13:21:30.347805 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:30.347748    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8ee5a4af-eb64-470e-a42e-7d334cedce2d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ee5a4af-eb64-470e-a42e-7d334cedce2d-istio-podinfo\") pod \"8ee5a4af-eb64-470e-a42e-7d334cedce2d\" (UID: \"8ee5a4af-eb64-470e-a42e-7d334cedce2d\") "
Oct 30 13:21:30.347805 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:30.347786    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ee5a4af-eb64-470e-a42e-7d334cedce2d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ee5a4af-eb64-470e-a42e-7d334cedce2d-workload-socket\") pod \"8ee5a4af-eb64-470e-a42e-7d334cedce2d\" (UID: \"8ee5a4af-eb64-470e-a42e-7d334cedce2d\") "
Oct 30 13:21:30.347962 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:30.347923    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ee5a4af-eb64-470e-a42e-7d334cedce2d-istio-data" pod "8ee5a4af-eb64-470e-a42e-7d334cedce2d" (UID: "8ee5a4af-eb64-470e-a42e-7d334cedce2d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:21:30.348024 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:30.347977    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ee5a4af-eb64-470e-a42e-7d334cedce2d-istiod-ca-cert" pod "8ee5a4af-eb64-470e-a42e-7d334cedce2d" (UID: "8ee5a4af-eb64-470e-a42e-7d334cedce2d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:21:30.348111 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:30.347806    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ee5a4af-eb64-470e-a42e-7d334cedce2d-credential-socket" pod "8ee5a4af-eb64-470e-a42e-7d334cedce2d" (UID: "8ee5a4af-eb64-470e-a42e-7d334cedce2d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:21:30.348321 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:30.348205    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ee5a4af-eb64-470e-a42e-7d334cedce2d-workload-socket" pod "8ee5a4af-eb64-470e-a42e-7d334cedce2d" (UID: "8ee5a4af-eb64-470e-a42e-7d334cedce2d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:21:30.348644 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:30.348623    4364 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ee5a4af-eb64-470e-a42e-7d334cedce2d-workload-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:21:30.348731 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:30.348649    4364 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ee5a4af-eb64-470e-a42e-7d334cedce2d-workload-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:21:30.348731 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:30.348662    4364 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ee5a4af-eb64-470e-a42e-7d334cedce2d-istiod-ca-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:21:30.348731 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:30.348675    4364 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ee5a4af-eb64-470e-a42e-7d334cedce2d-credential-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:21:30.348731 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:30.348691    4364 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ee5a4af-eb64-470e-a42e-7d334cedce2d-istio-data\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:21:30.349692 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:30.349672    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ee5a4af-eb64-470e-a42e-7d334cedce2d-istio-envoy" pod "8ee5a4af-eb64-470e-a42e-7d334cedce2d" (UID: "8ee5a4af-eb64-470e-a42e-7d334cedce2d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:21:30.349850 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:30.349836    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ee5a4af-eb64-470e-a42e-7d334cedce2d-istio-token" pod "8ee5a4af-eb64-470e-a42e-7d334cedce2d" (UID: "8ee5a4af-eb64-470e-a42e-7d334cedce2d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:21:30.350012 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:30.349997    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ee5a4af-eb64-470e-a42e-7d334cedce2d-kube-api-access-5xz6r" pod "8ee5a4af-eb64-470e-a42e-7d334cedce2d" (UID: "8ee5a4af-eb64-470e-a42e-7d334cedce2d"). InnerVolumeSpecName "kube-api-access-5xz6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:21:30.350094 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:30.350080    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ee5a4af-eb64-470e-a42e-7d334cedce2d-istio-podinfo" pod "8ee5a4af-eb64-470e-a42e-7d334cedce2d" (UID: "8ee5a4af-eb64-470e-a42e-7d334cedce2d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:21:30.450563 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:30.450527    4364 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ee5a4af-eb64-470e-a42e-7d334cedce2d-istio-envoy\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:21:30.450563 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:30.450564    4364 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ee5a4af-eb64-470e-a42e-7d334cedce2d-istio-podinfo\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:21:30.450835 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:30.450578    4364 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ee5a4af-eb64-470e-a42e-7d334cedce2d-istio-token\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:21:30.450835 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:30.450592    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xz6r\" (UniqueName: \"kubernetes.io/projected/8ee5a4af-eb64-470e-a42e-7d334cedce2d-kube-api-access-5xz6r\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:21:30.464806 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:30.464776    4364 generic.go:356] "Generic (PLEG): container finished" podID="8ee5a4af-eb64-470e-a42e-7d334cedce2d" containerID="6eb1fee14b03910c6dca350f0c270f6ff32beab86fbd388efc10ab386fca21bd" exitCode=0
Oct 30 13:21:30.464961 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:30.464846    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5"
Oct 30 13:21:30.464961 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:30.464861    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5" event={"ID":"8ee5a4af-eb64-470e-a42e-7d334cedce2d","Type":"ContainerDied","Data":"6eb1fee14b03910c6dca350f0c270f6ff32beab86fbd388efc10ab386fca21bd"}
Oct 30 13:21:30.464961 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:30.464931    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5" event={"ID":"8ee5a4af-eb64-470e-a42e-7d334cedce2d","Type":"ContainerDied","Data":"6b0dab6b207a353230cc0c964fa2ace1ac1526bbd0208e268381f1484345b43f"}
Oct 30 13:21:30.464961 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:30.464955    4364 scope.go:122] "RemoveContainer" containerID="6eb1fee14b03910c6dca350f0c270f6ff32beab86fbd388efc10ab386fca21bd"
Oct 30 13:21:30.519899 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:30.501864    4364 scope.go:122] "RemoveContainer" containerID="6eb1fee14b03910c6dca350f0c270f6ff32beab86fbd388efc10ab386fca21bd"
Oct 30 13:21:30.520773 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:21:30.520731    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6eb1fee14b03910c6dca350f0c270f6ff32beab86fbd388efc10ab386fca21bd\": container with ID starting with 6eb1fee14b03910c6dca350f0c270f6ff32beab86fbd388efc10ab386fca21bd not found: ID does not exist" containerID="6eb1fee14b03910c6dca350f0c270f6ff32beab86fbd388efc10ab386fca21bd"
Oct 30 13:21:30.520903 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:30.520776    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6eb1fee14b03910c6dca350f0c270f6ff32beab86fbd388efc10ab386fca21bd"} err="failed to get container status \"6eb1fee14b03910c6dca350f0c270f6ff32beab86fbd388efc10ab386fca21bd\": rpc error: code = NotFound desc = could not find container \"6eb1fee14b03910c6dca350f0c270f6ff32beab86fbd388efc10ab386fca21bd\": container with ID starting with 6eb1fee14b03910c6dca350f0c270f6ff32beab86fbd388efc10ab386fca21bd not found: ID does not exist"
Oct 30 13:21:30.546210 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:30.546175    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5"]
Oct 30 13:21:30.550822 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:30.550798    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-4ngq5"]
Oct 30 13:21:30.699624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:30.691936    4364 patch_prober.go:29] interesting pod/e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Oct 30 13:21:30.699624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:30.691998    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn" podUID="ebad4af8-38a2-4da7-a1ec-b56ef4c48a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Oct 30 13:21:30.933454 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:30.933422    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn"
Oct 30 13:21:30.972721 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:30.972683    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ee5a4af-eb64-470e-a42e-7d334cedce2d" path="/var/lib/kubelet/pods/8ee5a4af-eb64-470e-a42e-7d334cedce2d/volumes"
Oct 30 13:21:31.703648 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:31.702705    4364 patch_prober.go:29] interesting pod/e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Oct 30 13:21:31.703648 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:31.702765    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn" podUID="ebad4af8-38a2-4da7-a1ec-b56ef4c48a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Oct 30 13:21:32.688096 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:32.688059    4364 patch_prober.go:29] interesting pod/e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Oct 30 13:21:32.688270 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:32.688115    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn" podUID="ebad4af8-38a2-4da7-a1ec-b56ef4c48a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Oct 30 13:21:33.693108 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:33.693081    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn"
Oct 30 13:21:33.694171 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:33.694149    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn"
Oct 30 13:21:35.923676 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:35.923646    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-b5a369/test-model-b5a369-kserve-6bfb684495-gckcs"
Oct 30 13:21:35.936966 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:35.936932    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-b5a369/test-model-b5a369-kserve-6bfb684495-gckcs"
Oct 30 13:21:36.384819 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:36.384782    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp"]
Oct 30 13:21:36.385904 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:36.385878    4364 cpu_manager.go:414] "containerMap: removing container" podUID="8ee5a4af-eb64-470e-a42e-7d334cedce2d" containerName="istio-proxy"
Oct 30 13:21:36.385904 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:36.385904    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ee5a4af-eb64-470e-a42e-7d334cedce2d" containerName="istio-proxy"
Oct 30 13:21:36.386078 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:36.385997    4364 cpu_manager.go:414] "containerMap: removing container" podUID="134eb93c-b6ee-43e9-8977-950cb0188887" containerName="istio-proxy"
Oct 30 13:21:36.386078 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:36.386007    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="134eb93c-b6ee-43e9-8977-950cb0188887" containerName="istio-proxy"
Oct 30 13:21:36.386220 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:36.386206    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="370ee2a9-2643-4e24-81e4-ab56f3caefb3" containerName="istio-proxy"
Oct 30 13:21:36.386283 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:36.386226    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ee5a4af-eb64-470e-a42e-7d334cedce2d" containerName="istio-proxy"
Oct 30 13:21:36.390357 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:36.390338    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp"
Oct 30 13:21:36.409089 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:36.409070    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-77254d-openshift-default-dockercfg-4wm72\""
Oct 30 13:21:36.429498 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:36.429468    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp"]
Oct 30 13:21:36.565816 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:36.565786    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-b5a369/test-model-b5a369-kserve-6bfb684495-gckcs"
Oct 30 13:21:36.581266 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:36.581232    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9ee49f0-1a2e-4b0b-8046-36b55a049fad-workload-certs\") pod \"e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp\" (UID: \"a9ee49f0-1a2e-4b0b-8046-36b55a049fad\") " pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp"
Oct 30 13:21:36.581449 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:36.581403    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9ee49f0-1a2e-4b0b-8046-36b55a049fad-workload-socket\") pod \"e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp\" (UID: \"a9ee49f0-1a2e-4b0b-8046-36b55a049fad\") " pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp"
Oct 30 13:21:36.581542 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:36.581524    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9ee49f0-1a2e-4b0b-8046-36b55a049fad-istio-podinfo\") pod \"e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp\" (UID: \"a9ee49f0-1a2e-4b0b-8046-36b55a049fad\") " pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp"
Oct 30 13:21:36.581665 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:36.581649    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9ee49f0-1a2e-4b0b-8046-36b55a049fad-istio-envoy\") pod \"e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp\" (UID: \"a9ee49f0-1a2e-4b0b-8046-36b55a049fad\") " pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp"
Oct 30 13:21:36.581727 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:36.581692    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9ee49f0-1a2e-4b0b-8046-36b55a049fad-istio-token\") pod \"e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp\" (UID: \"a9ee49f0-1a2e-4b0b-8046-36b55a049fad\") " pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp"
Oct 30 13:21:36.581789 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:36.581729    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9ee49f0-1a2e-4b0b-8046-36b55a049fad-credential-socket\") pod \"e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp\" (UID: \"a9ee49f0-1a2e-4b0b-8046-36b55a049fad\") " pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp"
Oct 30 13:21:36.581835 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:36.581810    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9ee49f0-1a2e-4b0b-8046-36b55a049fad-istio-data\") pod \"e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp\" (UID: \"a9ee49f0-1a2e-4b0b-8046-36b55a049fad\") " pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp"
Oct 30 13:21:36.581882 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:36.581837    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9ee49f0-1a2e-4b0b-8046-36b55a049fad-istiod-ca-cert\") pod \"e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp\" (UID: \"a9ee49f0-1a2e-4b0b-8046-36b55a049fad\") " pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp"
Oct 30 13:21:36.581882 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:36.581867    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmbsc\" (UniqueName: \"kubernetes.io/projected/a9ee49f0-1a2e-4b0b-8046-36b55a049fad-kube-api-access-qmbsc\") pod \"e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp\" (UID: \"a9ee49f0-1a2e-4b0b-8046-36b55a049fad\") " pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp"
Oct 30 13:21:36.603730 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:36.603706    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-8d82d5-65c88b865b-xt6lk"]
Oct 30 13:21:36.609853 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:36.609836    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-8d82d5-65c88b865b-xt6lk"
Oct 30 13:21:36.631630 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:36.626300    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-8d82d5-serving-cert\""
Oct 30 13:21:36.631874 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:36.631844    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-8d82d5-65c88b865b-xt6lk"]
Oct 30 13:21:36.683396 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:36.683322    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9ee49f0-1a2e-4b0b-8046-36b55a049fad-istio-podinfo\") pod \"e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp\" (UID: \"a9ee49f0-1a2e-4b0b-8046-36b55a049fad\") " pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp"
Oct 30 13:21:36.683563 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:36.683432    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9ee49f0-1a2e-4b0b-8046-36b55a049fad-istio-envoy\") pod \"e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp\" (UID: \"a9ee49f0-1a2e-4b0b-8046-36b55a049fad\") " pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp"
Oct 30 13:21:36.683563 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:36.683478    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9ee49f0-1a2e-4b0b-8046-36b55a049fad-istio-token\") pod \"e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp\" (UID: \"a9ee49f0-1a2e-4b0b-8046-36b55a049fad\") " pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp"
Oct 30 13:21:36.683563 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:36.683516    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9ee49f0-1a2e-4b0b-8046-36b55a049fad-credential-socket\") pod \"e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp\" (UID: \"a9ee49f0-1a2e-4b0b-8046-36b55a049fad\") " pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp"
Oct 30 13:21:36.683751 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:36.683571    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9ee49f0-1a2e-4b0b-8046-36b55a049fad-istio-data\") pod \"e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp\" (UID: \"a9ee49f0-1a2e-4b0b-8046-36b55a049fad\") " pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp"
Oct 30 13:21:36.683751 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:36.683612    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9ee49f0-1a2e-4b0b-8046-36b55a049fad-istiod-ca-cert\") pod \"e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp\" (UID: \"a9ee49f0-1a2e-4b0b-8046-36b55a049fad\") " pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp"
Oct 30 13:21:36.683751 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:36.683657    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmbsc\" (UniqueName: \"kubernetes.io/projected/a9ee49f0-1a2e-4b0b-8046-36b55a049fad-kube-api-access-qmbsc\") pod \"e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp\" (UID: \"a9ee49f0-1a2e-4b0b-8046-36b55a049fad\") " pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp"
Oct 30 13:21:36.683909 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:36.683783    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9ee49f0-1a2e-4b0b-8046-36b55a049fad-workload-certs\") pod \"e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp\" (UID: \"a9ee49f0-1a2e-4b0b-8046-36b55a049fad\") " pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp"
Oct 30 13:21:36.685330 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:36.683982    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9ee49f0-1a2e-4b0b-8046-36b55a049fad-workload-socket\") pod \"e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp\" (UID: \"a9ee49f0-1a2e-4b0b-8046-36b55a049fad\") " pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp"
Oct 30 13:21:36.685330 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:36.684313    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9ee49f0-1a2e-4b0b-8046-36b55a049fad-workload-certs\") pod \"e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp\" (UID: \"a9ee49f0-1a2e-4b0b-8046-36b55a049fad\") " pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp"
Oct 30 13:21:36.685330 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:36.684454    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9ee49f0-1a2e-4b0b-8046-36b55a049fad-istio-data\") pod \"e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp\" (UID: \"a9ee49f0-1a2e-4b0b-8046-36b55a049fad\") " pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp"
Oct 30 13:21:36.685330 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:36.684582    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9ee49f0-1a2e-4b0b-8046-36b55a049fad-credential-socket\") pod \"e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp\" (UID: \"a9ee49f0-1a2e-4b0b-8046-36b55a049fad\") " pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp"
Oct 30 13:21:36.685330 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:36.685278    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9ee49f0-1a2e-4b0b-8046-36b55a049fad-istiod-ca-cert\") pod \"e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp\" (UID: \"a9ee49f0-1a2e-4b0b-8046-36b55a049fad\") " pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp"
Oct 30 13:21:36.685330 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:36.685305    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9ee49f0-1a2e-4b0b-8046-36b55a049fad-workload-socket\") pod \"e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp\" (UID: \"a9ee49f0-1a2e-4b0b-8046-36b55a049fad\") " pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp"
Oct 30 13:21:36.686564 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:36.686544    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9ee49f0-1a2e-4b0b-8046-36b55a049fad-istio-podinfo\") pod \"e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp\" (UID: \"a9ee49f0-1a2e-4b0b-8046-36b55a049fad\") " pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp"
Oct 30 13:21:36.686729 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:36.686714    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9ee49f0-1a2e-4b0b-8046-36b55a049fad-istio-envoy\") pod \"e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp\" (UID: \"a9ee49f0-1a2e-4b0b-8046-36b55a049fad\") " pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp"
Oct 30 13:21:36.708896 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:36.708867    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9ee49f0-1a2e-4b0b-8046-36b55a049fad-istio-token\") pod \"e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp\" (UID: \"a9ee49f0-1a2e-4b0b-8046-36b55a049fad\") " pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp"
Oct 30 13:21:36.709057 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:36.709040    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmbsc\" (UniqueName: \"kubernetes.io/projected/a9ee49f0-1a2e-4b0b-8046-36b55a049fad-kube-api-access-qmbsc\") pod \"e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp\" (UID: \"a9ee49f0-1a2e-4b0b-8046-36b55a049fad\") " pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp"
Oct 30 13:21:36.785453 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:36.785421    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4813393d-187f-4a3a-a77b-6151b211d2f8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-8d82d5-65c88b865b-xt6lk\" (UID: \"4813393d-187f-4a3a-a77b-6151b211d2f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-8d82d5-65c88b865b-xt6lk"
Oct 30 13:21:36.785652 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:36.785581    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4813393d-187f-4a3a-a77b-6151b211d2f8-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-8d82d5-65c88b865b-xt6lk\" (UID: \"4813393d-187f-4a3a-a77b-6151b211d2f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-8d82d5-65c88b865b-xt6lk"
Oct 30 13:21:36.785839 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:36.785820    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4813393d-187f-4a3a-a77b-6151b211d2f8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-8d82d5-65c88b865b-xt6lk\" (UID: \"4813393d-187f-4a3a-a77b-6151b211d2f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-8d82d5-65c88b865b-xt6lk"
Oct 30 13:21:36.785884 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:36.785877    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgcxl\" (UniqueName: \"kubernetes.io/projected/4813393d-187f-4a3a-a77b-6151b211d2f8-kube-api-access-tgcxl\") pod \"maas-api-e2e-shared-b-w1-8d82d5-65c88b865b-xt6lk\" (UID: \"4813393d-187f-4a3a-a77b-6151b211d2f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-8d82d5-65c88b865b-xt6lk"
Oct 30 13:21:36.896682 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:36.895296    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4813393d-187f-4a3a-a77b-6151b211d2f8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-8d82d5-65c88b865b-xt6lk\" (UID: \"4813393d-187f-4a3a-a77b-6151b211d2f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-8d82d5-65c88b865b-xt6lk"
Oct 30 13:21:36.896682 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:36.895441    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgcxl\" (UniqueName: \"kubernetes.io/projected/4813393d-187f-4a3a-a77b-6151b211d2f8-kube-api-access-tgcxl\") pod \"maas-api-e2e-shared-b-w1-8d82d5-65c88b865b-xt6lk\" (UID: \"4813393d-187f-4a3a-a77b-6151b211d2f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-8d82d5-65c88b865b-xt6lk"
Oct 30 13:21:36.896682 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:36.895511    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4813393d-187f-4a3a-a77b-6151b211d2f8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-8d82d5-65c88b865b-xt6lk\" (UID: \"4813393d-187f-4a3a-a77b-6151b211d2f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-8d82d5-65c88b865b-xt6lk"
Oct 30 13:21:36.896682 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:36.895684    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4813393d-187f-4a3a-a77b-6151b211d2f8-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-8d82d5-65c88b865b-xt6lk\" (UID: \"4813393d-187f-4a3a-a77b-6151b211d2f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-8d82d5-65c88b865b-xt6lk"
Oct 30 13:21:36.899630 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:36.897331    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4813393d-187f-4a3a-a77b-6151b211d2f8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-8d82d5-65c88b865b-xt6lk\" (UID: \"4813393d-187f-4a3a-a77b-6151b211d2f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-8d82d5-65c88b865b-xt6lk"
Oct 30 13:21:36.932582 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:36.932546    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4813393d-187f-4a3a-a77b-6151b211d2f8-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-8d82d5-65c88b865b-xt6lk\" (UID: \"4813393d-187f-4a3a-a77b-6151b211d2f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-8d82d5-65c88b865b-xt6lk"
Oct 30 13:21:36.935898 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:36.935740    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4813393d-187f-4a3a-a77b-6151b211d2f8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-8d82d5-65c88b865b-xt6lk\" (UID: \"4813393d-187f-4a3a-a77b-6151b211d2f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-8d82d5-65c88b865b-xt6lk"
Oct 30 13:21:36.936521 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:36.936500    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgcxl\" (UniqueName: \"kubernetes.io/projected/4813393d-187f-4a3a-a77b-6151b211d2f8-kube-api-access-tgcxl\") pod \"maas-api-e2e-shared-b-w1-8d82d5-65c88b865b-xt6lk\" (UID: \"4813393d-187f-4a3a-a77b-6151b211d2f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-8d82d5-65c88b865b-xt6lk"
Oct 30 13:21:37.005021 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:37.004994    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp"
Oct 30 13:21:37.225700 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:37.225591    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-8d82d5-65c88b865b-xt6lk"
Oct 30 13:21:37.335416 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:37.335386    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp"]
Oct 30 13:21:37.336904 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:21:37.336873    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9ee49f0_1a2e_4b0b_8046_36b55a049fad.slice/crio-d5d0ca4c46e04b7440e8f88ac3cd9efc18ddb72ca56da78d560a8f882aacefb2 WatchSource:0}: Error finding container d5d0ca4c46e04b7440e8f88ac3cd9efc18ddb72ca56da78d560a8f882aacefb2: Status 404 returned error can't find the container with id d5d0ca4c46e04b7440e8f88ac3cd9efc18ddb72ca56da78d560a8f882aacefb2
Oct 30 13:21:37.341477 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:37.341429    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:21:37.341593 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:37.341528    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:21:37.341593 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:37.341579    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:21:37.569314 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:37.567872    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-8d82d5-65c88b865b-xt6lk"]
Oct 30 13:21:37.570773 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:21:37.570735    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4813393d_187f_4a3a_a77b_6151b211d2f8.slice/crio-70a5480174238e72e167548405e15adbb23486f4980c61ac5b8a7f106a0c1d4e WatchSource:0}: Error finding container 70a5480174238e72e167548405e15adbb23486f4980c61ac5b8a7f106a0c1d4e: Status 404 returned error can't find the container with id 70a5480174238e72e167548405e15adbb23486f4980c61ac5b8a7f106a0c1d4e
Oct 30 13:21:37.581380 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:37.581346    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp" event={"ID":"a9ee49f0-1a2e-4b0b-8046-36b55a049fad","Type":"ContainerStarted","Data":"892f73e4ec65ed02bf1f273204b3c5baeb79940b7ac2e3cdfb03d8d30f3abe2d"}
Oct 30 13:21:37.581498 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:37.581403    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp" event={"ID":"a9ee49f0-1a2e-4b0b-8046-36b55a049fad","Type":"ContainerStarted","Data":"d5d0ca4c46e04b7440e8f88ac3cd9efc18ddb72ca56da78d560a8f882aacefb2"}
Oct 30 13:21:37.631098 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:37.631051    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp" podStartSLOduration=1.6310363049999999 podStartE2EDuration="1.631036305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:21:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:21:37.626195207 +0000 UTC m=+1796.792559855" watchObservedRunningTime="2026-10-30 13:21:37.631036305 +0000 UTC m=+1796.797400953"
Oct 30 13:21:38.009814 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:38.009767    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp"
Oct 30 13:21:38.026801 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:38.026753    4364 patch_prober.go:29] interesting pod/e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Oct 30 13:21:38.027091 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:38.027047    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp" podUID="a9ee49f0-1a2e-4b0b-8046-36b55a049fad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Oct 30 13:21:38.617162 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:38.617136    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-8d82d5-65c88b865b-xt6lk" event={"ID":"4813393d-187f-4a3a-a77b-6151b211d2f8","Type":"ContainerStarted","Data":"b652338b553545dd0fc3e24484182bc304b269266e06e732bc56b139550c884f"}
Oct 30 13:21:38.617340 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:38.617189    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-8d82d5-65c88b865b-xt6lk" event={"ID":"4813393d-187f-4a3a-a77b-6151b211d2f8","Type":"ContainerStarted","Data":"70a5480174238e72e167548405e15adbb23486f4980c61ac5b8a7f106a0c1d4e"}
Oct 30 13:21:38.617340 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:38.617217    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-8d82d5-65c88b865b-xt6lk"
Oct 30 13:21:38.660981 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:38.660918    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-8d82d5-65c88b865b-xt6lk" podStartSLOduration=2.278394956 podStartE2EDuration="2.660897997s" totalImagesPullingTime="382.503041ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:21:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:21:38.659278409 +0000 UTC m=+1797.825643060" watchObservedRunningTime="2026-10-30 13:21:38.660897997 +0000 UTC m=+1797.827262647"
Oct 30 13:21:39.006345 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:39.006266    4364 patch_prober.go:29] interesting pod/e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Oct 30 13:21:39.006345 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:39.006315    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp" podUID="a9ee49f0-1a2e-4b0b-8046-36b55a049fad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Oct 30 13:21:40.009579 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:40.009049    4364 patch_prober.go:29] interesting pod/e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Oct 30 13:21:40.009579 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:40.009115    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp" podUID="a9ee49f0-1a2e-4b0b-8046-36b55a049fad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Oct 30 13:21:41.006268 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:41.006219    4364 patch_prober.go:29] interesting pod/e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Oct 30 13:21:41.006462 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:41.006294    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp" podUID="a9ee49f0-1a2e-4b0b-8046-36b55a049fad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Oct 30 13:21:41.532137 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:21:41.532078    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:21:42.008691 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:42.005943    4364 patch_prober.go:29] interesting pod/e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Oct 30 13:21:42.008691 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:42.006008    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp" podUID="a9ee49f0-1a2e-4b0b-8046-36b55a049fad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Oct 30 13:21:42.338477 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:42.338383    4364 scope.go:122] "RemoveContainer" containerID="d18fc03b84b7028bf478c34e31c7360691f7f8f28f6266bd0a9c30c69777bb0e"
Oct 30 13:21:42.466659 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:42.466160    4364 scope.go:122] "RemoveContainer" containerID="e1949dc3f91430f727412b23553db768137f2138714f69fa30c6f588196dd9c8"
Oct 30 13:21:42.514859 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:42.514835    4364 scope.go:122] "RemoveContainer" containerID="dbf83f3f7e181237d23f1661ea38da051cb0b66dfc33975236f2dee22e8de1a8"
Oct 30 13:21:42.526326 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:42.526300    4364 scope.go:122] "RemoveContainer" containerID="ed885b2a549c22047b0e1ed8a282fcdc976343bf43bfea0834e24c9dc1c435fc"
Oct 30 13:21:43.006280 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:43.006177    4364 patch_prober.go:29] interesting pod/e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Oct 30 13:21:43.006280 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:43.006239    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp" podUID="a9ee49f0-1a2e-4b0b-8046-36b55a049fad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Oct 30 13:21:44.211311 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:44.211276    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp"
Oct 30 13:21:44.212203 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:44.212171    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp"
Oct 30 13:21:44.212895 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:44.212873    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp"
Oct 30 13:21:44.633215 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:44.633182    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-8d82d5-65c88b865b-xt6lk"
Oct 30 13:21:55.464881 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:55.464849    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc"]
Oct 30 13:21:55.465310 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:55.465075    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc" podUID="4710cc0d-1402-489e-826b-5c44a8b6f9f0" containerName="istio-proxy" containerID="cri-o://c2420b0d52c0c1fce4aae2114400d2c4a23e4850cd720b7e4909ce684e796ac9" gracePeriod=30
Oct 30 13:21:59.678907 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:21:59.678870    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:22:00.755769 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:00.755702    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc"
Oct 30 13:22:00.904288 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:00.904255    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4710cc0d-1402-489e-826b-5c44a8b6f9f0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4710cc0d-1402-489e-826b-5c44a8b6f9f0-credential-socket\") pod \"4710cc0d-1402-489e-826b-5c44a8b6f9f0\" (UID: \"4710cc0d-1402-489e-826b-5c44a8b6f9f0\") "
Oct 30 13:22:00.904477 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:00.904308    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4710cc0d-1402-489e-826b-5c44a8b6f9f0-kube-api-access-wr9l4\" (UniqueName: \"kubernetes.io/projected/4710cc0d-1402-489e-826b-5c44a8b6f9f0-kube-api-access-wr9l4\") pod \"4710cc0d-1402-489e-826b-5c44a8b6f9f0\" (UID: \"4710cc0d-1402-489e-826b-5c44a8b6f9f0\") "
Oct 30 13:22:00.904477 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:00.904345    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4710cc0d-1402-489e-826b-5c44a8b6f9f0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4710cc0d-1402-489e-826b-5c44a8b6f9f0-istiod-ca-cert\") pod \"4710cc0d-1402-489e-826b-5c44a8b6f9f0\" (UID: \"4710cc0d-1402-489e-826b-5c44a8b6f9f0\") "
Oct 30 13:22:00.904477 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:00.904449    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4710cc0d-1402-489e-826b-5c44a8b6f9f0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4710cc0d-1402-489e-826b-5c44a8b6f9f0-istio-data\") pod \"4710cc0d-1402-489e-826b-5c44a8b6f9f0\" (UID: \"4710cc0d-1402-489e-826b-5c44a8b6f9f0\") "
Oct 30 13:22:00.904950 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:00.904534    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4710cc0d-1402-489e-826b-5c44a8b6f9f0-istio-token\" (UniqueName: \"kubernetes.io/projected/4710cc0d-1402-489e-826b-5c44a8b6f9f0-istio-token\") pod \"4710cc0d-1402-489e-826b-5c44a8b6f9f0\" (UID: \"4710cc0d-1402-489e-826b-5c44a8b6f9f0\") "
Oct 30 13:22:00.904950 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:00.904570    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4710cc0d-1402-489e-826b-5c44a8b6f9f0-credential-socket" pod "4710cc0d-1402-489e-826b-5c44a8b6f9f0" (UID: "4710cc0d-1402-489e-826b-5c44a8b6f9f0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:22:00.904950 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:00.904624    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4710cc0d-1402-489e-826b-5c44a8b6f9f0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4710cc0d-1402-489e-826b-5c44a8b6f9f0-istio-envoy\") pod \"4710cc0d-1402-489e-826b-5c44a8b6f9f0\" (UID: \"4710cc0d-1402-489e-826b-5c44a8b6f9f0\") "
Oct 30 13:22:00.904950 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:00.904705    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4710cc0d-1402-489e-826b-5c44a8b6f9f0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4710cc0d-1402-489e-826b-5c44a8b6f9f0-istio-podinfo\") pod \"4710cc0d-1402-489e-826b-5c44a8b6f9f0\" (UID: \"4710cc0d-1402-489e-826b-5c44a8b6f9f0\") "
Oct 30 13:22:00.904950 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:00.904883    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4710cc0d-1402-489e-826b-5c44a8b6f9f0-istiod-ca-cert" pod "4710cc0d-1402-489e-826b-5c44a8b6f9f0" (UID: "4710cc0d-1402-489e-826b-5c44a8b6f9f0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:22:00.904950 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:00.904901    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4710cc0d-1402-489e-826b-5c44a8b6f9f0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4710cc0d-1402-489e-826b-5c44a8b6f9f0-workload-socket\") pod \"4710cc0d-1402-489e-826b-5c44a8b6f9f0\" (UID: \"4710cc0d-1402-489e-826b-5c44a8b6f9f0\") "
Oct 30 13:22:00.904950 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:00.904933    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4710cc0d-1402-489e-826b-5c44a8b6f9f0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4710cc0d-1402-489e-826b-5c44a8b6f9f0-workload-certs\") pod \"4710cc0d-1402-489e-826b-5c44a8b6f9f0\" (UID: \"4710cc0d-1402-489e-826b-5c44a8b6f9f0\") "
Oct 30 13:22:00.905431 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:00.905208    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4710cc0d-1402-489e-826b-5c44a8b6f9f0-istio-data" pod "4710cc0d-1402-489e-826b-5c44a8b6f9f0" (UID: "4710cc0d-1402-489e-826b-5c44a8b6f9f0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:22:00.905431 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:00.905227    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4710cc0d-1402-489e-826b-5c44a8b6f9f0-workload-socket" pod "4710cc0d-1402-489e-826b-5c44a8b6f9f0" (UID: "4710cc0d-1402-489e-826b-5c44a8b6f9f0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:22:00.905538 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:00.905509    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4710cc0d-1402-489e-826b-5c44a8b6f9f0-workload-certs" pod "4710cc0d-1402-489e-826b-5c44a8b6f9f0" (UID: "4710cc0d-1402-489e-826b-5c44a8b6f9f0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:22:00.905993 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:00.905973    4364 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4710cc0d-1402-489e-826b-5c44a8b6f9f0-istio-data\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:22:00.906056 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:00.906001    4364 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4710cc0d-1402-489e-826b-5c44a8b6f9f0-workload-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:22:00.906056 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:00.906014    4364 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4710cc0d-1402-489e-826b-5c44a8b6f9f0-workload-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:22:00.906056 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:00.906026    4364 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4710cc0d-1402-489e-826b-5c44a8b6f9f0-credential-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:22:00.906056 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:00.906039    4364 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4710cc0d-1402-489e-826b-5c44a8b6f9f0-istiod-ca-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:22:00.906932 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:00.906910    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4710cc0d-1402-489e-826b-5c44a8b6f9f0-kube-api-access-wr9l4" pod "4710cc0d-1402-489e-826b-5c44a8b6f9f0" (UID: "4710cc0d-1402-489e-826b-5c44a8b6f9f0"). InnerVolumeSpecName "kube-api-access-wr9l4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:22:00.907220 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:00.907201    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4710cc0d-1402-489e-826b-5c44a8b6f9f0-istio-podinfo" pod "4710cc0d-1402-489e-826b-5c44a8b6f9f0" (UID: "4710cc0d-1402-489e-826b-5c44a8b6f9f0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:22:00.907393 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:00.907372    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4710cc0d-1402-489e-826b-5c44a8b6f9f0-istio-token" pod "4710cc0d-1402-489e-826b-5c44a8b6f9f0" (UID: "4710cc0d-1402-489e-826b-5c44a8b6f9f0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:22:00.907393 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:00.907384    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4710cc0d-1402-489e-826b-5c44a8b6f9f0-istio-envoy" pod "4710cc0d-1402-489e-826b-5c44a8b6f9f0" (UID: "4710cc0d-1402-489e-826b-5c44a8b6f9f0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:22:01.007290 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:01.007212    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wr9l4\" (UniqueName: \"kubernetes.io/projected/4710cc0d-1402-489e-826b-5c44a8b6f9f0-kube-api-access-wr9l4\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:22:01.007290 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:01.007244    4364 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4710cc0d-1402-489e-826b-5c44a8b6f9f0-istio-token\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:22:01.007290 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:01.007254    4364 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4710cc0d-1402-489e-826b-5c44a8b6f9f0-istio-envoy\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:22:01.007290 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:01.007264    4364 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4710cc0d-1402-489e-826b-5c44a8b6f9f0-istio-podinfo\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:22:01.061799 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:01.061758    4364 generic.go:356] "Generic (PLEG): container finished" podID="4710cc0d-1402-489e-826b-5c44a8b6f9f0" containerID="c2420b0d52c0c1fce4aae2114400d2c4a23e4850cd720b7e4909ce684e796ac9" exitCode=0
Oct 30 13:22:01.061974 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:01.061807    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc"
Oct 30 13:22:01.061974 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:01.061841    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc" event={"ID":"4710cc0d-1402-489e-826b-5c44a8b6f9f0","Type":"ContainerDied","Data":"c2420b0d52c0c1fce4aae2114400d2c4a23e4850cd720b7e4909ce684e796ac9"}
Oct 30 13:22:01.061974 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:01.061900    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc" event={"ID":"4710cc0d-1402-489e-826b-5c44a8b6f9f0","Type":"ContainerDied","Data":"9796fac3320cfec9612196021d5f43926ffb793eab0ecb1a4fa64deb618882f6"}
Oct 30 13:22:01.061974 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:01.061927    4364 scope.go:122] "RemoveContainer" containerID="c2420b0d52c0c1fce4aae2114400d2c4a23e4850cd720b7e4909ce684e796ac9"
Oct 30 13:22:01.077815 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:01.077074    4364 scope.go:122] "RemoveContainer" containerID="cf0f3f3a05778ebb10b88bdada7234adf4c2c51ebe5e2dce344bdd78a1e7b373"
Oct 30 13:22:01.089423 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:01.089403    4364 scope.go:122] "RemoveContainer" containerID="c2420b0d52c0c1fce4aae2114400d2c4a23e4850cd720b7e4909ce684e796ac9"
Oct 30 13:22:01.089754 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:22:01.089733    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2420b0d52c0c1fce4aae2114400d2c4a23e4850cd720b7e4909ce684e796ac9\": container with ID starting with c2420b0d52c0c1fce4aae2114400d2c4a23e4850cd720b7e4909ce684e796ac9 not found: ID does not exist" containerID="c2420b0d52c0c1fce4aae2114400d2c4a23e4850cd720b7e4909ce684e796ac9"
Oct 30 13:22:01.089829 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:01.089765    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2420b0d52c0c1fce4aae2114400d2c4a23e4850cd720b7e4909ce684e796ac9"} err="failed to get container status \"c2420b0d52c0c1fce4aae2114400d2c4a23e4850cd720b7e4909ce684e796ac9\": rpc error: code = NotFound desc = could not find container \"c2420b0d52c0c1fce4aae2114400d2c4a23e4850cd720b7e4909ce684e796ac9\": container with ID starting with c2420b0d52c0c1fce4aae2114400d2c4a23e4850cd720b7e4909ce684e796ac9 not found: ID does not exist"
Oct 30 13:22:01.089829 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:01.089785    4364 scope.go:122] "RemoveContainer" containerID="cf0f3f3a05778ebb10b88bdada7234adf4c2c51ebe5e2dce344bdd78a1e7b373"
Oct 30 13:22:01.090048 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:22:01.090031    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf0f3f3a05778ebb10b88bdada7234adf4c2c51ebe5e2dce344bdd78a1e7b373\": container with ID starting with cf0f3f3a05778ebb10b88bdada7234adf4c2c51ebe5e2dce344bdd78a1e7b373 not found: ID does not exist" containerID="cf0f3f3a05778ebb10b88bdada7234adf4c2c51ebe5e2dce344bdd78a1e7b373"
Oct 30 13:22:01.090095 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:01.090053    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf0f3f3a05778ebb10b88bdada7234adf4c2c51ebe5e2dce344bdd78a1e7b373"} err="failed to get container status \"cf0f3f3a05778ebb10b88bdada7234adf4c2c51ebe5e2dce344bdd78a1e7b373\": rpc error: code = NotFound desc = could not find container \"cf0f3f3a05778ebb10b88bdada7234adf4c2c51ebe5e2dce344bdd78a1e7b373\": container with ID starting with cf0f3f3a05778ebb10b88bdada7234adf4c2c51ebe5e2dce344bdd78a1e7b373 not found: ID does not exist"
Oct 30 13:22:01.132013 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:01.131980    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc"]
Oct 30 13:22:01.139555 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:01.139520    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-sbzvc"]
Oct 30 13:22:02.774621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:02.770528    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6"]
Oct 30 13:22:02.774621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:02.772102    4364 cpu_manager.go:414] "containerMap: removing container" podUID="4710cc0d-1402-489e-826b-5c44a8b6f9f0" containerName="istio-proxy"
Oct 30 13:22:02.774621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:02.772124    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4710cc0d-1402-489e-826b-5c44a8b6f9f0" containerName="istio-proxy"
Oct 30 13:22:02.774621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:02.772140    4364 cpu_manager.go:414] "containerMap: removing container" podUID="4710cc0d-1402-489e-826b-5c44a8b6f9f0" containerName="istio-proxy"
Oct 30 13:22:02.774621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:02.772147    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4710cc0d-1402-489e-826b-5c44a8b6f9f0" containerName="istio-proxy"
Oct 30 13:22:02.774621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:02.772374    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="4710cc0d-1402-489e-826b-5c44a8b6f9f0" containerName="istio-proxy"
Oct 30 13:22:02.774621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:02.772389    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="4710cc0d-1402-489e-826b-5c44a8b6f9f0" containerName="istio-proxy"
Oct 30 13:22:03.642700 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:03.630476    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-b5a369/test-model-b5a369-kserve-6bfb684495-gckcs" podUID="d52aab51-7fdf-4be6-8893-b5fe88dbe341" containerName="main" probeResult="failure" output="Get \"https://10.133.0.188:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:22:03.716350 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:03.716310    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6"]
Oct 30 13:22:03.717381 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:03.717361    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6"
Oct 30 13:22:03.761959 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:03.760139    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-376c89-openshift-default-dockercfg-twmvq\""
Oct 30 13:22:03.848783 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:03.837285    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9846347-72dc-472f-8bb1-57d39f1be48b-istiod-ca-cert\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6\" (UID: \"f9846347-72dc-472f-8bb1-57d39f1be48b\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6"
Oct 30 13:22:03.850701 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:03.850638    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9846347-72dc-472f-8bb1-57d39f1be48b-istio-token\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6\" (UID: \"f9846347-72dc-472f-8bb1-57d39f1be48b\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6"
Oct 30 13:22:03.850937 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:03.850919    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9846347-72dc-472f-8bb1-57d39f1be48b-workload-certs\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6\" (UID: \"f9846347-72dc-472f-8bb1-57d39f1be48b\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6"
Oct 30 13:22:03.851154 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:03.851132    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9846347-72dc-472f-8bb1-57d39f1be48b-credential-socket\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6\" (UID: \"f9846347-72dc-472f-8bb1-57d39f1be48b\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6"
Oct 30 13:22:03.866624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:03.851410    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9846347-72dc-472f-8bb1-57d39f1be48b-istio-envoy\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6\" (UID: \"f9846347-72dc-472f-8bb1-57d39f1be48b\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6"
Oct 30 13:22:03.866624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:03.864027    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9846347-72dc-472f-8bb1-57d39f1be48b-istio-podinfo\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6\" (UID: \"f9846347-72dc-472f-8bb1-57d39f1be48b\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6"
Oct 30 13:22:03.866624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:03.864155    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6d4l\" (UniqueName: \"kubernetes.io/projected/f9846347-72dc-472f-8bb1-57d39f1be48b-kube-api-access-f6d4l\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6\" (UID: \"f9846347-72dc-472f-8bb1-57d39f1be48b\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6"
Oct 30 13:22:03.866624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:03.864266    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9846347-72dc-472f-8bb1-57d39f1be48b-istio-data\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6\" (UID: \"f9846347-72dc-472f-8bb1-57d39f1be48b\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6"
Oct 30 13:22:03.866624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:03.864357    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9846347-72dc-472f-8bb1-57d39f1be48b-workload-socket\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6\" (UID: \"f9846347-72dc-472f-8bb1-57d39f1be48b\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6"
Oct 30 13:22:03.867199 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:03.867151    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4710cc0d-1402-489e-826b-5c44a8b6f9f0" path="/var/lib/kubelet/pods/4710cc0d-1402-489e-826b-5c44a8b6f9f0/volumes"
Oct 30 13:22:03.971030 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:03.970924    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9846347-72dc-472f-8bb1-57d39f1be48b-istio-envoy\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6\" (UID: \"f9846347-72dc-472f-8bb1-57d39f1be48b\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6"
Oct 30 13:22:03.971341 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:03.971318    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9846347-72dc-472f-8bb1-57d39f1be48b-istio-podinfo\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6\" (UID: \"f9846347-72dc-472f-8bb1-57d39f1be48b\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6"
Oct 30 13:22:03.971540 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:03.971524    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6d4l\" (UniqueName: \"kubernetes.io/projected/f9846347-72dc-472f-8bb1-57d39f1be48b-kube-api-access-f6d4l\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6\" (UID: \"f9846347-72dc-472f-8bb1-57d39f1be48b\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6"
Oct 30 13:22:03.971736 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:03.971718    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9846347-72dc-472f-8bb1-57d39f1be48b-istio-data\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6\" (UID: \"f9846347-72dc-472f-8bb1-57d39f1be48b\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6"
Oct 30 13:22:03.971929 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:03.971912    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9846347-72dc-472f-8bb1-57d39f1be48b-workload-socket\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6\" (UID: \"f9846347-72dc-472f-8bb1-57d39f1be48b\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6"
Oct 30 13:22:03.972158 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:03.972142    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9846347-72dc-472f-8bb1-57d39f1be48b-istiod-ca-cert\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6\" (UID: \"f9846347-72dc-472f-8bb1-57d39f1be48b\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6"
Oct 30 13:22:03.972310 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:03.972296    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9846347-72dc-472f-8bb1-57d39f1be48b-istio-token\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6\" (UID: \"f9846347-72dc-472f-8bb1-57d39f1be48b\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6"
Oct 30 13:22:03.972431 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:03.972416    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9846347-72dc-472f-8bb1-57d39f1be48b-workload-certs\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6\" (UID: \"f9846347-72dc-472f-8bb1-57d39f1be48b\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6"
Oct 30 13:22:03.972613 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:03.972580    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9846347-72dc-472f-8bb1-57d39f1be48b-credential-socket\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6\" (UID: \"f9846347-72dc-472f-8bb1-57d39f1be48b\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6"
Oct 30 13:22:03.975128 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:03.975093    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9846347-72dc-472f-8bb1-57d39f1be48b-workload-certs\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6\" (UID: \"f9846347-72dc-472f-8bb1-57d39f1be48b\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6"
Oct 30 13:22:03.975814 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:03.975789    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9846347-72dc-472f-8bb1-57d39f1be48b-istiod-ca-cert\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6\" (UID: \"f9846347-72dc-472f-8bb1-57d39f1be48b\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6"
Oct 30 13:22:04.020638 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:03.976079    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9846347-72dc-472f-8bb1-57d39f1be48b-istio-envoy\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6\" (UID: \"f9846347-72dc-472f-8bb1-57d39f1be48b\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6"
Oct 30 13:22:04.020638 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:03.976439    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9846347-72dc-472f-8bb1-57d39f1be48b-workload-socket\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6\" (UID: \"f9846347-72dc-472f-8bb1-57d39f1be48b\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6"
Oct 30 13:22:04.020638 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:03.977324    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9846347-72dc-472f-8bb1-57d39f1be48b-credential-socket\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6\" (UID: \"f9846347-72dc-472f-8bb1-57d39f1be48b\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6"
Oct 30 13:22:04.020638 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:03.977369    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9846347-72dc-472f-8bb1-57d39f1be48b-istio-podinfo\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6\" (UID: \"f9846347-72dc-472f-8bb1-57d39f1be48b\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6"
Oct 30 13:22:04.020638 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:03.978319    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9846347-72dc-472f-8bb1-57d39f1be48b-istio-data\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6\" (UID: \"f9846347-72dc-472f-8bb1-57d39f1be48b\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6"
Oct 30 13:22:04.049590 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:04.049519    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6d4l\" (UniqueName: \"kubernetes.io/projected/f9846347-72dc-472f-8bb1-57d39f1be48b-kube-api-access-f6d4l\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6\" (UID: \"f9846347-72dc-472f-8bb1-57d39f1be48b\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6"
Oct 30 13:22:04.054622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:04.050272    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9846347-72dc-472f-8bb1-57d39f1be48b-istio-token\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6\" (UID: \"f9846347-72dc-472f-8bb1-57d39f1be48b\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6"
Oct 30 13:22:04.082624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:04.070810    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6"
Oct 30 13:22:05.059044 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:05.052470    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:22:05.059044 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:05.052582    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:22:05.059044 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:05.052649    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:22:05.059044 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:05.054968    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6"]
Oct 30 13:22:06.068677 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:06.066826    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6" event={"ID":"f9846347-72dc-472f-8bb1-57d39f1be48b","Type":"ContainerStarted","Data":"ed645de8c0778c120d3496bc0a7411e7104faa949fcc3b120a03bf060d83e434"}
Oct 30 13:22:06.068677 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:06.066899    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6" event={"ID":"f9846347-72dc-472f-8bb1-57d39f1be48b","Type":"ContainerStarted","Data":"0b91d24eed723e32d9a4fb478237dbad36b93feb0cfc3742b1267e48833ede83"}
Oct 30 13:22:06.091564 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:06.091478    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6"
Oct 30 13:22:06.093001 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:06.092697    4364 patch_prober.go:29] interesting pod/e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Oct 30 13:22:06.093001 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:06.092751    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6" podUID="f9846347-72dc-472f-8bb1-57d39f1be48b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Oct 30 13:22:06.158312 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:06.158216    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6" podStartSLOduration=4.158196674 podStartE2EDuration="4.158196674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:22:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:22:06.155501929 +0000 UTC m=+1825.321866578" watchObservedRunningTime="2026-10-30 13:22:06.158196674 +0000 UTC m=+1825.324561324"
Oct 30 13:22:07.133625 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:07.126371    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6"
Oct 30 13:22:07.167613 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:07.167557    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6"
Oct 30 13:22:07.167912 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:07.167882    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6"
Oct 30 13:22:10.172789 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:10.172745    4364 patch_prober.go:29] interesting pod/e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:22:10.173315 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:10.172824    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz" podUID="52bf85ca-e35d-4a85-85f4-21d6d20a191d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:22:11.851293 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:22:11.851211    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:22:13.643750 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:13.643713    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-31721c/test-model-31721c-kserve-64c44bfc-dmnrj"]
Oct 30 13:22:13.712019 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:13.711970    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-31721c/test-model-31721c-kserve-64c44bfc-dmnrj"]
Oct 30 13:22:13.712190 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:13.712097    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-31721c/test-model-31721c-kserve-64c44bfc-dmnrj"
Oct 30 13:22:13.728242 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:13.728212    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-31721c\"/\"openshift-service-ca.crt\""
Oct 30 13:22:13.728428 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:13.728252    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-31721c\"/\"kube-root-ca.crt\""
Oct 30 13:22:13.740335 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:13.740289    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-31721c\"/\"test-model-31721c-kserve-self-signed-certs\""
Oct 30 13:22:13.740527 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:13.740348    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-31721c\"/\"default-dockercfg-47mqw\""
Oct 30 13:22:13.872554 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:13.872512    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac457ffd-caeb-419f-b448-1b21942ead0f-dshm\") pod \"test-model-31721c-kserve-64c44bfc-dmnrj\" (UID: \"ac457ffd-caeb-419f-b448-1b21942ead0f\") " pod="ai-tenant-e2e-inf-a-31721c/test-model-31721c-kserve-64c44bfc-dmnrj"
Oct 30 13:22:15.121758 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:13.872567    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac457ffd-caeb-419f-b448-1b21942ead0f-model-cache\") pod \"test-model-31721c-kserve-64c44bfc-dmnrj\" (UID: \"ac457ffd-caeb-419f-b448-1b21942ead0f\") " pod="ai-tenant-e2e-inf-a-31721c/test-model-31721c-kserve-64c44bfc-dmnrj"
Oct 30 13:22:15.121758 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:13.872635    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt984\" (UniqueName: \"kubernetes.io/projected/ac457ffd-caeb-419f-b448-1b21942ead0f-kube-api-access-tt984\") pod \"test-model-31721c-kserve-64c44bfc-dmnrj\" (UID: \"ac457ffd-caeb-419f-b448-1b21942ead0f\") " pod="ai-tenant-e2e-inf-a-31721c/test-model-31721c-kserve-64c44bfc-dmnrj"
Oct 30 13:22:15.121758 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:13.872818    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac457ffd-caeb-419f-b448-1b21942ead0f-home\") pod \"test-model-31721c-kserve-64c44bfc-dmnrj\" (UID: \"ac457ffd-caeb-419f-b448-1b21942ead0f\") " pod="ai-tenant-e2e-inf-a-31721c/test-model-31721c-kserve-64c44bfc-dmnrj"
Oct 30 13:22:15.121758 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:13.872933    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac457ffd-caeb-419f-b448-1b21942ead0f-tls-certs\") pod \"test-model-31721c-kserve-64c44bfc-dmnrj\" (UID: \"ac457ffd-caeb-419f-b448-1b21942ead0f\") " pod="ai-tenant-e2e-inf-a-31721c/test-model-31721c-kserve-64c44bfc-dmnrj"
Oct 30 13:22:15.121758 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:13.873058    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac457ffd-caeb-419f-b448-1b21942ead0f-tmp-dir\") pod \"test-model-31721c-kserve-64c44bfc-dmnrj\" (UID: \"ac457ffd-caeb-419f-b448-1b21942ead0f\") " pod="ai-tenant-e2e-inf-a-31721c/test-model-31721c-kserve-64c44bfc-dmnrj"
Oct 30 13:22:15.121758 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:13.990955    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac457ffd-caeb-419f-b448-1b21942ead0f-model-cache\") pod \"test-model-31721c-kserve-64c44bfc-dmnrj\" (UID: \"ac457ffd-caeb-419f-b448-1b21942ead0f\") " pod="ai-tenant-e2e-inf-a-31721c/test-model-31721c-kserve-64c44bfc-dmnrj"
Oct 30 13:22:15.121758 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:13.991461    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac457ffd-caeb-419f-b448-1b21942ead0f-model-cache\") pod \"test-model-31721c-kserve-64c44bfc-dmnrj\" (UID: \"ac457ffd-caeb-419f-b448-1b21942ead0f\") " pod="ai-tenant-e2e-inf-a-31721c/test-model-31721c-kserve-64c44bfc-dmnrj"
Oct 30 13:22:15.121758 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:13.991552    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt984\" (UniqueName: \"kubernetes.io/projected/ac457ffd-caeb-419f-b448-1b21942ead0f-kube-api-access-tt984\") pod \"test-model-31721c-kserve-64c44bfc-dmnrj\" (UID: \"ac457ffd-caeb-419f-b448-1b21942ead0f\") " pod="ai-tenant-e2e-inf-a-31721c/test-model-31721c-kserve-64c44bfc-dmnrj"
Oct 30 13:22:15.121758 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:14.011954    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac457ffd-caeb-419f-b448-1b21942ead0f-home\") pod \"test-model-31721c-kserve-64c44bfc-dmnrj\" (UID: \"ac457ffd-caeb-419f-b448-1b21942ead0f\") " pod="ai-tenant-e2e-inf-a-31721c/test-model-31721c-kserve-64c44bfc-dmnrj"
Oct 30 13:22:15.121758 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:14.012467    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac457ffd-caeb-419f-b448-1b21942ead0f-home\") pod \"test-model-31721c-kserve-64c44bfc-dmnrj\" (UID: \"ac457ffd-caeb-419f-b448-1b21942ead0f\") " pod="ai-tenant-e2e-inf-a-31721c/test-model-31721c-kserve-64c44bfc-dmnrj"
Oct 30 13:22:15.121758 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:14.012699    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac457ffd-caeb-419f-b448-1b21942ead0f-tls-certs\") pod \"test-model-31721c-kserve-64c44bfc-dmnrj\" (UID: \"ac457ffd-caeb-419f-b448-1b21942ead0f\") " pod="ai-tenant-e2e-inf-a-31721c/test-model-31721c-kserve-64c44bfc-dmnrj"
Oct 30 13:22:15.121758 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:14.013324    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac457ffd-caeb-419f-b448-1b21942ead0f-tmp-dir\") pod \"test-model-31721c-kserve-64c44bfc-dmnrj\" (UID: \"ac457ffd-caeb-419f-b448-1b21942ead0f\") " pod="ai-tenant-e2e-inf-a-31721c/test-model-31721c-kserve-64c44bfc-dmnrj"
Oct 30 13:22:15.121758 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:14.013642    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac457ffd-caeb-419f-b448-1b21942ead0f-tmp-dir\") pod \"test-model-31721c-kserve-64c44bfc-dmnrj\" (UID: \"ac457ffd-caeb-419f-b448-1b21942ead0f\") " pod="ai-tenant-e2e-inf-a-31721c/test-model-31721c-kserve-64c44bfc-dmnrj"
Oct 30 13:22:15.121758 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:14.013954    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac457ffd-caeb-419f-b448-1b21942ead0f-dshm\") pod \"test-model-31721c-kserve-64c44bfc-dmnrj\" (UID: \"ac457ffd-caeb-419f-b448-1b21942ead0f\") " pod="ai-tenant-e2e-inf-a-31721c/test-model-31721c-kserve-64c44bfc-dmnrj"
Oct 30 13:22:15.121758 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:14.075972    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac457ffd-caeb-419f-b448-1b21942ead0f-dshm\") pod \"test-model-31721c-kserve-64c44bfc-dmnrj\" (UID: \"ac457ffd-caeb-419f-b448-1b21942ead0f\") " pod="ai-tenant-e2e-inf-a-31721c/test-model-31721c-kserve-64c44bfc-dmnrj"
Oct 30 13:22:15.121758 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:14.076383    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac457ffd-caeb-419f-b448-1b21942ead0f-tls-certs\") pod \"test-model-31721c-kserve-64c44bfc-dmnrj\" (UID: \"ac457ffd-caeb-419f-b448-1b21942ead0f\") " pod="ai-tenant-e2e-inf-a-31721c/test-model-31721c-kserve-64c44bfc-dmnrj"
Oct 30 13:22:15.121758 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:14.080457    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt984\" (UniqueName: \"kubernetes.io/projected/ac457ffd-caeb-419f-b448-1b21942ead0f-kube-api-access-tt984\") pod \"test-model-31721c-kserve-64c44bfc-dmnrj\" (UID: \"ac457ffd-caeb-419f-b448-1b21942ead0f\") " pod="ai-tenant-e2e-inf-a-31721c/test-model-31721c-kserve-64c44bfc-dmnrj"
Oct 30 13:22:15.121758 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:14.349266    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-31721c/test-model-31721c-kserve-64c44bfc-dmnrj"
Oct 30 13:22:15.121758 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:22:14.682676    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podac457ffd_caeb_419f_b448_1b21942ead0f.slice/crio-763f85de5cadb71a16518c209e5fe71346ba02b5298107b618186d12e62f096d WatchSource:0}: Error finding container 763f85de5cadb71a16518c209e5fe71346ba02b5298107b618186d12e62f096d: Status 404 returned error can't find the container with id 763f85de5cadb71a16518c209e5fe71346ba02b5298107b618186d12e62f096d
Oct 30 13:22:15.121758 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:14.683546    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-31721c/test-model-31721c-kserve-64c44bfc-dmnrj"]
Oct 30 13:22:15.313999 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:15.247038    4364 generic.go:356] "Generic (PLEG): container finished" podID="fb9e1069-efce-4243-b809-ad470e124bd1" containerID="cae3c76fb0576f5591a338bc3d5958baa7cc4110ba759c4d7f4e2419ce57be51" exitCode=0
Oct 30 13:22:15.313999 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:15.247112    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks" event={"ID":"fb9e1069-efce-4243-b809-ad470e124bd1","Type":"ContainerDied","Data":"cae3c76fb0576f5591a338bc3d5958baa7cc4110ba759c4d7f4e2419ce57be51"}
Oct 30 13:22:15.313999 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:15.247811    4364 scope.go:122] "RemoveContainer" containerID="cae3c76fb0576f5591a338bc3d5958baa7cc4110ba759c4d7f4e2419ce57be51"
Oct 30 13:22:15.313999 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:15.250460    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:22:15.313999 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:15.250554    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:22:15.313999 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:15.250624    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:22:15.313999 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:15.254382    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-31721c/test-model-31721c-kserve-64c44bfc-dmnrj" event={"ID":"ac457ffd-caeb-419f-b448-1b21942ead0f","Type":"ContainerStarted","Data":"15677d30c7a218cfb710b093bd2a792102cd52cadff96f8edcd5bc23c16a1097"}
Oct 30 13:22:15.313999 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:15.254428    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-31721c/test-model-31721c-kserve-64c44bfc-dmnrj" event={"ID":"ac457ffd-caeb-419f-b448-1b21942ead0f","Type":"ContainerStarted","Data":"763f85de5cadb71a16518c209e5fe71346ba02b5298107b618186d12e62f096d"}
Oct 30 13:22:15.313999 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:15.259103    4364 generic.go:356] "Generic (PLEG): container finished" podID="52bf85ca-e35d-4a85-85f4-21d6d20a191d" containerID="383280d0ea2ed966b15a3e751091cf11b0522a50adeef0edab2295f885e8120d" exitCode=0
Oct 30 13:22:15.313999 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:15.259153    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz" event={"ID":"52bf85ca-e35d-4a85-85f4-21d6d20a191d","Type":"ContainerDied","Data":"383280d0ea2ed966b15a3e751091cf11b0522a50adeef0edab2295f885e8120d"}
Oct 30 13:22:15.313999 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:15.259669    4364 scope.go:122] "RemoveContainer" containerID="383280d0ea2ed966b15a3e751091cf11b0522a50adeef0edab2295f885e8120d"
Oct 30 13:22:15.313999 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:15.266529    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:22:15.313999 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:15.266619    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:22:15.313999 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:15.266657    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:22:15.350130 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:15.350065    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-31721c/test-model-31721c-kserve-64c44bfc-dmnrj"
Oct 30 13:22:15.350452 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:15.350154    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-31721c/test-model-31721c-kserve-64c44bfc-dmnrj" podStartSLOduration=2.350134924 podStartE2EDuration="2.350134924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:22:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:22:15.349865567 +0000 UTC m=+1834.516230217" watchObservedRunningTime="2026-10-30 13:22:15.350134924 +0000 UTC m=+1834.516499579"
Oct 30 13:22:16.032929 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:16.032894    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k"]
Oct 30 13:22:16.037119 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:16.037096    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k"
Oct 30 13:22:16.088009 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:16.087470    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dcwk\" (UniqueName: \"kubernetes.io/projected/9bed178a-380c-456c-ac60-ed0020bc32f5-kube-api-access-6dcwk\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k\" (UID: \"9bed178a-380c-456c-ac60-ed0020bc32f5\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k"
Oct 30 13:22:16.088009 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:16.087550    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bed178a-380c-456c-ac60-ed0020bc32f5-istio-data\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k\" (UID: \"9bed178a-380c-456c-ac60-ed0020bc32f5\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k"
Oct 30 13:22:16.088009 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:16.087625    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bed178a-380c-456c-ac60-ed0020bc32f5-credential-socket\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k\" (UID: \"9bed178a-380c-456c-ac60-ed0020bc32f5\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k"
Oct 30 13:22:16.088009 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:16.087696    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bed178a-380c-456c-ac60-ed0020bc32f5-workload-socket\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k\" (UID: \"9bed178a-380c-456c-ac60-ed0020bc32f5\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k"
Oct 30 13:22:16.088009 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:16.087781    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bed178a-380c-456c-ac60-ed0020bc32f5-istiod-ca-cert\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k\" (UID: \"9bed178a-380c-456c-ac60-ed0020bc32f5\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k"
Oct 30 13:22:16.088009 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:16.087821    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bed178a-380c-456c-ac60-ed0020bc32f5-istio-podinfo\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k\" (UID: \"9bed178a-380c-456c-ac60-ed0020bc32f5\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k"
Oct 30 13:22:16.088009 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:16.087861    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bed178a-380c-456c-ac60-ed0020bc32f5-istio-envoy\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k\" (UID: \"9bed178a-380c-456c-ac60-ed0020bc32f5\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k"
Oct 30 13:22:16.088511 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:16.088076    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bed178a-380c-456c-ac60-ed0020bc32f5-workload-certs\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k\" (UID: \"9bed178a-380c-456c-ac60-ed0020bc32f5\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k"
Oct 30 13:22:16.088511 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:16.088150    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bed178a-380c-456c-ac60-ed0020bc32f5-istio-token\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k\" (UID: \"9bed178a-380c-456c-ac60-ed0020bc32f5\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k"
Oct 30 13:22:16.089074 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:16.088851    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k"]
Oct 30 13:22:16.189967 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:16.189928    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bed178a-380c-456c-ac60-ed0020bc32f5-workload-certs\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k\" (UID: \"9bed178a-380c-456c-ac60-ed0020bc32f5\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k"
Oct 30 13:22:16.190437 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:16.189992    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bed178a-380c-456c-ac60-ed0020bc32f5-istio-token\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k\" (UID: \"9bed178a-380c-456c-ac60-ed0020bc32f5\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k"
Oct 30 13:22:16.190437 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:16.190110    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dcwk\" (UniqueName: \"kubernetes.io/projected/9bed178a-380c-456c-ac60-ed0020bc32f5-kube-api-access-6dcwk\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k\" (UID: \"9bed178a-380c-456c-ac60-ed0020bc32f5\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k"
Oct 30 13:22:16.190437 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:16.190153    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bed178a-380c-456c-ac60-ed0020bc32f5-istio-data\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k\" (UID: \"9bed178a-380c-456c-ac60-ed0020bc32f5\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k"
Oct 30 13:22:16.190437 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:16.190273    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bed178a-380c-456c-ac60-ed0020bc32f5-credential-socket\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k\" (UID: \"9bed178a-380c-456c-ac60-ed0020bc32f5\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k"
Oct 30 13:22:16.190437 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:16.190339    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bed178a-380c-456c-ac60-ed0020bc32f5-workload-socket\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k\" (UID: \"9bed178a-380c-456c-ac60-ed0020bc32f5\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k"
Oct 30 13:22:16.190437 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:16.190401    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bed178a-380c-456c-ac60-ed0020bc32f5-istiod-ca-cert\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k\" (UID: \"9bed178a-380c-456c-ac60-ed0020bc32f5\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k"
Oct 30 13:22:16.190865 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:16.190456    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bed178a-380c-456c-ac60-ed0020bc32f5-istio-podinfo\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k\" (UID: \"9bed178a-380c-456c-ac60-ed0020bc32f5\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k"
Oct 30 13:22:16.190865 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:16.190489    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bed178a-380c-456c-ac60-ed0020bc32f5-istio-envoy\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k\" (UID: \"9bed178a-380c-456c-ac60-ed0020bc32f5\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k"
Oct 30 13:22:16.190865 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:16.190706    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bed178a-380c-456c-ac60-ed0020bc32f5-istio-data\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k\" (UID: \"9bed178a-380c-456c-ac60-ed0020bc32f5\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k"
Oct 30 13:22:16.191033 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:16.190860    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bed178a-380c-456c-ac60-ed0020bc32f5-credential-socket\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k\" (UID: \"9bed178a-380c-456c-ac60-ed0020bc32f5\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k"
Oct 30 13:22:16.191100 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:16.191024    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bed178a-380c-456c-ac60-ed0020bc32f5-workload-socket\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k\" (UID: \"9bed178a-380c-456c-ac60-ed0020bc32f5\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k"
Oct 30 13:22:16.191201 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:16.191172    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bed178a-380c-456c-ac60-ed0020bc32f5-workload-certs\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k\" (UID: \"9bed178a-380c-456c-ac60-ed0020bc32f5\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k"
Oct 30 13:22:16.191626 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:16.191569    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bed178a-380c-456c-ac60-ed0020bc32f5-istiod-ca-cert\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k\" (UID: \"9bed178a-380c-456c-ac60-ed0020bc32f5\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k"
Oct 30 13:22:16.193121 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:16.193093    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bed178a-380c-456c-ac60-ed0020bc32f5-istio-envoy\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k\" (UID: \"9bed178a-380c-456c-ac60-ed0020bc32f5\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k"
Oct 30 13:22:16.201237 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:16.193304    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bed178a-380c-456c-ac60-ed0020bc32f5-istio-podinfo\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k\" (UID: \"9bed178a-380c-456c-ac60-ed0020bc32f5\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k"
Oct 30 13:22:16.241662 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:16.230365    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dcwk\" (UniqueName: \"kubernetes.io/projected/9bed178a-380c-456c-ac60-ed0020bc32f5-kube-api-access-6dcwk\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k\" (UID: \"9bed178a-380c-456c-ac60-ed0020bc32f5\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k"
Oct 30 13:22:16.247623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:16.246942    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bed178a-380c-456c-ac60-ed0020bc32f5-istio-token\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k\" (UID: \"9bed178a-380c-456c-ac60-ed0020bc32f5\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k"
Oct 30 13:22:16.276416 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:16.276374    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz" event={"ID":"52bf85ca-e35d-4a85-85f4-21d6d20a191d","Type":"ContainerStarted","Data":"83f8744d68777f39cfac6983cf5de47dd6c2a5c57c8234fe41a8e2720924131f"}
Oct 30 13:22:16.278290 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:16.278262    4364 generic.go:356] "Generic (PLEG): container finished" podID="654161bd-9cf8-4f63-9a86-b312763e75cc" containerID="c61d09677f91d362767a518d53c66a58b0fac3495ef391ec0421b63a1c4a6c95" exitCode=0
Oct 30 13:22:16.278430 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:16.278343    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942" event={"ID":"654161bd-9cf8-4f63-9a86-b312763e75cc","Type":"ContainerDied","Data":"c61d09677f91d362767a518d53c66a58b0fac3495ef391ec0421b63a1c4a6c95"}
Oct 30 13:22:16.282643 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:16.278878    4364 scope.go:122] "RemoveContainer" containerID="c61d09677f91d362767a518d53c66a58b0fac3495ef391ec0421b63a1c4a6c95"
Oct 30 13:22:16.303671 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:16.301897    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:22:16.303671 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:16.302011    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:22:16.303671 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:16.302061    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:22:16.330619 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:16.330541    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks" event={"ID":"fb9e1069-efce-4243-b809-ad470e124bd1","Type":"ContainerStarted","Data":"8f51baee7dce5aa432e881fd4599e65672b53548272a2126d277054e6185aad7"}
Oct 30 13:22:16.357128 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:16.357098    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz"
Oct 30 13:22:16.357814 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:16.357777    4364 patch_prober.go:29] interesting pod/e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Oct 30 13:22:16.357965 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:16.357884    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz" podUID="52bf85ca-e35d-4a85-85f4-21d6d20a191d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Oct 30 13:22:16.431460 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:16.405579    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k"
Oct 30 13:22:16.658437 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:16.648787    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks"
Oct 30 13:22:16.672740 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:16.672470    4364 patch_prober.go:29] interesting pod/e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body=
Oct 30 13:22:16.676445 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:16.676406    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks" podUID="fb9e1069-efce-4243-b809-ad470e124bd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused"
Oct 30 13:22:17.151973 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:17.151928    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k"]
Oct 30 13:22:17.153181 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:22:17.153156    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bed178a_380c_456c_ac60_ed0020bc32f5.slice/crio-7fb6d7d0b112c98a9e1e0b3874207299bdf6e8acd43f232a05e8040fca68d1fd WatchSource:0}: Error finding container 7fb6d7d0b112c98a9e1e0b3874207299bdf6e8acd43f232a05e8040fca68d1fd: Status 404 returned error can't find the container with id 7fb6d7d0b112c98a9e1e0b3874207299bdf6e8acd43f232a05e8040fca68d1fd
Oct 30 13:22:17.156935 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:17.156902    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:22:17.157036 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:17.156970    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:22:17.157036 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:17.156999    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:22:17.348307 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:17.348275    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k" event={"ID":"9bed178a-380c-456c-ac60-ed0020bc32f5","Type":"ContainerStarted","Data":"7fb6d7d0b112c98a9e1e0b3874207299bdf6e8acd43f232a05e8040fca68d1fd"}
Oct 30 13:22:17.352588 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:17.352564    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942" event={"ID":"654161bd-9cf8-4f63-9a86-b312763e75cc","Type":"ContainerStarted","Data":"1f172dad62aacefaf343b3e23df93faaa6b45b4430a568d93053d8210fb265cc"}
Oct 30 13:22:17.357557 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:17.357430    4364 patch_prober.go:29] interesting pod/e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Oct 30 13:22:17.357557 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:17.357485    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz" podUID="52bf85ca-e35d-4a85-85f4-21d6d20a191d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Oct 30 13:22:17.568950 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:17.568380    4364 patch_prober.go:29] interesting pod/e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body=
Oct 30 13:22:17.568950 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:17.568441    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks" podUID="fb9e1069-efce-4243-b809-ad470e124bd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused"
Oct 30 13:22:17.671849 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:17.671812    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942"
Oct 30 13:22:17.673253 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:17.673228    4364 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Oct 30 13:22:17.673314 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:17.673284    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942" podUID="654161bd-9cf8-4f63-9a86-b312763e75cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Oct 30 13:22:17.960292 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:17.960262    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks"
Oct 30 13:22:18.360108 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:18.359886    4364 patch_prober.go:29] interesting pod/e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Oct 30 13:22:18.360108 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:18.359946    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz" podUID="52bf85ca-e35d-4a85-85f4-21d6d20a191d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Oct 30 13:22:18.476623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:18.472197    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k" event={"ID":"9bed178a-380c-456c-ac60-ed0020bc32f5","Type":"ContainerStarted","Data":"c0eaf1f1ede3e5d2e012b256f720c4b52985b3ffc4058fcb70a0e6ff72e18a3d"}
Oct 30 13:22:18.568006 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:18.567963    4364 patch_prober.go:29] interesting pod/e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body=
Oct 30 13:22:18.568293 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:18.568262    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks" podUID="fb9e1069-efce-4243-b809-ad470e124bd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused"
Oct 30 13:22:18.579624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:18.578020    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k" podStartSLOduration=2.577997499 podStartE2EDuration="2.577997499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:22:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:22:18.545285327 +0000 UTC m=+1837.711649976" watchObservedRunningTime="2026-10-30 13:22:18.577997499 +0000 UTC m=+1837.744362147"
Oct 30 13:22:18.672980 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:18.672941    4364 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Oct 30 13:22:18.673303 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:18.673272    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942" podUID="654161bd-9cf8-4f63-9a86-b312763e75cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Oct 30 13:22:18.734624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:18.733977    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942"
Oct 30 13:22:19.379625 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:19.370622    4364 patch_prober.go:29] interesting pod/e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Oct 30 13:22:19.379625 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:19.370692    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz" podUID="52bf85ca-e35d-4a85-85f4-21d6d20a191d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Oct 30 13:22:19.419622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:19.418662    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k"
Oct 30 13:22:19.442625 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:19.430876    4364 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 13:22:19.442625 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:19.430940    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k" podUID="9bed178a-380c-456c-ac60-ed0020bc32f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 13:22:19.568625 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:19.567994    4364 patch_prober.go:29] interesting pod/e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body=
Oct 30 13:22:19.568625 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:19.568071    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks" podUID="fb9e1069-efce-4243-b809-ad470e124bd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused"
Oct 30 13:22:19.692688 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:19.688488    4364 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Oct 30 13:22:19.692688 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:19.688553    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942" podUID="654161bd-9cf8-4f63-9a86-b312763e75cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Oct 30 13:22:20.236621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:20.235415    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74" podUID="4f2b06ac-9bf8-4b57-bd22-85532576f9c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded"
Oct 30 13:22:20.293638 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:20.290699    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz" podUID="a568f1ce-602d-4069-a41c-9fe1a7635c5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:22:20.317741 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:20.313720    4364 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr" podUID="446c12df-de12-4c7c-8c71-dfed93cc33f1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:22:20.358718 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:20.357001    4364 patch_prober.go:29] interesting pod/e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Oct 30 13:22:20.358718 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:20.357069    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz" podUID="52bf85ca-e35d-4a85-85f4-21d6d20a191d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Oct 30 13:22:20.433624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:20.432818    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz" podUID="bc5caf69-a649-47e5-8910-9b8f331dfd4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 13:22:20.434146 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:20.433816    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr" podUID="446c12df-de12-4c7c-8c71-dfed93cc33f1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded"
Oct 30 13:22:20.467626 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:20.438757    4364 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 13:22:20.467626 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:20.438813    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k" podUID="9bed178a-380c-456c-ac60-ed0020bc32f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 13:22:20.467626 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:20.358042    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4" podUID="0cbe6af7-d884-4613-90b8-192e5ffdb027" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:22:20.596686 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:20.595995    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-b5a369/test-model-b5a369-kserve-6bfb684495-gckcs" podUID="d52aab51-7fdf-4be6-8893-b5fe88dbe341" containerName="main" probeResult="failure" output="Get \"https://10.133.0.188:8000/ready\": context deadline exceeded"
Oct 30 13:22:20.621626 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:20.611668    4364 patch_prober.go:29] interesting pod/e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body=
Oct 30 13:22:20.621626 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:20.611743    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks" podUID="fb9e1069-efce-4243-b809-ad470e124bd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused"
Oct 30 13:22:20.686082 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:20.686034    4364 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Oct 30 13:22:20.686400 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:20.686366    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942" podUID="654161bd-9cf8-4f63-9a86-b312763e75cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Oct 30 13:22:20.889838 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:20.888684    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6" podUID="b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:22:21.167079 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:21.166947    4364 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-77254d/test-model-77254d-kserve-5476b887bf-2zjpk" podUID="e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.180:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:22:21.262623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:21.239688    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh" podUID="d41ac9e5-eeed-4e07-a7fa-05d3260630b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 30 13:22:21.262623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:21.241022    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-77254d/test-model-77254d-kserve-5476b887bf-2zjpk" podUID="e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.180:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:22:21.262623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:21.256698    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc" podUID="8911b790-331d-490c-876b-647e6d2fcd14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:22:21.291625 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:21.291159    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz" podUID="a568f1ce-602d-4069-a41c-9fe1a7635c5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:22:21.291625 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:21.291534    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz"
Oct 30 13:22:21.357624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:21.357408    4364 patch_prober.go:29] interesting pod/e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Oct 30 13:22:21.357624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:21.357470    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz" podUID="52bf85ca-e35d-4a85-85f4-21d6d20a191d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Oct 30 13:22:21.418685 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:21.418281    4364 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 13:22:21.418685 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:21.418357    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k" podUID="9bed178a-380c-456c-ac60-ed0020bc32f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 13:22:21.437635 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:21.434717    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr" podUID="446c12df-de12-4c7c-8c71-dfed93cc33f1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:22:21.437635 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:21.435282    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr"
Oct 30 13:22:21.542624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:21.484701    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz" podUID="bc5caf69-a649-47e5-8910-9b8f331dfd4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 13:22:21.542624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:21.485519    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz"
Oct 30 13:22:21.588624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:21.581710    4364 patch_prober.go:29] interesting pod/e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body=
Oct 30 13:22:21.588624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:21.581814    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks" podUID="fb9e1069-efce-4243-b809-ad470e124bd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused"
Oct 30 13:22:21.588624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:21.582335    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4" podUID="0cbe6af7-d884-4613-90b8-192e5ffdb027" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:22:21.588624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:21.582822    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4"
Oct 30 13:22:21.750687 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:21.614703    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-b5a369/test-model-b5a369-kserve-6bfb684495-gckcs" podUID="d52aab51-7fdf-4be6-8893-b5fe88dbe341" containerName="main" probeResult="failure" output="Get \"https://10.133.0.188:8000/ready\": context deadline exceeded"
Oct 30 13:22:21.750687 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:21.615967    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-b5a369/test-model-b5a369-kserve-6bfb684495-gckcs"
Oct 30 13:22:21.750687 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:21.716729    4364 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Oct 30 13:22:21.750687 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:21.716794    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942" podUID="654161bd-9cf8-4f63-9a86-b312763e75cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Oct 30 13:22:21.893625 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:21.891275    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6" podUID="b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:22:21.922652 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:21.911532    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6"
Oct 30 13:22:22.234624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:22.233700    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74" podUID="4f2b06ac-9bf8-4b57-bd22-85532576f9c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:22:22.259625 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:22.259230    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc" podUID="8911b790-331d-490c-876b-647e6d2fcd14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:22:22.276907 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:22.276845    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh" podUID="d41ac9e5-eeed-4e07-a7fa-05d3260630b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:22:22.278634 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:22.278555    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-77254d/test-model-77254d-kserve-5476b887bf-2zjpk" podUID="e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.180:8000/ready\": context deadline exceeded"
Oct 30 13:22:22.281078 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:22.280200    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc"
Oct 30 13:22:22.281078 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:22.280261    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh"
Oct 30 13:22:22.281078 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:22.280296    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-77254d/test-model-77254d-kserve-5476b887bf-2zjpk"
Oct 30 13:22:22.292589 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:22.291798    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz" podUID="a568f1ce-602d-4069-a41c-9fe1a7635c5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:22:22.378568 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:22.378530    4364 patch_prober.go:29] interesting pod/e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Oct 30 13:22:22.390913 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:22.390869    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz" podUID="52bf85ca-e35d-4a85-85f4-21d6d20a191d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Oct 30 13:22:22.460081 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:22.454029    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr" podUID="446c12df-de12-4c7c-8c71-dfed93cc33f1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:22:22.551870 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:22.551765    4364 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 13:22:22.659295 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:22.659249    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k" podUID="9bed178a-380c-456c-ac60-ed0020bc32f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 13:22:22.663816 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:22.663773    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz" podUID="bc5caf69-a649-47e5-8910-9b8f331dfd4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 13:22:22.721272 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:22.670263    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-b5a369/test-model-b5a369-kserve-6bfb684495-gckcs" podUID="d52aab51-7fdf-4be6-8893-b5fe88dbe341" containerName="main" probeResult="failure" output="Get \"https://10.133.0.188:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:22:22.721272 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:22.671569    4364 patch_prober.go:29] interesting pod/e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body=
Oct 30 13:22:22.721272 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:22.716032    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks" podUID="fb9e1069-efce-4243-b809-ad470e124bd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused"
Oct 30 13:22:22.721272 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:22.693359    4364 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Oct 30 13:22:22.721272 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:22.716094    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942" podUID="654161bd-9cf8-4f63-9a86-b312763e75cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Oct 30 13:22:22.721272 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:22.673028    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4" podUID="0cbe6af7-d884-4613-90b8-192e5ffdb027" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 13:22:22.809850 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:22.809744    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6"
Oct 30 13:22:22.847347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:22.847315    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh"
Oct 30 13:22:23.088930 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:23.088815    4364 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz" podUID="a568f1ce-602d-4069-a41c-9fe1a7635c5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:22:23.235516 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:23.235471    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74" podUID="4f2b06ac-9bf8-4b57-bd22-85532576f9c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:22:23.296763 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:23.296716    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz" podUID="a568f1ce-602d-4069-a41c-9fe1a7635c5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:22:23.455315 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:23.455277    4364 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 13:22:23.459309 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:23.458783    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k" podUID="9bed178a-380c-456c-ac60-ed0020bc32f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 13:22:23.461065 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:23.460905    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr" podUID="446c12df-de12-4c7c-8c71-dfed93cc33f1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:22:23.471732 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:23.466903    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc"
Oct 30 13:22:23.471732 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:23.466976    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d833bd/distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74"
Oct 30 13:22:23.471732 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:23.467013    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-77254d/test-model-77254d-kserve-5476b887bf-2zjpk"
Oct 30 13:22:23.471732 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:23.467047    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4"
Oct 30 13:22:23.471732 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:23.467080    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz"
Oct 30 13:22:23.471732 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:23.467124    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d833bd/distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74"
Oct 30 13:22:23.471732 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:23.467155    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-b5a369/test-model-b5a369-kserve-6bfb684495-gckcs"
Oct 30 13:22:23.471732 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:23.467187    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz"
Oct 30 13:22:23.475234 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:23.474845    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr"
Oct 30 13:22:23.575960 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:23.575920    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks"
Oct 30 13:22:23.577364 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:23.577339    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks"
Oct 30 13:22:23.672631 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:23.671749    4364 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Oct 30 13:22:23.672631 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:23.671798    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942" podUID="654161bd-9cf8-4f63-9a86-b312763e75cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Oct 30 13:22:23.704647 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:23.699533    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz"
Oct 30 13:22:23.704647 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:23.701225    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz"
Oct 30 13:22:23.737678 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:23.736945    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz"
Oct 30 13:22:24.349891 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:24.349857    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-31721c/test-model-31721c-kserve-64c44bfc-dmnrj"
Oct 30 13:22:24.364517 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:24.364482    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-31721c/test-model-31721c-kserve-64c44bfc-dmnrj"
Oct 30 13:22:24.418903 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:24.418868    4364 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 13:22:24.419196 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:24.419168    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k" podUID="9bed178a-380c-456c-ac60-ed0020bc32f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 13:22:24.672439 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:24.672401    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-31721c/test-model-31721c-kserve-64c44bfc-dmnrj"
Oct 30 13:22:24.672439 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:24.672418    4364 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Oct 30 13:22:24.672898 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:24.672458    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942" podUID="654161bd-9cf8-4f63-9a86-b312763e75cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Oct 30 13:22:25.416453 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:25.416409    4364 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 13:22:25.416671 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:25.416500    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k" podUID="9bed178a-380c-456c-ac60-ed0020bc32f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 13:22:25.677409 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:25.677336    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942"
Oct 30 13:22:25.678504 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:25.678480    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942"
Oct 30 13:22:26.420885 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:26.420855    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k"
Oct 30 13:22:26.421310 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:26.421284    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k"
Oct 30 13:22:26.421805 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:26.421786    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k"
Oct 30 13:22:41.507692 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:22:41.507645    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:22:42.653523 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:42.653488    4364 scope.go:122] "RemoveContainer" containerID="4715374acb1486834f3cc751a2079c3ff0a97bc45ac5aeae4394a5aa7e67b9d5"
Oct 30 13:22:43.120498 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:43.120467    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5c7ba3-574966bcc5-nvlwk"]
Oct 30 13:22:43.125218 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:43.125196    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5c7ba3-574966bcc5-nvlwk"
Oct 30 13:22:43.146396 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:43.146363    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5c7ba3-574966bcc5-nvlwk"]
Oct 30 13:22:43.152247 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:43.152220    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-5c7ba3-serving-cert\""
Oct 30 13:22:43.261478 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:43.261441    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b41649d1-4106-43bd-a16c-8c814f08de90-metrics-certs\") pod \"maas-api-e2e-inf-b-5c7ba3-574966bcc5-nvlwk\" (UID: \"b41649d1-4106-43bd-a16c-8c814f08de90\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5c7ba3-574966bcc5-nvlwk"
Oct 30 13:22:43.261812 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:43.261787    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcj97\" (UniqueName: \"kubernetes.io/projected/b41649d1-4106-43bd-a16c-8c814f08de90-kube-api-access-wcj97\") pod \"maas-api-e2e-inf-b-5c7ba3-574966bcc5-nvlwk\" (UID: \"b41649d1-4106-43bd-a16c-8c814f08de90\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5c7ba3-574966bcc5-nvlwk"
Oct 30 13:22:43.261940 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:43.261926    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b41649d1-4106-43bd-a16c-8c814f08de90-maas-api-tls\") pod \"maas-api-e2e-inf-b-5c7ba3-574966bcc5-nvlwk\" (UID: \"b41649d1-4106-43bd-a16c-8c814f08de90\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5c7ba3-574966bcc5-nvlwk"
Oct 30 13:22:43.261976 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:43.261961    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b41649d1-4106-43bd-a16c-8c814f08de90-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5c7ba3-574966bcc5-nvlwk\" (UID: \"b41649d1-4106-43bd-a16c-8c814f08de90\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5c7ba3-574966bcc5-nvlwk"
Oct 30 13:22:43.367156 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:43.367112    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcj97\" (UniqueName: \"kubernetes.io/projected/b41649d1-4106-43bd-a16c-8c814f08de90-kube-api-access-wcj97\") pod \"maas-api-e2e-inf-b-5c7ba3-574966bcc5-nvlwk\" (UID: \"b41649d1-4106-43bd-a16c-8c814f08de90\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5c7ba3-574966bcc5-nvlwk"
Oct 30 13:22:43.368097 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:43.368057    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b41649d1-4106-43bd-a16c-8c814f08de90-maas-api-tls\") pod \"maas-api-e2e-inf-b-5c7ba3-574966bcc5-nvlwk\" (UID: \"b41649d1-4106-43bd-a16c-8c814f08de90\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5c7ba3-574966bcc5-nvlwk"
Oct 30 13:22:43.368443 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:43.368419    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b41649d1-4106-43bd-a16c-8c814f08de90-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5c7ba3-574966bcc5-nvlwk\" (UID: \"b41649d1-4106-43bd-a16c-8c814f08de90\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5c7ba3-574966bcc5-nvlwk"
Oct 30 13:22:43.369928 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:22:43.368346    4364 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-5c7ba3-serving-cert: secret "maas-api-e2e-inf-b-5c7ba3-serving-cert" not found
Oct 30 13:22:43.370130 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:22:43.370118    4364 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b41649d1-4106-43bd-a16c-8c814f08de90-maas-api-tls podName:b41649d1-4106-43bd-a16c-8c814f08de90 nodeName:}" failed. No retries permitted until 2026-10-30 13:22:43.870096303 +0000 UTC m=+1863.036460933 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b41649d1-4106-43bd-a16c-8c814f08de90-maas-api-tls") pod "maas-api-e2e-inf-b-5c7ba3-574966bcc5-nvlwk" (UID: "b41649d1-4106-43bd-a16c-8c814f08de90") : secret "maas-api-e2e-inf-b-5c7ba3-serving-cert" not found
Oct 30 13:22:43.370642 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:43.369527    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b41649d1-4106-43bd-a16c-8c814f08de90-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5c7ba3-574966bcc5-nvlwk\" (UID: \"b41649d1-4106-43bd-a16c-8c814f08de90\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5c7ba3-574966bcc5-nvlwk"
Oct 30 13:22:43.370832 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:43.370814    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b41649d1-4106-43bd-a16c-8c814f08de90-metrics-certs\") pod \"maas-api-e2e-inf-b-5c7ba3-574966bcc5-nvlwk\" (UID: \"b41649d1-4106-43bd-a16c-8c814f08de90\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5c7ba3-574966bcc5-nvlwk"
Oct 30 13:22:43.409928 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:43.409851    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcj97\" (UniqueName: \"kubernetes.io/projected/b41649d1-4106-43bd-a16c-8c814f08de90-kube-api-access-wcj97\") pod \"maas-api-e2e-inf-b-5c7ba3-574966bcc5-nvlwk\" (UID: \"b41649d1-4106-43bd-a16c-8c814f08de90\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5c7ba3-574966bcc5-nvlwk"
Oct 30 13:22:43.413437 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:43.413401    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b41649d1-4106-43bd-a16c-8c814f08de90-metrics-certs\") pod \"maas-api-e2e-inf-b-5c7ba3-574966bcc5-nvlwk\" (UID: \"b41649d1-4106-43bd-a16c-8c814f08de90\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5c7ba3-574966bcc5-nvlwk"
Oct 30 13:22:43.880083 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:43.880050    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b41649d1-4106-43bd-a16c-8c814f08de90-maas-api-tls\") pod \"maas-api-e2e-inf-b-5c7ba3-574966bcc5-nvlwk\" (UID: \"b41649d1-4106-43bd-a16c-8c814f08de90\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5c7ba3-574966bcc5-nvlwk"
Oct 30 13:22:43.882665 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:43.882640    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b41649d1-4106-43bd-a16c-8c814f08de90-maas-api-tls\") pod \"maas-api-e2e-inf-b-5c7ba3-574966bcc5-nvlwk\" (UID: \"b41649d1-4106-43bd-a16c-8c814f08de90\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5c7ba3-574966bcc5-nvlwk"
Oct 30 13:22:44.041957 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:44.041922    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5c7ba3-574966bcc5-nvlwk"
Oct 30 13:22:44.368970 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:44.368938    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5c7ba3-574966bcc5-nvlwk"]
Oct 30 13:22:44.375126 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:22:44.375092    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb41649d1_4106_43bd_a16c_8c814f08de90.slice/crio-15e928870996b7aa055a84851cf16716bd260a51a83fa6cc9beed5edbd838fe6 WatchSource:0}: Error finding container 15e928870996b7aa055a84851cf16716bd260a51a83fa6cc9beed5edbd838fe6: Status 404 returned error can't find the container with id 15e928870996b7aa055a84851cf16716bd260a51a83fa6cc9beed5edbd838fe6
Oct 30 13:22:44.992782 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:44.992694    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5c7ba3-574966bcc5-nvlwk" event={"ID":"b41649d1-4106-43bd-a16c-8c814f08de90","Type":"ContainerStarted","Data":"0fdc9161720893fd043167244786622614b7d6239f6d9c8affda0a22049ee55e"}
Oct 30 13:22:44.992782 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:44.992745    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5c7ba3-574966bcc5-nvlwk" event={"ID":"b41649d1-4106-43bd-a16c-8c814f08de90","Type":"ContainerStarted","Data":"15e928870996b7aa055a84851cf16716bd260a51a83fa6cc9beed5edbd838fe6"}
Oct 30 13:22:44.993210 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:44.992808    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5c7ba3-574966bcc5-nvlwk"
Oct 30 13:22:45.038019 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:45.037951    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5c7ba3-574966bcc5-nvlwk" podStartSLOduration=1.701648332 podStartE2EDuration="2.037930584s" totalImagesPullingTime="336.282252ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:22:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:22:45.035178661 +0000 UTC m=+1864.201543310" watchObservedRunningTime="2026-10-30 13:22:45.037930584 +0000 UTC m=+1864.204295232"
Oct 30 13:22:51.008874 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:22:51.008846    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5c7ba3-574966bcc5-nvlwk"
Oct 30 13:23:09.955306 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:09.955270    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-0ad384/test-model-0ad384-kserve-6875d6fcff-9mwp9"]
Oct 30 13:23:09.960006 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:09.959980    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0ad384/test-model-0ad384-kserve-6875d6fcff-9mwp9"
Oct 30 13:23:09.983826 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:09.983789    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-0ad384/test-model-0ad384-kserve-6875d6fcff-9mwp9"]
Oct 30 13:23:09.986038 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:09.986008    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-0ad384\"/\"openshift-service-ca.crt\""
Oct 30 13:23:09.986038 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:09.986032    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-0ad384\"/\"test-model-0ad384-kserve-self-signed-certs\""
Oct 30 13:23:09.986287 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:09.986052    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-0ad384\"/\"kube-root-ca.crt\""
Oct 30 13:23:09.986695 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:09.986668    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-0ad384\"/\"default-dockercfg-wqcds\""
Oct 30 13:23:09.988840 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:09.988815    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6kf7\" (UniqueName: \"kubernetes.io/projected/517cf09e-dcec-4fe5-ad42-d7e15cd1e48a-kube-api-access-x6kf7\") pod \"test-model-0ad384-kserve-6875d6fcff-9mwp9\" (UID: \"517cf09e-dcec-4fe5-ad42-d7e15cd1e48a\") " pod="ai-tenant-e2e-inf-a-0ad384/test-model-0ad384-kserve-6875d6fcff-9mwp9"
Oct 30 13:23:09.989259 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:09.989232    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/517cf09e-dcec-4fe5-ad42-d7e15cd1e48a-tls-certs\") pod \"test-model-0ad384-kserve-6875d6fcff-9mwp9\" (UID: \"517cf09e-dcec-4fe5-ad42-d7e15cd1e48a\") " pod="ai-tenant-e2e-inf-a-0ad384/test-model-0ad384-kserve-6875d6fcff-9mwp9"
Oct 30 13:23:09.989412 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:09.989391    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/517cf09e-dcec-4fe5-ad42-d7e15cd1e48a-model-cache\") pod \"test-model-0ad384-kserve-6875d6fcff-9mwp9\" (UID: \"517cf09e-dcec-4fe5-ad42-d7e15cd1e48a\") " pod="ai-tenant-e2e-inf-a-0ad384/test-model-0ad384-kserve-6875d6fcff-9mwp9"
Oct 30 13:23:09.989525 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:09.989509    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/517cf09e-dcec-4fe5-ad42-d7e15cd1e48a-dshm\") pod \"test-model-0ad384-kserve-6875d6fcff-9mwp9\" (UID: \"517cf09e-dcec-4fe5-ad42-d7e15cd1e48a\") " pod="ai-tenant-e2e-inf-a-0ad384/test-model-0ad384-kserve-6875d6fcff-9mwp9"
Oct 30 13:23:09.989573 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:09.989559    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/517cf09e-dcec-4fe5-ad42-d7e15cd1e48a-tmp-dir\") pod \"test-model-0ad384-kserve-6875d6fcff-9mwp9\" (UID: \"517cf09e-dcec-4fe5-ad42-d7e15cd1e48a\") " pod="ai-tenant-e2e-inf-a-0ad384/test-model-0ad384-kserve-6875d6fcff-9mwp9"
Oct 30 13:23:09.989813 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:09.989790    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/517cf09e-dcec-4fe5-ad42-d7e15cd1e48a-home\") pod \"test-model-0ad384-kserve-6875d6fcff-9mwp9\" (UID: \"517cf09e-dcec-4fe5-ad42-d7e15cd1e48a\") " pod="ai-tenant-e2e-inf-a-0ad384/test-model-0ad384-kserve-6875d6fcff-9mwp9"
Oct 30 13:23:10.092267 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:10.092228    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6kf7\" (UniqueName: \"kubernetes.io/projected/517cf09e-dcec-4fe5-ad42-d7e15cd1e48a-kube-api-access-x6kf7\") pod \"test-model-0ad384-kserve-6875d6fcff-9mwp9\" (UID: \"517cf09e-dcec-4fe5-ad42-d7e15cd1e48a\") " pod="ai-tenant-e2e-inf-a-0ad384/test-model-0ad384-kserve-6875d6fcff-9mwp9"
Oct 30 13:23:10.092446 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:10.092433    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/517cf09e-dcec-4fe5-ad42-d7e15cd1e48a-tls-certs\") pod \"test-model-0ad384-kserve-6875d6fcff-9mwp9\" (UID: \"517cf09e-dcec-4fe5-ad42-d7e15cd1e48a\") " pod="ai-tenant-e2e-inf-a-0ad384/test-model-0ad384-kserve-6875d6fcff-9mwp9"
Oct 30 13:23:10.092518 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:10.092501    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/517cf09e-dcec-4fe5-ad42-d7e15cd1e48a-model-cache\") pod \"test-model-0ad384-kserve-6875d6fcff-9mwp9\" (UID: \"517cf09e-dcec-4fe5-ad42-d7e15cd1e48a\") " pod="ai-tenant-e2e-inf-a-0ad384/test-model-0ad384-kserve-6875d6fcff-9mwp9"
Oct 30 13:23:10.092578 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:10.092553    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/517cf09e-dcec-4fe5-ad42-d7e15cd1e48a-dshm\") pod \"test-model-0ad384-kserve-6875d6fcff-9mwp9\" (UID: \"517cf09e-dcec-4fe5-ad42-d7e15cd1e48a\") " pod="ai-tenant-e2e-inf-a-0ad384/test-model-0ad384-kserve-6875d6fcff-9mwp9"
Oct 30 13:23:10.092686 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:10.092615    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/517cf09e-dcec-4fe5-ad42-d7e15cd1e48a-tmp-dir\") pod \"test-model-0ad384-kserve-6875d6fcff-9mwp9\" (UID: \"517cf09e-dcec-4fe5-ad42-d7e15cd1e48a\") " pod="ai-tenant-e2e-inf-a-0ad384/test-model-0ad384-kserve-6875d6fcff-9mwp9"
Oct 30 13:23:10.092756 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:10.092698    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/517cf09e-dcec-4fe5-ad42-d7e15cd1e48a-home\") pod \"test-model-0ad384-kserve-6875d6fcff-9mwp9\" (UID: \"517cf09e-dcec-4fe5-ad42-d7e15cd1e48a\") " pod="ai-tenant-e2e-inf-a-0ad384/test-model-0ad384-kserve-6875d6fcff-9mwp9"
Oct 30 13:23:10.092953 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:10.092931    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/517cf09e-dcec-4fe5-ad42-d7e15cd1e48a-model-cache\") pod \"test-model-0ad384-kserve-6875d6fcff-9mwp9\" (UID: \"517cf09e-dcec-4fe5-ad42-d7e15cd1e48a\") " pod="ai-tenant-e2e-inf-a-0ad384/test-model-0ad384-kserve-6875d6fcff-9mwp9"
Oct 30 13:23:10.093094 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:10.093073    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/517cf09e-dcec-4fe5-ad42-d7e15cd1e48a-home\") pod \"test-model-0ad384-kserve-6875d6fcff-9mwp9\" (UID: \"517cf09e-dcec-4fe5-ad42-d7e15cd1e48a\") " pod="ai-tenant-e2e-inf-a-0ad384/test-model-0ad384-kserve-6875d6fcff-9mwp9"
Oct 30 13:23:10.093180 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:10.093158    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/517cf09e-dcec-4fe5-ad42-d7e15cd1e48a-tmp-dir\") pod \"test-model-0ad384-kserve-6875d6fcff-9mwp9\" (UID: \"517cf09e-dcec-4fe5-ad42-d7e15cd1e48a\") " pod="ai-tenant-e2e-inf-a-0ad384/test-model-0ad384-kserve-6875d6fcff-9mwp9"
Oct 30 13:23:10.095095 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:10.095064    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/517cf09e-dcec-4fe5-ad42-d7e15cd1e48a-dshm\") pod \"test-model-0ad384-kserve-6875d6fcff-9mwp9\" (UID: \"517cf09e-dcec-4fe5-ad42-d7e15cd1e48a\") " pod="ai-tenant-e2e-inf-a-0ad384/test-model-0ad384-kserve-6875d6fcff-9mwp9"
Oct 30 13:23:10.095338 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:10.095322    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/517cf09e-dcec-4fe5-ad42-d7e15cd1e48a-tls-certs\") pod \"test-model-0ad384-kserve-6875d6fcff-9mwp9\" (UID: \"517cf09e-dcec-4fe5-ad42-d7e15cd1e48a\") " pod="ai-tenant-e2e-inf-a-0ad384/test-model-0ad384-kserve-6875d6fcff-9mwp9"
Oct 30 13:23:10.117884 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:10.117857    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6kf7\" (UniqueName: \"kubernetes.io/projected/517cf09e-dcec-4fe5-ad42-d7e15cd1e48a-kube-api-access-x6kf7\") pod \"test-model-0ad384-kserve-6875d6fcff-9mwp9\" (UID: \"517cf09e-dcec-4fe5-ad42-d7e15cd1e48a\") " pod="ai-tenant-e2e-inf-a-0ad384/test-model-0ad384-kserve-6875d6fcff-9mwp9"
Oct 30 13:23:10.290673 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:10.285843    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0ad384/test-model-0ad384-kserve-6875d6fcff-9mwp9"
Oct 30 13:23:10.723493 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:10.723468    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-0ad384/test-model-0ad384-kserve-6875d6fcff-9mwp9"]
Oct 30 13:23:10.725546 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:23:10.725513    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod517cf09e_dcec_4fe5_ad42_d7e15cd1e48a.slice/crio-dd69532c6b35791f302c4c3b42ee153469eae4e0772926af6e6a8907253e36d6 WatchSource:0}: Error finding container dd69532c6b35791f302c4c3b42ee153469eae4e0772926af6e6a8907253e36d6: Status 404 returned error can't find the container with id dd69532c6b35791f302c4c3b42ee153469eae4e0772926af6e6a8907253e36d6
Oct 30 13:23:11.306810 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:11.306768    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0ad384/test-model-0ad384-kserve-6875d6fcff-9mwp9" event={"ID":"517cf09e-dcec-4fe5-ad42-d7e15cd1e48a","Type":"ContainerStarted","Data":"b49c78a98579e4a9234c02cda834e8bb197fca0c15650d31b923daf8b2f1be61"}
Oct 30 13:23:11.307226 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:11.306815    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0ad384/test-model-0ad384-kserve-6875d6fcff-9mwp9" event={"ID":"517cf09e-dcec-4fe5-ad42-d7e15cd1e48a","Type":"ContainerStarted","Data":"dd69532c6b35791f302c4c3b42ee153469eae4e0772926af6e6a8907253e36d6"}
Oct 30 13:23:11.350259 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:11.350208    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-0ad384/test-model-0ad384-kserve-6875d6fcff-9mwp9" podStartSLOduration=2.350194859 podStartE2EDuration="2.350194859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:23:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:23:11.349484856 +0000 UTC m=+1890.515849503" watchObservedRunningTime="2026-10-30 13:23:11.350194859 +0000 UTC m=+1890.516559507"
Oct 30 13:23:11.570839 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:23:11.570753    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:23:12.289187 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:12.289156    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-0ad384/test-model-0ad384-kserve-6875d6fcff-9mwp9"
Oct 30 13:23:19.943624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:19.941883    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:23:20.290495 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:20.290398    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-0ad384/test-model-0ad384-kserve-6875d6fcff-9mwp9"
Oct 30 13:23:20.346223 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:20.346191    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-0ad384/test-model-0ad384-kserve-6875d6fcff-9mwp9"
Oct 30 13:23:20.415763 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:20.415734    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-0ad384/test-model-0ad384-kserve-6875d6fcff-9mwp9"
Oct 30 13:23:22.704915 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:22.704872    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-09e4c0/test-model-09e4c0-kserve-56cdb9c5b5-nrjtk"]
Oct 30 13:23:22.780114 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:22.780076    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-09e4c0/test-model-09e4c0-kserve-56cdb9c5b5-nrjtk"]
Oct 30 13:23:22.780521 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:22.780454    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-09e4c0/test-model-09e4c0-kserve-56cdb9c5b5-nrjtk"
Oct 30 13:23:22.816321 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:22.816287    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-09e4c0\"/\"openshift-service-ca.crt\""
Oct 30 13:23:22.816670 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:22.816650    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-09e4c0\"/\"default-dockercfg-g2x6z\""
Oct 30 13:23:22.816912 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:22.816894    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-09e4c0\"/\"test-model-09e4c0-kserve-self-signed-certs\""
Oct 30 13:23:22.817133 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:22.817117    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-09e4c0\"/\"kube-root-ca.crt\""
Oct 30 13:23:22.948628 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:22.944940    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p5nz\" (UniqueName: \"kubernetes.io/projected/3fd28ec8-f8ba-400b-940a-9c6b5c516936-kube-api-access-9p5nz\") pod \"test-model-09e4c0-kserve-56cdb9c5b5-nrjtk\" (UID: \"3fd28ec8-f8ba-400b-940a-9c6b5c516936\") " pod="ai-tenant-e2e-inf-b-09e4c0/test-model-09e4c0-kserve-56cdb9c5b5-nrjtk"
Oct 30 13:23:22.948956 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:22.948927    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fd28ec8-f8ba-400b-940a-9c6b5c516936-model-cache\") pod \"test-model-09e4c0-kserve-56cdb9c5b5-nrjtk\" (UID: \"3fd28ec8-f8ba-400b-940a-9c6b5c516936\") " pod="ai-tenant-e2e-inf-b-09e4c0/test-model-09e4c0-kserve-56cdb9c5b5-nrjtk"
Oct 30 13:23:22.953538 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:22.953503    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fd28ec8-f8ba-400b-940a-9c6b5c516936-home\") pod \"test-model-09e4c0-kserve-56cdb9c5b5-nrjtk\" (UID: \"3fd28ec8-f8ba-400b-940a-9c6b5c516936\") " pod="ai-tenant-e2e-inf-b-09e4c0/test-model-09e4c0-kserve-56cdb9c5b5-nrjtk"
Oct 30 13:23:22.953909 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:22.953882    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fd28ec8-f8ba-400b-940a-9c6b5c516936-dshm\") pod \"test-model-09e4c0-kserve-56cdb9c5b5-nrjtk\" (UID: \"3fd28ec8-f8ba-400b-940a-9c6b5c516936\") " pod="ai-tenant-e2e-inf-b-09e4c0/test-model-09e4c0-kserve-56cdb9c5b5-nrjtk"
Oct 30 13:23:22.957771 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:22.957694    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fd28ec8-f8ba-400b-940a-9c6b5c516936-tls-certs\") pod \"test-model-09e4c0-kserve-56cdb9c5b5-nrjtk\" (UID: \"3fd28ec8-f8ba-400b-940a-9c6b5c516936\") " pod="ai-tenant-e2e-inf-b-09e4c0/test-model-09e4c0-kserve-56cdb9c5b5-nrjtk"
Oct 30 13:23:22.958032 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:22.958012    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fd28ec8-f8ba-400b-940a-9c6b5c516936-tmp-dir\") pod \"test-model-09e4c0-kserve-56cdb9c5b5-nrjtk\" (UID: \"3fd28ec8-f8ba-400b-940a-9c6b5c516936\") " pod="ai-tenant-e2e-inf-b-09e4c0/test-model-09e4c0-kserve-56cdb9c5b5-nrjtk"
Oct 30 13:23:23.089779 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:23.089734    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fd28ec8-f8ba-400b-940a-9c6b5c516936-tmp-dir\") pod \"test-model-09e4c0-kserve-56cdb9c5b5-nrjtk\" (UID: \"3fd28ec8-f8ba-400b-940a-9c6b5c516936\") " pod="ai-tenant-e2e-inf-b-09e4c0/test-model-09e4c0-kserve-56cdb9c5b5-nrjtk"
Oct 30 13:23:23.090327 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:23.090297    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p5nz\" (UniqueName: \"kubernetes.io/projected/3fd28ec8-f8ba-400b-940a-9c6b5c516936-kube-api-access-9p5nz\") pod \"test-model-09e4c0-kserve-56cdb9c5b5-nrjtk\" (UID: \"3fd28ec8-f8ba-400b-940a-9c6b5c516936\") " pod="ai-tenant-e2e-inf-b-09e4c0/test-model-09e4c0-kserve-56cdb9c5b5-nrjtk"
Oct 30 13:23:23.110622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:23.090496    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fd28ec8-f8ba-400b-940a-9c6b5c516936-model-cache\") pod \"test-model-09e4c0-kserve-56cdb9c5b5-nrjtk\" (UID: \"3fd28ec8-f8ba-400b-940a-9c6b5c516936\") " pod="ai-tenant-e2e-inf-b-09e4c0/test-model-09e4c0-kserve-56cdb9c5b5-nrjtk"
Oct 30 13:23:23.111497 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:23.111466    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fd28ec8-f8ba-400b-940a-9c6b5c516936-home\") pod \"test-model-09e4c0-kserve-56cdb9c5b5-nrjtk\" (UID: \"3fd28ec8-f8ba-400b-940a-9c6b5c516936\") " pod="ai-tenant-e2e-inf-b-09e4c0/test-model-09e4c0-kserve-56cdb9c5b5-nrjtk"
Oct 30 13:23:23.111888 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:23.111864    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fd28ec8-f8ba-400b-940a-9c6b5c516936-dshm\") pod \"test-model-09e4c0-kserve-56cdb9c5b5-nrjtk\" (UID: \"3fd28ec8-f8ba-400b-940a-9c6b5c516936\") " pod="ai-tenant-e2e-inf-b-09e4c0/test-model-09e4c0-kserve-56cdb9c5b5-nrjtk"
Oct 30 13:23:23.112095 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:23.112074    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fd28ec8-f8ba-400b-940a-9c6b5c516936-tls-certs\") pod \"test-model-09e4c0-kserve-56cdb9c5b5-nrjtk\" (UID: \"3fd28ec8-f8ba-400b-940a-9c6b5c516936\") " pod="ai-tenant-e2e-inf-b-09e4c0/test-model-09e4c0-kserve-56cdb9c5b5-nrjtk"
Oct 30 13:23:23.120773 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:23.105426    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fd28ec8-f8ba-400b-940a-9c6b5c516936-tmp-dir\") pod \"test-model-09e4c0-kserve-56cdb9c5b5-nrjtk\" (UID: \"3fd28ec8-f8ba-400b-940a-9c6b5c516936\") " pod="ai-tenant-e2e-inf-b-09e4c0/test-model-09e4c0-kserve-56cdb9c5b5-nrjtk"
Oct 30 13:23:23.120773 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:23.118330    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fd28ec8-f8ba-400b-940a-9c6b5c516936-model-cache\") pod \"test-model-09e4c0-kserve-56cdb9c5b5-nrjtk\" (UID: \"3fd28ec8-f8ba-400b-940a-9c6b5c516936\") " pod="ai-tenant-e2e-inf-b-09e4c0/test-model-09e4c0-kserve-56cdb9c5b5-nrjtk"
Oct 30 13:23:23.120773 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:23.118575    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fd28ec8-f8ba-400b-940a-9c6b5c516936-home\") pod \"test-model-09e4c0-kserve-56cdb9c5b5-nrjtk\" (UID: \"3fd28ec8-f8ba-400b-940a-9c6b5c516936\") " pod="ai-tenant-e2e-inf-b-09e4c0/test-model-09e4c0-kserve-56cdb9c5b5-nrjtk"
Oct 30 13:23:23.189065 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:23.161576    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fd28ec8-f8ba-400b-940a-9c6b5c516936-tls-certs\") pod \"test-model-09e4c0-kserve-56cdb9c5b5-nrjtk\" (UID: \"3fd28ec8-f8ba-400b-940a-9c6b5c516936\") " pod="ai-tenant-e2e-inf-b-09e4c0/test-model-09e4c0-kserve-56cdb9c5b5-nrjtk"
Oct 30 13:23:23.290549 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:23.290466    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fd28ec8-f8ba-400b-940a-9c6b5c516936-dshm\") pod \"test-model-09e4c0-kserve-56cdb9c5b5-nrjtk\" (UID: \"3fd28ec8-f8ba-400b-940a-9c6b5c516936\") " pod="ai-tenant-e2e-inf-b-09e4c0/test-model-09e4c0-kserve-56cdb9c5b5-nrjtk"
Oct 30 13:23:23.346374 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:23.346305    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p5nz\" (UniqueName: \"kubernetes.io/projected/3fd28ec8-f8ba-400b-940a-9c6b5c516936-kube-api-access-9p5nz\") pod \"test-model-09e4c0-kserve-56cdb9c5b5-nrjtk\" (UID: \"3fd28ec8-f8ba-400b-940a-9c6b5c516936\") " pod="ai-tenant-e2e-inf-b-09e4c0/test-model-09e4c0-kserve-56cdb9c5b5-nrjtk"
Oct 30 13:23:23.537620 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:23.535975    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-09e4c0/test-model-09e4c0-kserve-56cdb9c5b5-nrjtk"
Oct 30 13:23:23.932771 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:23:23.932732    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3fd28ec8_f8ba_400b_940a_9c6b5c516936.slice/crio-02189c6407c7d7affa92c2172e8b44b042c4b5a2c99224f2f2a02907afee90d7 WatchSource:0}: Error finding container 02189c6407c7d7affa92c2172e8b44b042c4b5a2c99224f2f2a02907afee90d7: Status 404 returned error can't find the container with id 02189c6407c7d7affa92c2172e8b44b042c4b5a2c99224f2f2a02907afee90d7
Oct 30 13:23:23.933115 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:23.933071    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-09e4c0/test-model-09e4c0-kserve-56cdb9c5b5-nrjtk"]
Oct 30 13:23:24.728072 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:24.728036    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-09e4c0/test-model-09e4c0-kserve-56cdb9c5b5-nrjtk" event={"ID":"3fd28ec8-f8ba-400b-940a-9c6b5c516936","Type":"ContainerStarted","Data":"d08f515d537a59787af0dfcbcf5c8a7c14fe7410b9f270e28c68edba48817f25"}
Oct 30 13:23:24.728283 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:24.728088    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-09e4c0/test-model-09e4c0-kserve-56cdb9c5b5-nrjtk" event={"ID":"3fd28ec8-f8ba-400b-940a-9c6b5c516936","Type":"ContainerStarted","Data":"02189c6407c7d7affa92c2172e8b44b042c4b5a2c99224f2f2a02907afee90d7"}
Oct 30 13:23:24.773269 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:24.773216    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-09e4c0/test-model-09e4c0-kserve-56cdb9c5b5-nrjtk" podStartSLOduration=2.77320299 podStartE2EDuration="2.77320299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:23:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:23:24.769174454 +0000 UTC m=+1903.935539102" watchObservedRunningTime="2026-10-30 13:23:24.77320299 +0000 UTC m=+1903.939567638"
Oct 30 13:23:25.537334 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:25.537298    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-09e4c0/test-model-09e4c0-kserve-56cdb9c5b5-nrjtk"
Oct 30 13:23:33.537324 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:33.537282    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-09e4c0/test-model-09e4c0-kserve-56cdb9c5b5-nrjtk"
Oct 30 13:23:33.550085 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:33.550061    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-09e4c0/test-model-09e4c0-kserve-56cdb9c5b5-nrjtk"
Oct 30 13:23:33.817508 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:33.817411    4364 generic.go:356] "Generic (PLEG): container finished" podID="f9846347-72dc-472f-8bb1-57d39f1be48b" containerID="ed645de8c0778c120d3496bc0a7411e7104faa949fcc3b120a03bf060d83e434" exitCode=0
Oct 30 13:23:33.817508 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:33.817484    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6" event={"ID":"f9846347-72dc-472f-8bb1-57d39f1be48b","Type":"ContainerDied","Data":"ed645de8c0778c120d3496bc0a7411e7104faa949fcc3b120a03bf060d83e434"}
Oct 30 13:23:33.818098 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:33.818076    4364 scope.go:122] "RemoveContainer" containerID="ed645de8c0778c120d3496bc0a7411e7104faa949fcc3b120a03bf060d83e434"
Oct 30 13:23:33.820232 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:33.820200    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:23:33.820351 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:33.820289    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:23:33.820427 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:33.820364    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:23:33.831574 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:33.831552    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-09e4c0/test-model-09e4c0-kserve-56cdb9c5b5-nrjtk"
Oct 30 13:23:34.829173 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:34.829123    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6" event={"ID":"f9846347-72dc-472f-8bb1-57d39f1be48b","Type":"ContainerStarted","Data":"94816d09f019e13ca2216dfbd2b861a9fd1015f8deccabe1aaff68663dfb0341"}
Oct 30 13:23:35.084385 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:35.084295    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6"
Oct 30 13:23:35.089591 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:35.089545    4364 patch_prober.go:29] interesting pod/e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Oct 30 13:23:35.090000 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:35.089959    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6" podUID="f9846347-72dc-472f-8bb1-57d39f1be48b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Oct 30 13:23:36.097624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:36.086986    4364 patch_prober.go:29] interesting pod/e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Oct 30 13:23:36.097624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:36.087051    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6" podUID="f9846347-72dc-472f-8bb1-57d39f1be48b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Oct 30 13:23:37.099624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:37.084711    4364 patch_prober.go:29] interesting pod/e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Oct 30 13:23:37.099624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:37.084780    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6" podUID="f9846347-72dc-472f-8bb1-57d39f1be48b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Oct 30 13:23:37.155303 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:37.155267    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6"
Oct 30 13:23:38.093624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:38.087038    4364 patch_prober.go:29] interesting pod/e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Oct 30 13:23:38.093624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:38.087103    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6" podUID="f9846347-72dc-472f-8bb1-57d39f1be48b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Oct 30 13:23:39.089057 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:39.089016    4364 patch_prober.go:29] interesting pod/e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Oct 30 13:23:39.089801 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:39.089768    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6" podUID="f9846347-72dc-472f-8bb1-57d39f1be48b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Oct 30 13:23:40.094731 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:40.090078    4364 patch_prober.go:29] interesting pod/e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Oct 30 13:23:40.094731 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:40.090167    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6" podUID="f9846347-72dc-472f-8bb1-57d39f1be48b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Oct 30 13:23:41.088618 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:41.088577    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6"
Oct 30 13:23:41.089485 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:23:41.089463    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6"
Oct 30 13:23:41.508864 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:23:41.508827    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:24:01.720116 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:01.720029    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-77254d/test-model-77254d-kserve-5476b887bf-2zjpk"]
Oct 30 13:24:11.517529 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:24:11.517482    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:24:14.913136 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:14.913095    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:24:16.755569 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:16.755524    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-77254d/test-model-77254d-kserve-5476b887bf-2zjpk" podUID="e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a" containerName="main" containerID="cri-o://dbddda331f04a98ef97bd01673922d0add5831dd85f39411a913f537deaee3e7" gracePeriod=45
Oct 30 13:24:17.243083 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:17.240750    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74" podUID="4f2b06ac-9bf8-4b57-bd22-85532576f9c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:24:17.243083 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:17.240782    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-77254d/test-model-77254d-kserve-5476b887bf-2zjpk" podUID="e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.180:8000/ready\": EOF"
Oct 30 13:24:17.255631 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:17.255404    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-77254d/test-model-77254d-kserve-5476b887bf-2zjpk" podUID="e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.180:8000/ready\": dial tcp 10.133.0.180:8000: connect: connection refused"
Oct 30 13:24:17.255631 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:17.255522    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-77254d/test-model-77254d-kserve-5476b887bf-2zjpk"
Oct 30 13:24:17.314769 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:17.302801    4364 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh" podUID="d41ac9e5-eeed-4e07-a7fa-05d3260630b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:24:17.314769 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:17.302913    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4" podUID="0cbe6af7-d884-4613-90b8-192e5ffdb027" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 13:24:17.314769 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:17.304761    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz" podUID="bc5caf69-a649-47e5-8910-9b8f331dfd4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 13:24:17.399630 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:17.378762    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-b5a369/test-model-b5a369-kserve-6bfb684495-gckcs" podUID="d52aab51-7fdf-4be6-8893-b5fe88dbe341" containerName="main" probeResult="failure" output="Get \"https://10.133.0.188:8000/ready\": context deadline exceeded"
Oct 30 13:24:17.496825 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:17.496725    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr" podUID="446c12df-de12-4c7c-8c71-dfed93cc33f1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:24:17.497368 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:17.497330    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-0ad384/test-model-0ad384-kserve-6875d6fcff-9mwp9" podUID="517cf09e-dcec-4fe5-ad42-d7e15cd1e48a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.195:8000/ready\": context deadline exceeded"
Oct 30 13:24:17.663226 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:17.663194    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-31721c/test-model-31721c-kserve-64c44bfc-dmnrj" podUID="ac457ffd-caeb-419f-b448-1b21942ead0f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.192:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:24:18.275624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:18.270065    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-77254d/test-model-77254d-kserve-5476b887bf-2zjpk" podUID="e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.180:8000/ready\": dial tcp 10.133.0.180:8000: connect: connection refused"
Oct 30 13:24:18.596862 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:18.596832    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-77254d/test-model-77254d-kserve-5476b887bf-2zjpk"
Oct 30 13:24:18.627068 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:18.627035    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a-tls-certs\" (UniqueName: \"kubernetes.io/secret/e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a-tls-certs\") pod \"e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a\" (UID: \"e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a\") "
Oct 30 13:24:18.627260 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:18.627095    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a-home\" (UniqueName: \"kubernetes.io/empty-dir/e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a-home\") pod \"e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a\" (UID: \"e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a\") "
Oct 30 13:24:18.627340 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:18.627277    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a-tmp-dir\") pod \"e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a\" (UID: \"e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a\") "
Oct 30 13:24:18.627442 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:18.627414    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a-home" pod "e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a" (UID: "e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:18.627505 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:18.627473    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a-kube-api-access-64hx5\" (UniqueName: \"kubernetes.io/projected/e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a-kube-api-access-64hx5\") pod \"e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a\" (UID: \"e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a\") "
Oct 30 13:24:18.627548 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:18.627530    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a-dshm\") pod \"e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a\" (UID: \"e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a\") "
Oct 30 13:24:18.627593 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:18.627555    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a-tmp-dir" pod "e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a" (UID: "e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:18.627681 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:18.627658    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a-model-cache\") pod \"e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a\" (UID: \"e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a\") "
Oct 30 13:24:18.627923 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:18.627901    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a-model-cache" pod "e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a" (UID: "e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:18.628466 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:18.628447    4364 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a-home\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:24:18.628563 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:18.628471    4364 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a-tmp-dir\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:24:18.628563 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:18.628485    4364 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a-model-cache\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:24:18.629772 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:18.629748    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a-tls-certs" pod "e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a" (UID: "e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:24:18.629879 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:18.629788    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a-kube-api-access-64hx5" pod "e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a" (UID: "e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a"). InnerVolumeSpecName "kube-api-access-64hx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:24:18.629879 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:18.629873    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a-dshm" pod "e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a" (UID: "e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:18.705835 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:18.705802    4364 generic.go:356] "Generic (PLEG): container finished" podID="e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a" containerID="dbddda331f04a98ef97bd01673922d0add5831dd85f39411a913f537deaee3e7" exitCode=0
Oct 30 13:24:18.706016 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:18.705853    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-77254d/test-model-77254d-kserve-5476b887bf-2zjpk"
Oct 30 13:24:18.706016 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:18.705884    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-77254d/test-model-77254d-kserve-5476b887bf-2zjpk" event={"ID":"e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a","Type":"ContainerDied","Data":"dbddda331f04a98ef97bd01673922d0add5831dd85f39411a913f537deaee3e7"}
Oct 30 13:24:18.706016 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:18.705930    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-77254d/test-model-77254d-kserve-5476b887bf-2zjpk" event={"ID":"e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a","Type":"ContainerDied","Data":"cdb3a0b33c5d6c79e303db1d51ce7c792f796db1350a39712f29eb35522bddaa"}
Oct 30 13:24:18.706016 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:18.705952    4364 scope.go:122] "RemoveContainer" containerID="dbddda331f04a98ef97bd01673922d0add5831dd85f39411a913f537deaee3e7"
Oct 30 13:24:18.718778 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:18.718763    4364 scope.go:122] "RemoveContainer" containerID="dbddda331f04a98ef97bd01673922d0add5831dd85f39411a913f537deaee3e7"
Oct 30 13:24:18.719103 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:24:18.719044    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbddda331f04a98ef97bd01673922d0add5831dd85f39411a913f537deaee3e7\": container with ID starting with dbddda331f04a98ef97bd01673922d0add5831dd85f39411a913f537deaee3e7 not found: ID does not exist" containerID="dbddda331f04a98ef97bd01673922d0add5831dd85f39411a913f537deaee3e7"
Oct 30 13:24:18.719103 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:18.719076    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbddda331f04a98ef97bd01673922d0add5831dd85f39411a913f537deaee3e7"} err="failed to get container status \"dbddda331f04a98ef97bd01673922d0add5831dd85f39411a913f537deaee3e7\": rpc error: code = NotFound desc = could not find container \"dbddda331f04a98ef97bd01673922d0add5831dd85f39411a913f537deaee3e7\": container with ID starting with dbddda331f04a98ef97bd01673922d0add5831dd85f39411a913f537deaee3e7 not found: ID does not exist"
Oct 30 13:24:18.730333 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:18.730310    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64hx5\" (UniqueName: \"kubernetes.io/projected/e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a-kube-api-access-64hx5\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:24:18.730333 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:18.730333    4364 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a-dshm\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:24:18.730481 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:18.730342    4364 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a-tls-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:24:18.760381 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:18.760360    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-77254d/test-model-77254d-kserve-5476b887bf-2zjpk"]
Oct 30 13:24:18.765256 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:18.765232    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-77254d/test-model-77254d-kserve-5476b887bf-2zjpk"]
Oct 30 13:24:18.973855 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:18.973822    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a" path="/var/lib/kubelet/pods/e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a/volumes"
Oct 30 13:24:22.104846 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:22.104812    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-4651e8/test-model-4651e8-kserve-5c7fcb7f7-8n9zr"]
Oct 30 13:24:22.105761 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:22.105713    4364 cpu_manager.go:414] "containerMap: removing container" podUID="e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a" containerName="main"
Oct 30 13:24:22.105761 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:22.105758    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a" containerName="main"
Oct 30 13:24:22.105997 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:22.105978    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3b39cfd-9e3a-4b05-bb20-451a89f5ea6a" containerName="main"
Oct 30 13:24:22.109646 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:22.109624    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4651e8/test-model-4651e8-kserve-5c7fcb7f7-8n9zr"
Oct 30 13:24:22.135939 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:22.135905    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-4651e8/test-model-4651e8-kserve-5c7fcb7f7-8n9zr"]
Oct 30 13:24:22.136263 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:22.136241    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-4651e8\"/\"kube-root-ca.crt\""
Oct 30 13:24:22.136406 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:22.136332    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-4651e8\"/\"default-dockercfg-5nx94\""
Oct 30 13:24:22.136406 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:22.136244    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-4651e8\"/\"openshift-service-ca.crt\""
Oct 30 13:24:22.140463 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:22.140431    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-4651e8\"/\"test-model-4651e8-kserve-self-signed-certs\""
Oct 30 13:24:22.199583 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:22.199547    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff5899b6-dd04-481f-bf5d-d4ce4e1f981d-home\") pod \"test-model-4651e8-kserve-5c7fcb7f7-8n9zr\" (UID: \"ff5899b6-dd04-481f-bf5d-d4ce4e1f981d\") " pod="ai-tenant-e2e-inf-a-4651e8/test-model-4651e8-kserve-5c7fcb7f7-8n9zr"
Oct 30 13:24:22.199794 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:22.199773    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbfzj\" (UniqueName: \"kubernetes.io/projected/ff5899b6-dd04-481f-bf5d-d4ce4e1f981d-kube-api-access-gbfzj\") pod \"test-model-4651e8-kserve-5c7fcb7f7-8n9zr\" (UID: \"ff5899b6-dd04-481f-bf5d-d4ce4e1f981d\") " pod="ai-tenant-e2e-inf-a-4651e8/test-model-4651e8-kserve-5c7fcb7f7-8n9zr"
Oct 30 13:24:22.199855 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:22.199802    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff5899b6-dd04-481f-bf5d-d4ce4e1f981d-tls-certs\") pod \"test-model-4651e8-kserve-5c7fcb7f7-8n9zr\" (UID: \"ff5899b6-dd04-481f-bf5d-d4ce4e1f981d\") " pod="ai-tenant-e2e-inf-a-4651e8/test-model-4651e8-kserve-5c7fcb7f7-8n9zr"
Oct 30 13:24:22.199957 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:22.199945    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff5899b6-dd04-481f-bf5d-d4ce4e1f981d-tmp-dir\") pod \"test-model-4651e8-kserve-5c7fcb7f7-8n9zr\" (UID: \"ff5899b6-dd04-481f-bf5d-d4ce4e1f981d\") " pod="ai-tenant-e2e-inf-a-4651e8/test-model-4651e8-kserve-5c7fcb7f7-8n9zr"
Oct 30 13:24:22.200001 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:22.199991    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff5899b6-dd04-481f-bf5d-d4ce4e1f981d-dshm\") pod \"test-model-4651e8-kserve-5c7fcb7f7-8n9zr\" (UID: \"ff5899b6-dd04-481f-bf5d-d4ce4e1f981d\") " pod="ai-tenant-e2e-inf-a-4651e8/test-model-4651e8-kserve-5c7fcb7f7-8n9zr"
Oct 30 13:24:22.200049 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:22.200016    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff5899b6-dd04-481f-bf5d-d4ce4e1f981d-model-cache\") pod \"test-model-4651e8-kserve-5c7fcb7f7-8n9zr\" (UID: \"ff5899b6-dd04-481f-bf5d-d4ce4e1f981d\") " pod="ai-tenant-e2e-inf-a-4651e8/test-model-4651e8-kserve-5c7fcb7f7-8n9zr"
Oct 30 13:24:22.301615 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:22.301572    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbfzj\" (UniqueName: \"kubernetes.io/projected/ff5899b6-dd04-481f-bf5d-d4ce4e1f981d-kube-api-access-gbfzj\") pod \"test-model-4651e8-kserve-5c7fcb7f7-8n9zr\" (UID: \"ff5899b6-dd04-481f-bf5d-d4ce4e1f981d\") " pod="ai-tenant-e2e-inf-a-4651e8/test-model-4651e8-kserve-5c7fcb7f7-8n9zr"
Oct 30 13:24:22.301789 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:22.301635    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff5899b6-dd04-481f-bf5d-d4ce4e1f981d-tls-certs\") pod \"test-model-4651e8-kserve-5c7fcb7f7-8n9zr\" (UID: \"ff5899b6-dd04-481f-bf5d-d4ce4e1f981d\") " pod="ai-tenant-e2e-inf-a-4651e8/test-model-4651e8-kserve-5c7fcb7f7-8n9zr"
Oct 30 13:24:22.301871 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:22.301804    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff5899b6-dd04-481f-bf5d-d4ce4e1f981d-tmp-dir\") pod \"test-model-4651e8-kserve-5c7fcb7f7-8n9zr\" (UID: \"ff5899b6-dd04-481f-bf5d-d4ce4e1f981d\") " pod="ai-tenant-e2e-inf-a-4651e8/test-model-4651e8-kserve-5c7fcb7f7-8n9zr"
Oct 30 13:24:22.301939 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:22.301886    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff5899b6-dd04-481f-bf5d-d4ce4e1f981d-dshm\") pod \"test-model-4651e8-kserve-5c7fcb7f7-8n9zr\" (UID: \"ff5899b6-dd04-481f-bf5d-d4ce4e1f981d\") " pod="ai-tenant-e2e-inf-a-4651e8/test-model-4651e8-kserve-5c7fcb7f7-8n9zr"
Oct 30 13:24:22.301939 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:22.301932    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff5899b6-dd04-481f-bf5d-d4ce4e1f981d-model-cache\") pod \"test-model-4651e8-kserve-5c7fcb7f7-8n9zr\" (UID: \"ff5899b6-dd04-481f-bf5d-d4ce4e1f981d\") " pod="ai-tenant-e2e-inf-a-4651e8/test-model-4651e8-kserve-5c7fcb7f7-8n9zr"
Oct 30 13:24:22.302049 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:22.302034    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff5899b6-dd04-481f-bf5d-d4ce4e1f981d-home\") pod \"test-model-4651e8-kserve-5c7fcb7f7-8n9zr\" (UID: \"ff5899b6-dd04-481f-bf5d-d4ce4e1f981d\") " pod="ai-tenant-e2e-inf-a-4651e8/test-model-4651e8-kserve-5c7fcb7f7-8n9zr"
Oct 30 13:24:22.302288 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:22.302262    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff5899b6-dd04-481f-bf5d-d4ce4e1f981d-tmp-dir\") pod \"test-model-4651e8-kserve-5c7fcb7f7-8n9zr\" (UID: \"ff5899b6-dd04-481f-bf5d-d4ce4e1f981d\") " pod="ai-tenant-e2e-inf-a-4651e8/test-model-4651e8-kserve-5c7fcb7f7-8n9zr"
Oct 30 13:24:22.302436 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:22.302318    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff5899b6-dd04-481f-bf5d-d4ce4e1f981d-model-cache\") pod \"test-model-4651e8-kserve-5c7fcb7f7-8n9zr\" (UID: \"ff5899b6-dd04-481f-bf5d-d4ce4e1f981d\") " pod="ai-tenant-e2e-inf-a-4651e8/test-model-4651e8-kserve-5c7fcb7f7-8n9zr"
Oct 30 13:24:22.302436 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:22.302385    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff5899b6-dd04-481f-bf5d-d4ce4e1f981d-home\") pod \"test-model-4651e8-kserve-5c7fcb7f7-8n9zr\" (UID: \"ff5899b6-dd04-481f-bf5d-d4ce4e1f981d\") " pod="ai-tenant-e2e-inf-a-4651e8/test-model-4651e8-kserve-5c7fcb7f7-8n9zr"
Oct 30 13:24:22.304158 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:22.304138    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff5899b6-dd04-481f-bf5d-d4ce4e1f981d-dshm\") pod \"test-model-4651e8-kserve-5c7fcb7f7-8n9zr\" (UID: \"ff5899b6-dd04-481f-bf5d-d4ce4e1f981d\") " pod="ai-tenant-e2e-inf-a-4651e8/test-model-4651e8-kserve-5c7fcb7f7-8n9zr"
Oct 30 13:24:22.304338 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:22.304319    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff5899b6-dd04-481f-bf5d-d4ce4e1f981d-tls-certs\") pod \"test-model-4651e8-kserve-5c7fcb7f7-8n9zr\" (UID: \"ff5899b6-dd04-481f-bf5d-d4ce4e1f981d\") " pod="ai-tenant-e2e-inf-a-4651e8/test-model-4651e8-kserve-5c7fcb7f7-8n9zr"
Oct 30 13:24:22.324334 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:22.324308    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbfzj\" (UniqueName: \"kubernetes.io/projected/ff5899b6-dd04-481f-bf5d-d4ce4e1f981d-kube-api-access-gbfzj\") pod \"test-model-4651e8-kserve-5c7fcb7f7-8n9zr\" (UID: \"ff5899b6-dd04-481f-bf5d-d4ce4e1f981d\") " pod="ai-tenant-e2e-inf-a-4651e8/test-model-4651e8-kserve-5c7fcb7f7-8n9zr"
Oct 30 13:24:22.432743 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:22.431569    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4651e8/test-model-4651e8-kserve-5c7fcb7f7-8n9zr"
Oct 30 13:24:22.900224 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:22.900194    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-4651e8/test-model-4651e8-kserve-5c7fcb7f7-8n9zr"]
Oct 30 13:24:22.900522 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:24:22.900498    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podff5899b6_dd04_481f_bf5d_d4ce4e1f981d.slice/crio-0bc5b674f267270180b595dcd39c0dc868879e4c4eb7296d66df703e44ee0230 WatchSource:0}: Error finding container 0bc5b674f267270180b595dcd39c0dc868879e4c4eb7296d66df703e44ee0230: Status 404 returned error can't find the container with id 0bc5b674f267270180b595dcd39c0dc868879e4c4eb7296d66df703e44ee0230
Oct 30 13:24:23.783203 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:23.783156    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4651e8/test-model-4651e8-kserve-5c7fcb7f7-8n9zr" event={"ID":"ff5899b6-dd04-481f-bf5d-d4ce4e1f981d","Type":"ContainerStarted","Data":"25c4b6cb4be2601706f0fd9eb0d6ee888769d027159f33f5fcf4e97628b6b4a3"}
Oct 30 13:24:23.783578 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:23.783212    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4651e8/test-model-4651e8-kserve-5c7fcb7f7-8n9zr" event={"ID":"ff5899b6-dd04-481f-bf5d-d4ce4e1f981d","Type":"ContainerStarted","Data":"0bc5b674f267270180b595dcd39c0dc868879e4c4eb7296d66df703e44ee0230"}
Oct 30 13:24:23.830941 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:23.830895    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-4651e8/test-model-4651e8-kserve-5c7fcb7f7-8n9zr" podStartSLOduration=1.830877429 podStartE2EDuration="1.830877429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:24:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:24:23.824123391 +0000 UTC m=+1962.990488040" watchObservedRunningTime="2026-10-30 13:24:23.830877429 +0000 UTC m=+1962.997242079"
Oct 30 13:24:24.432458 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:24.432426    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-4651e8/test-model-4651e8-kserve-5c7fcb7f7-8n9zr"
Oct 30 13:24:32.432658 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:32.432627    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-4651e8/test-model-4651e8-kserve-5c7fcb7f7-8n9zr"
Oct 30 13:24:32.445437 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:32.445412    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-4651e8/test-model-4651e8-kserve-5c7fcb7f7-8n9zr"
Oct 30 13:24:32.889099 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:32.889065    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-4651e8/test-model-4651e8-kserve-5c7fcb7f7-8n9zr"
Oct 30 13:24:41.507704 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:24:41.507664    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:24:52.803622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:52.801354    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6" podUID="b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:24:52.885623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:52.861727    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-09e4c0/test-model-09e4c0-kserve-56cdb9c5b5-nrjtk" podUID="3fd28ec8-f8ba-400b-940a-9c6b5c516936" containerName="main" probeResult="failure" output="Get \"https://10.133.0.196:8000/ready\": context deadline exceeded"
Oct 30 13:24:52.885623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:52.862698    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh" podUID="d41ac9e5-eeed-4e07-a7fa-05d3260630b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 30 13:24:52.886010 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:52.885971    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-4651e8/test-model-4651e8-kserve-5c7fcb7f7-8n9zr" podUID="ff5899b6-dd04-481f-bf5d-d4ce4e1f981d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.197:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:24:53.102686 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:53.100013    4364 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz" podUID="a568f1ce-602d-4069-a41c-9fe1a7635c5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:24:53.184622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:53.173785    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc" podUID="8911b790-331d-490c-876b-647e6d2fcd14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:24:53.236474 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:53.236425    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74" podUID="4f2b06ac-9bf8-4b57-bd22-85532576f9c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:24:53.282724 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:53.282680    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4" podUID="0cbe6af7-d884-4613-90b8-192e5ffdb027" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:24:53.284062 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:53.284026    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz" podUID="bc5caf69-a649-47e5-8910-9b8f331dfd4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 13:24:53.299619 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:53.294002    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz" podUID="a568f1ce-602d-4069-a41c-9fe1a7635c5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:24:53.367028 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:53.366984    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-b5a369/test-model-b5a369-kserve-6bfb684495-gckcs" podUID="d52aab51-7fdf-4be6-8893-b5fe88dbe341" containerName="main" probeResult="failure" output="Get \"https://10.133.0.188:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:24:53.417239 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:53.417192    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-0ad384/test-model-0ad384-kserve-6875d6fcff-9mwp9" podUID="517cf09e-dcec-4fe5-ad42-d7e15cd1e48a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.195:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:24:53.454620 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:53.452817    4364 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-4651e8/test-model-4651e8-kserve-5c7fcb7f7-8n9zr" podUID="ff5899b6-dd04-481f-bf5d-d4ce4e1f981d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.197:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:24:53.479751 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:53.479703    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr" podUID="446c12df-de12-4c7c-8c71-dfed93cc33f1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:24:53.736366 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:53.736239    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-31721c/test-model-31721c-kserve-64c44bfc-dmnrj" podUID="ac457ffd-caeb-419f-b448-1b21942ead0f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.192:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:24:53.802775 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:53.802726    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6" podUID="b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:24:53.803824 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:53.803798    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6"
Oct 30 13:24:53.884622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:53.870842    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh" podUID="d41ac9e5-eeed-4e07-a7fa-05d3260630b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:24:53.899621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:53.871656    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-09e4c0/test-model-09e4c0-kserve-56cdb9c5b5-nrjtk" podUID="3fd28ec8-f8ba-400b-940a-9c6b5c516936" containerName="main" probeResult="failure" output="Get \"https://10.133.0.196:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:24:53.899621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:53.893577    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh"
Oct 30 13:24:53.899621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:53.893657    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-09e4c0/test-model-09e4c0-kserve-56cdb9c5b5-nrjtk"
Oct 30 13:24:53.899621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:53.886844    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-4651e8/test-model-4651e8-kserve-5c7fcb7f7-8n9zr" podUID="ff5899b6-dd04-481f-bf5d-d4ce4e1f981d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.197:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:24:53.899621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:53.896436    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-4651e8/test-model-4651e8-kserve-5c7fcb7f7-8n9zr"
Oct 30 13:24:54.033040 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:54.032959    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-09e4c0/test-model-09e4c0-kserve-56cdb9c5b5-nrjtk"
Oct 30 13:24:54.168759 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:54.127458    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-4651e8/test-model-4651e8-kserve-5c7fcb7f7-8n9zr"
Oct 30 13:24:54.297835 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:54.296739    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh"
Oct 30 13:24:54.327078 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:54.326862    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4" podUID="0cbe6af7-d884-4613-90b8-192e5ffdb027" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:24:54.328306 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:54.327845    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4"
Oct 30 13:24:54.347428 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:54.347398    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4"
Oct 30 13:24:54.412761 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:24:54.412724    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6"
Oct 30 13:25:06.835878 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:06.835844    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-77254d-55f7f48fd6-twrk7"]
Oct 30 13:25:06.836508 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:06.836107    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-77254d-55f7f48fd6-twrk7" podUID="2cf49036-ea5d-4282-b11e-5ff1c01262d4" containerName="maas-api" containerID="cri-o://7437f6fb9ff0976fa4b90510e464fa8ff2aea1129600b542083148a2b4cebb0b" gracePeriod=30
Oct 30 13:25:07.244621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:07.238770    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-77254d-55f7f48fd6-twrk7"
Oct 30 13:25:07.357673 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:07.339384    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cf49036-ea5d-4282-b11e-5ff1c01262d4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cf49036-ea5d-4282-b11e-5ff1c01262d4-maas-api-tls\") pod \"2cf49036-ea5d-4282-b11e-5ff1c01262d4\" (UID: \"2cf49036-ea5d-4282-b11e-5ff1c01262d4\") "
Oct 30 13:25:07.357673 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:07.339552    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cf49036-ea5d-4282-b11e-5ff1c01262d4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cf49036-ea5d-4282-b11e-5ff1c01262d4-openshift-service-ca\") pod \"2cf49036-ea5d-4282-b11e-5ff1c01262d4\" (UID: \"2cf49036-ea5d-4282-b11e-5ff1c01262d4\") "
Oct 30 13:25:07.357673 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:07.339617    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cf49036-ea5d-4282-b11e-5ff1c01262d4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cf49036-ea5d-4282-b11e-5ff1c01262d4-metrics-certs\") pod \"2cf49036-ea5d-4282-b11e-5ff1c01262d4\" (UID: \"2cf49036-ea5d-4282-b11e-5ff1c01262d4\") "
Oct 30 13:25:07.357673 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:07.339701    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cf49036-ea5d-4282-b11e-5ff1c01262d4-kube-api-access-wgjbj\" (UniqueName: \"kubernetes.io/projected/2cf49036-ea5d-4282-b11e-5ff1c01262d4-kube-api-access-wgjbj\") pod \"2cf49036-ea5d-4282-b11e-5ff1c01262d4\" (UID: \"2cf49036-ea5d-4282-b11e-5ff1c01262d4\") "
Oct 30 13:25:07.358456 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:07.358418    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cf49036-ea5d-4282-b11e-5ff1c01262d4-openshift-service-ca" pod "2cf49036-ea5d-4282-b11e-5ff1c01262d4" (UID: "2cf49036-ea5d-4282-b11e-5ff1c01262d4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:25:07.372113 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:07.372081    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cf49036-ea5d-4282-b11e-5ff1c01262d4-maas-api-tls" pod "2cf49036-ea5d-4282-b11e-5ff1c01262d4" (UID: "2cf49036-ea5d-4282-b11e-5ff1c01262d4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:25:07.389420 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:07.389378    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cf49036-ea5d-4282-b11e-5ff1c01262d4-metrics-certs" pod "2cf49036-ea5d-4282-b11e-5ff1c01262d4" (UID: "2cf49036-ea5d-4282-b11e-5ff1c01262d4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:25:07.404177 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:07.398992    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cf49036-ea5d-4282-b11e-5ff1c01262d4-kube-api-access-wgjbj" pod "2cf49036-ea5d-4282-b11e-5ff1c01262d4" (UID: "2cf49036-ea5d-4282-b11e-5ff1c01262d4"). InnerVolumeSpecName "kube-api-access-wgjbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:25:07.443761 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:07.443723    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cf49036-ea5d-4282-b11e-5ff1c01262d4-maas-api-tls\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:25:07.444037 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:07.444018    4364 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cf49036-ea5d-4282-b11e-5ff1c01262d4-openshift-service-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:25:07.444141 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:07.444127    4364 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cf49036-ea5d-4282-b11e-5ff1c01262d4-metrics-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:25:07.444228 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:07.444217    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgjbj\" (UniqueName: \"kubernetes.io/projected/2cf49036-ea5d-4282-b11e-5ff1c01262d4-kube-api-access-wgjbj\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:25:07.659241 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:07.659207    4364 generic.go:356] "Generic (PLEG): container finished" podID="2cf49036-ea5d-4282-b11e-5ff1c01262d4" containerID="7437f6fb9ff0976fa4b90510e464fa8ff2aea1129600b542083148a2b4cebb0b" exitCode=0
Oct 30 13:25:07.659706 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:07.659455    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-77254d-55f7f48fd6-twrk7" event={"ID":"2cf49036-ea5d-4282-b11e-5ff1c01262d4","Type":"ContainerDied","Data":"7437f6fb9ff0976fa4b90510e464fa8ff2aea1129600b542083148a2b4cebb0b"}
Oct 30 13:25:07.659706 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:07.659518    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-77254d-55f7f48fd6-twrk7" event={"ID":"2cf49036-ea5d-4282-b11e-5ff1c01262d4","Type":"ContainerDied","Data":"c9f8c2a7d1e4be43b8a4784e5c3c52415b27880ab7e9f43aa9ec3b05169eae72"}
Oct 30 13:25:07.659706 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:07.659555    4364 scope.go:122] "RemoveContainer" containerID="7437f6fb9ff0976fa4b90510e464fa8ff2aea1129600b542083148a2b4cebb0b"
Oct 30 13:25:07.660051 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:07.659754    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-77254d-55f7f48fd6-twrk7"
Oct 30 13:25:07.702517 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:07.702495    4364 scope.go:122] "RemoveContainer" containerID="7437f6fb9ff0976fa4b90510e464fa8ff2aea1129600b542083148a2b4cebb0b"
Oct 30 13:25:07.702894 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:25:07.702872    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7437f6fb9ff0976fa4b90510e464fa8ff2aea1129600b542083148a2b4cebb0b\": container with ID starting with 7437f6fb9ff0976fa4b90510e464fa8ff2aea1129600b542083148a2b4cebb0b not found: ID does not exist" containerID="7437f6fb9ff0976fa4b90510e464fa8ff2aea1129600b542083148a2b4cebb0b"
Oct 30 13:25:07.702982 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:07.702905    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7437f6fb9ff0976fa4b90510e464fa8ff2aea1129600b542083148a2b4cebb0b"} err="failed to get container status \"7437f6fb9ff0976fa4b90510e464fa8ff2aea1129600b542083148a2b4cebb0b\": rpc error: code = NotFound desc = could not find container \"7437f6fb9ff0976fa4b90510e464fa8ff2aea1129600b542083148a2b4cebb0b\": container with ID starting with 7437f6fb9ff0976fa4b90510e464fa8ff2aea1129600b542083148a2b4cebb0b not found: ID does not exist"
Oct 30 13:25:07.769628 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:07.769570    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-77254d-55f7f48fd6-twrk7"]
Oct 30 13:25:07.783145 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:07.783118    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-77254d-55f7f48fd6-twrk7"]
Oct 30 13:25:08.973813 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:08.973778    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cf49036-ea5d-4282-b11e-5ff1c01262d4" path="/var/lib/kubelet/pods/2cf49036-ea5d-4282-b11e-5ff1c01262d4/volumes"
Oct 30 13:25:11.507392 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:25:11.507349    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:25:16.316547 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:16.313303    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp"]
Oct 30 13:25:16.316547 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:16.313639    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp" podUID="a9ee49f0-1a2e-4b0b-8046-36b55a049fad" containerName="istio-proxy" containerID="cri-o://892f73e4ec65ed02bf1f273204b3c5baeb79940b7ac2e3cdfb03d8d30f3abe2d" gracePeriod=30
Oct 30 13:25:17.761093 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:17.761061    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw"]
Oct 30 13:25:17.762112 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:17.762090    4364 cpu_manager.go:414] "containerMap: removing container" podUID="2cf49036-ea5d-4282-b11e-5ff1c01262d4" containerName="maas-api"
Oct 30 13:25:17.762112 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:17.762110    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cf49036-ea5d-4282-b11e-5ff1c01262d4" containerName="maas-api"
Oct 30 13:25:17.762275 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:17.762262    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cf49036-ea5d-4282-b11e-5ff1c01262d4" containerName="maas-api"
Oct 30 13:25:17.765875 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:17.765854    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw"
Oct 30 13:25:17.781635 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:17.781565    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-5c7ba3-openshift-default-dockercfg-62dvk\""
Oct 30 13:25:17.800672 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:17.800638    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw"]
Oct 30 13:25:17.887410 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:17.887373    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/617447eb-2414-439d-9923-c47b809de841-workload-certs\") pod \"e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw\" (UID: \"617447eb-2414-439d-9923-c47b809de841\") " pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw"
Oct 30 13:25:17.887410 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:17.887413    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/617447eb-2414-439d-9923-c47b809de841-credential-socket\") pod \"e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw\" (UID: \"617447eb-2414-439d-9923-c47b809de841\") " pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw"
Oct 30 13:25:17.887677 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:17.887436    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/617447eb-2414-439d-9923-c47b809de841-istio-data\") pod \"e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw\" (UID: \"617447eb-2414-439d-9923-c47b809de841\") " pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw"
Oct 30 13:25:17.887677 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:17.887660    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/617447eb-2414-439d-9923-c47b809de841-istiod-ca-cert\") pod \"e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw\" (UID: \"617447eb-2414-439d-9923-c47b809de841\") " pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw"
Oct 30 13:25:17.887751 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:17.887688    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/617447eb-2414-439d-9923-c47b809de841-workload-socket\") pod \"e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw\" (UID: \"617447eb-2414-439d-9923-c47b809de841\") " pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw"
Oct 30 13:25:17.887751 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:17.887731    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/617447eb-2414-439d-9923-c47b809de841-istio-token\") pod \"e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw\" (UID: \"617447eb-2414-439d-9923-c47b809de841\") " pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw"
Oct 30 13:25:17.887833 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:17.887811    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/617447eb-2414-439d-9923-c47b809de841-istio-envoy\") pod \"e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw\" (UID: \"617447eb-2414-439d-9923-c47b809de841\") " pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw"
Oct 30 13:25:17.887960 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:17.887947    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/617447eb-2414-439d-9923-c47b809de841-istio-podinfo\") pod \"e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw\" (UID: \"617447eb-2414-439d-9923-c47b809de841\") " pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw"
Oct 30 13:25:17.887992 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:17.887969    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlbp5\" (UniqueName: \"kubernetes.io/projected/617447eb-2414-439d-9923-c47b809de841-kube-api-access-dlbp5\") pod \"e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw\" (UID: \"617447eb-2414-439d-9923-c47b809de841\") " pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw"
Oct 30 13:25:17.989416 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:17.989375    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/617447eb-2414-439d-9923-c47b809de841-istiod-ca-cert\") pod \"e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw\" (UID: \"617447eb-2414-439d-9923-c47b809de841\") " pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw"
Oct 30 13:25:17.989416 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:17.989424    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/617447eb-2414-439d-9923-c47b809de841-workload-socket\") pod \"e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw\" (UID: \"617447eb-2414-439d-9923-c47b809de841\") " pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw"
Oct 30 13:25:17.989697 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:17.989489    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/617447eb-2414-439d-9923-c47b809de841-istio-token\") pod \"e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw\" (UID: \"617447eb-2414-439d-9923-c47b809de841\") " pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw"
Oct 30 13:25:17.989697 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:17.989530    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/617447eb-2414-439d-9923-c47b809de841-istio-envoy\") pod \"e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw\" (UID: \"617447eb-2414-439d-9923-c47b809de841\") " pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw"
Oct 30 13:25:17.989808 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:17.989731    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/617447eb-2414-439d-9923-c47b809de841-istio-podinfo\") pod \"e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw\" (UID: \"617447eb-2414-439d-9923-c47b809de841\") " pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw"
Oct 30 13:25:17.989808 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:17.989763    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlbp5\" (UniqueName: \"kubernetes.io/projected/617447eb-2414-439d-9923-c47b809de841-kube-api-access-dlbp5\") pod \"e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw\" (UID: \"617447eb-2414-439d-9923-c47b809de841\") " pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw"
Oct 30 13:25:17.989921 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:17.989837    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/617447eb-2414-439d-9923-c47b809de841-workload-certs\") pod \"e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw\" (UID: \"617447eb-2414-439d-9923-c47b809de841\") " pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw"
Oct 30 13:25:17.989921 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:17.989884    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/617447eb-2414-439d-9923-c47b809de841-credential-socket\") pod \"e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw\" (UID: \"617447eb-2414-439d-9923-c47b809de841\") " pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw"
Oct 30 13:25:17.990025 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:17.989919    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/617447eb-2414-439d-9923-c47b809de841-istio-data\") pod \"e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw\" (UID: \"617447eb-2414-439d-9923-c47b809de841\") " pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw"
Oct 30 13:25:17.990081 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:17.989914    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/617447eb-2414-439d-9923-c47b809de841-workload-socket\") pod \"e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw\" (UID: \"617447eb-2414-439d-9923-c47b809de841\") " pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw"
Oct 30 13:25:17.990190 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:17.990165    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/617447eb-2414-439d-9923-c47b809de841-credential-socket\") pod \"e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw\" (UID: \"617447eb-2414-439d-9923-c47b809de841\") " pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw"
Oct 30 13:25:17.990275 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:17.990171    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/617447eb-2414-439d-9923-c47b809de841-istiod-ca-cert\") pod \"e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw\" (UID: \"617447eb-2414-439d-9923-c47b809de841\") " pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw"
Oct 30 13:25:17.990338 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:17.990268    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/617447eb-2414-439d-9923-c47b809de841-workload-certs\") pod \"e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw\" (UID: \"617447eb-2414-439d-9923-c47b809de841\") " pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw"
Oct 30 13:25:17.990402 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:17.990383    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/617447eb-2414-439d-9923-c47b809de841-istio-data\") pod \"e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw\" (UID: \"617447eb-2414-439d-9923-c47b809de841\") " pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw"
Oct 30 13:25:17.991963 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:17.991932    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/617447eb-2414-439d-9923-c47b809de841-istio-envoy\") pod \"e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw\" (UID: \"617447eb-2414-439d-9923-c47b809de841\") " pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw"
Oct 30 13:25:17.992202 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:17.992184    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/617447eb-2414-439d-9923-c47b809de841-istio-podinfo\") pod \"e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw\" (UID: \"617447eb-2414-439d-9923-c47b809de841\") " pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw"
Oct 30 13:25:18.021070 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:18.021003    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlbp5\" (UniqueName: \"kubernetes.io/projected/617447eb-2414-439d-9923-c47b809de841-kube-api-access-dlbp5\") pod \"e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw\" (UID: \"617447eb-2414-439d-9923-c47b809de841\") " pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw"
Oct 30 13:25:18.021896 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:18.021877    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/617447eb-2414-439d-9923-c47b809de841-istio-token\") pod \"e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw\" (UID: \"617447eb-2414-439d-9923-c47b809de841\") " pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw"
Oct 30 13:25:18.082508 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:18.082477    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw"
Oct 30 13:25:18.410811 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:25:18.410774    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod617447eb_2414_439d_9923_c47b809de841.slice/crio-7577b4f5a0c12afd187af7443dc0ec355a2f7b35073ef4b611d43920561c714e WatchSource:0}: Error finding container 7577b4f5a0c12afd187af7443dc0ec355a2f7b35073ef4b611d43920561c714e: Status 404 returned error can't find the container with id 7577b4f5a0c12afd187af7443dc0ec355a2f7b35073ef4b611d43920561c714e
Oct 30 13:25:18.411641 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:18.411611    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw"]
Oct 30 13:25:18.414654 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:18.414616    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:25:18.414743 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:18.414726    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:25:18.414802 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:18.414787    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:25:18.796468 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:18.796372    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw" event={"ID":"617447eb-2414-439d-9923-c47b809de841","Type":"ContainerStarted","Data":"1925ed105ee025e7459d40dead0c738808cda8573e6e0cf8bd847f8615448e50"}
Oct 30 13:25:18.796468 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:18.796436    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw" event={"ID":"617447eb-2414-439d-9923-c47b809de841","Type":"ContainerStarted","Data":"7577b4f5a0c12afd187af7443dc0ec355a2f7b35073ef4b611d43920561c714e"}
Oct 30 13:25:18.875045 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:18.874949    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw" podStartSLOduration=1.874933608 podStartE2EDuration="1.874933608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:25:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:25:18.858272876 +0000 UTC m=+2018.024637525" watchObservedRunningTime="2026-10-30 13:25:18.874933608 +0000 UTC m=+2018.041298256"
Oct 30 13:25:19.082983 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:19.082832    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw"
Oct 30 13:25:19.086325 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:19.085670    4364 patch_prober.go:29] interesting pod/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Oct 30 13:25:19.086325 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:19.085741    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw" podUID="617447eb-2414-439d-9923-c47b809de841" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Oct 30 13:25:19.149881 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:19.149840    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-306f377f-5pqdt"]
Oct 30 13:25:19.154460 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:19.154423    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-306f377f-5pqdt"
Oct 30 13:25:19.170059 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:19.170032    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:25:19.172195 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:19.172152    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\""
Oct 30 13:25:19.178391 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:19.178363    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-306f377f-5pqdt"]
Oct 30 13:25:19.316754 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:19.316717    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea8df410-6452-4d39-966f-feda1430956c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-306f377f-5pqdt\" (UID: \"ea8df410-6452-4d39-966f-feda1430956c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-306f377f-5pqdt"
Oct 30 13:25:19.316981 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:19.316955    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea8df410-6452-4d39-966f-feda1430956c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-306f377f-5pqdt\" (UID: \"ea8df410-6452-4d39-966f-feda1430956c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-306f377f-5pqdt"
Oct 30 13:25:19.317121 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:19.317074    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz6vx\" (UniqueName: \"kubernetes.io/projected/ea8df410-6452-4d39-966f-feda1430956c-kube-api-access-bz6vx\") pod \"maas-api-revoke-sub-306f377f-5pqdt\" (UID: \"ea8df410-6452-4d39-966f-feda1430956c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-306f377f-5pqdt"
Oct 30 13:25:19.420823 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:19.420780    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea8df410-6452-4d39-966f-feda1430956c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-306f377f-5pqdt\" (UID: \"ea8df410-6452-4d39-966f-feda1430956c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-306f377f-5pqdt"
Oct 30 13:25:19.421057 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:19.421037    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea8df410-6452-4d39-966f-feda1430956c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-306f377f-5pqdt\" (UID: \"ea8df410-6452-4d39-966f-feda1430956c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-306f377f-5pqdt"
Oct 30 13:25:19.421125 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:19.421091    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz6vx\" (UniqueName: \"kubernetes.io/projected/ea8df410-6452-4d39-966f-feda1430956c-kube-api-access-bz6vx\") pod \"maas-api-revoke-sub-306f377f-5pqdt\" (UID: \"ea8df410-6452-4d39-966f-feda1430956c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-306f377f-5pqdt"
Oct 30 13:25:19.422039 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:19.422007    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea8df410-6452-4d39-966f-feda1430956c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-306f377f-5pqdt\" (UID: \"ea8df410-6452-4d39-966f-feda1430956c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-306f377f-5pqdt"
Oct 30 13:25:19.427054 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:19.426942    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea8df410-6452-4d39-966f-feda1430956c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-306f377f-5pqdt\" (UID: \"ea8df410-6452-4d39-966f-feda1430956c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-306f377f-5pqdt"
Oct 30 13:25:19.446708 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:19.446672    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz6vx\" (UniqueName: \"kubernetes.io/projected/ea8df410-6452-4d39-966f-feda1430956c-kube-api-access-bz6vx\") pod \"maas-api-revoke-sub-306f377f-5pqdt\" (UID: \"ea8df410-6452-4d39-966f-feda1430956c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-306f377f-5pqdt"
Oct 30 13:25:19.474062 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:19.474028    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-306f377f-5pqdt"
Oct 30 13:25:19.891328 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:19.883390    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-306f377f-5pqdt"]
Oct 30 13:25:20.108039 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:20.107996    4364 patch_prober.go:29] interesting pod/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Oct 30 13:25:20.108220 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:20.108073    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw" podUID="617447eb-2414-439d-9923-c47b809de841" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Oct 30 13:25:20.877995 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:20.877952    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-306f377f-5pqdt" event={"ID":"ea8df410-6452-4d39-966f-feda1430956c","Type":"ContainerStarted","Data":"3e34b90a68dbbcf23877636968714111cb0e1352daf666dcbacc99053b06d538"}
Oct 30 13:25:20.878176 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:20.878059    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-306f377f-5pqdt" event={"ID":"ea8df410-6452-4d39-966f-feda1430956c","Type":"ContainerStarted","Data":"ab24a42aa3ac268da6a07ae3a3090effb54d7199ea257c58234ad3933b3c39e8"}
Oct 30 13:25:20.928025 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:20.927808    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-306f377f-5pqdt" podStartSLOduration=1.92779221 podStartE2EDuration="1.92779221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:25:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:25:20.926637602 +0000 UTC m=+2020.093002247" watchObservedRunningTime="2026-10-30 13:25:20.92779221 +0000 UTC m=+2020.094156859"
Oct 30 13:25:21.104503 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:21.104433    4364 patch_prober.go:29] interesting pod/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Oct 30 13:25:21.104741 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:21.104532    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw" podUID="617447eb-2414-439d-9923-c47b809de841" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Oct 30 13:25:21.753079 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:21.753052    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp"
Oct 30 13:25:21.807184 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:21.804827    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9ee49f0-1a2e-4b0b-8046-36b55a049fad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9ee49f0-1a2e-4b0b-8046-36b55a049fad-workload-certs\") pod \"a9ee49f0-1a2e-4b0b-8046-36b55a049fad\" (UID: \"a9ee49f0-1a2e-4b0b-8046-36b55a049fad\") "
Oct 30 13:25:21.807184 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:21.805038    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9ee49f0-1a2e-4b0b-8046-36b55a049fad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9ee49f0-1a2e-4b0b-8046-36b55a049fad-workload-socket\") pod \"a9ee49f0-1a2e-4b0b-8046-36b55a049fad\" (UID: \"a9ee49f0-1a2e-4b0b-8046-36b55a049fad\") "
Oct 30 13:25:21.807184 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:21.805179    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9ee49f0-1a2e-4b0b-8046-36b55a049fad-kube-api-access-qmbsc\" (UniqueName: \"kubernetes.io/projected/a9ee49f0-1a2e-4b0b-8046-36b55a049fad-kube-api-access-qmbsc\") pod \"a9ee49f0-1a2e-4b0b-8046-36b55a049fad\" (UID: \"a9ee49f0-1a2e-4b0b-8046-36b55a049fad\") "
Oct 30 13:25:21.807184 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:21.805199    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9ee49f0-1a2e-4b0b-8046-36b55a049fad-workload-certs" pod "a9ee49f0-1a2e-4b0b-8046-36b55a049fad" (UID: "a9ee49f0-1a2e-4b0b-8046-36b55a049fad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:25:21.807184 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:21.805228    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9ee49f0-1a2e-4b0b-8046-36b55a049fad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9ee49f0-1a2e-4b0b-8046-36b55a049fad-istiod-ca-cert\") pod \"a9ee49f0-1a2e-4b0b-8046-36b55a049fad\" (UID: \"a9ee49f0-1a2e-4b0b-8046-36b55a049fad\") "
Oct 30 13:25:21.807184 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:21.805293    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9ee49f0-1a2e-4b0b-8046-36b55a049fad-workload-socket" pod "a9ee49f0-1a2e-4b0b-8046-36b55a049fad" (UID: "a9ee49f0-1a2e-4b0b-8046-36b55a049fad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:25:21.807184 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:21.805462    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a9ee49f0-1a2e-4b0b-8046-36b55a049fad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9ee49f0-1a2e-4b0b-8046-36b55a049fad-istio-podinfo\") pod \"a9ee49f0-1a2e-4b0b-8046-36b55a049fad\" (UID: \"a9ee49f0-1a2e-4b0b-8046-36b55a049fad\") "
Oct 30 13:25:21.807184 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:21.805552    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9ee49f0-1a2e-4b0b-8046-36b55a049fad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9ee49f0-1a2e-4b0b-8046-36b55a049fad-credential-socket\") pod \"a9ee49f0-1a2e-4b0b-8046-36b55a049fad\" (UID: \"a9ee49f0-1a2e-4b0b-8046-36b55a049fad\") "
Oct 30 13:25:21.807184 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:21.805644    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9ee49f0-1a2e-4b0b-8046-36b55a049fad-istio-token\" (UniqueName: \"kubernetes.io/projected/a9ee49f0-1a2e-4b0b-8046-36b55a049fad-istio-token\") pod \"a9ee49f0-1a2e-4b0b-8046-36b55a049fad\" (UID: \"a9ee49f0-1a2e-4b0b-8046-36b55a049fad\") "
Oct 30 13:25:21.807184 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:21.805734    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9ee49f0-1a2e-4b0b-8046-36b55a049fad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9ee49f0-1a2e-4b0b-8046-36b55a049fad-istio-envoy\") pod \"a9ee49f0-1a2e-4b0b-8046-36b55a049fad\" (UID: \"a9ee49f0-1a2e-4b0b-8046-36b55a049fad\") "
Oct 30 13:25:21.807184 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:21.805853    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9ee49f0-1a2e-4b0b-8046-36b55a049fad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9ee49f0-1a2e-4b0b-8046-36b55a049fad-istio-data\") pod \"a9ee49f0-1a2e-4b0b-8046-36b55a049fad\" (UID: \"a9ee49f0-1a2e-4b0b-8046-36b55a049fad\") "
Oct 30 13:25:21.807184 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:21.806111    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9ee49f0-1a2e-4b0b-8046-36b55a049fad-credential-socket" pod "a9ee49f0-1a2e-4b0b-8046-36b55a049fad" (UID: "a9ee49f0-1a2e-4b0b-8046-36b55a049fad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:25:21.807184 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:21.806295    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9ee49f0-1a2e-4b0b-8046-36b55a049fad-istio-data" pod "a9ee49f0-1a2e-4b0b-8046-36b55a049fad" (UID: "a9ee49f0-1a2e-4b0b-8046-36b55a049fad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:25:21.807184 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:21.806806    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9ee49f0-1a2e-4b0b-8046-36b55a049fad-istiod-ca-cert" pod "a9ee49f0-1a2e-4b0b-8046-36b55a049fad" (UID: "a9ee49f0-1a2e-4b0b-8046-36b55a049fad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:25:21.807184 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:21.806981    4364 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9ee49f0-1a2e-4b0b-8046-36b55a049fad-istio-data\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:25:21.807184 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:21.807004    4364 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9ee49f0-1a2e-4b0b-8046-36b55a049fad-workload-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:25:21.807184 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:21.807020    4364 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9ee49f0-1a2e-4b0b-8046-36b55a049fad-workload-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:25:21.807184 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:21.807032    4364 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9ee49f0-1a2e-4b0b-8046-36b55a049fad-istiod-ca-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:25:21.807184 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:21.807065    4364 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9ee49f0-1a2e-4b0b-8046-36b55a049fad-credential-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:25:21.808384 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:21.808111    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9ee49f0-1a2e-4b0b-8046-36b55a049fad-kube-api-access-qmbsc" pod "a9ee49f0-1a2e-4b0b-8046-36b55a049fad" (UID: "a9ee49f0-1a2e-4b0b-8046-36b55a049fad"). InnerVolumeSpecName "kube-api-access-qmbsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:25:21.808838 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:21.808813    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9ee49f0-1a2e-4b0b-8046-36b55a049fad-istio-envoy" pod "a9ee49f0-1a2e-4b0b-8046-36b55a049fad" (UID: "a9ee49f0-1a2e-4b0b-8046-36b55a049fad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:25:21.808931 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:21.808894    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a9ee49f0-1a2e-4b0b-8046-36b55a049fad-istio-podinfo" pod "a9ee49f0-1a2e-4b0b-8046-36b55a049fad" (UID: "a9ee49f0-1a2e-4b0b-8046-36b55a049fad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:25:21.809171 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:21.809152    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9ee49f0-1a2e-4b0b-8046-36b55a049fad-istio-token" pod "a9ee49f0-1a2e-4b0b-8046-36b55a049fad" (UID: "a9ee49f0-1a2e-4b0b-8046-36b55a049fad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:25:21.909855 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:21.909803    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmbsc\" (UniqueName: \"kubernetes.io/projected/a9ee49f0-1a2e-4b0b-8046-36b55a049fad-kube-api-access-qmbsc\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:25:21.909855 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:21.909857    4364 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9ee49f0-1a2e-4b0b-8046-36b55a049fad-istio-podinfo\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:25:21.910084 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:21.909874    4364 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9ee49f0-1a2e-4b0b-8046-36b55a049fad-istio-token\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:25:21.910084 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:21.909902    4364 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9ee49f0-1a2e-4b0b-8046-36b55a049fad-istio-envoy\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:25:21.914925 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:21.914888    4364 generic.go:356] "Generic (PLEG): container finished" podID="a9ee49f0-1a2e-4b0b-8046-36b55a049fad" containerID="892f73e4ec65ed02bf1f273204b3c5baeb79940b7ac2e3cdfb03d8d30f3abe2d" exitCode=0
Oct 30 13:25:21.915064 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:21.914956    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp"
Oct 30 13:25:21.915064 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:21.914946    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp" event={"ID":"a9ee49f0-1a2e-4b0b-8046-36b55a049fad","Type":"ContainerDied","Data":"892f73e4ec65ed02bf1f273204b3c5baeb79940b7ac2e3cdfb03d8d30f3abe2d"}
Oct 30 13:25:21.915172 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:21.915106    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp" event={"ID":"a9ee49f0-1a2e-4b0b-8046-36b55a049fad","Type":"ContainerDied","Data":"d5d0ca4c46e04b7440e8f88ac3cd9efc18ddb72ca56da78d560a8f882aacefb2"}
Oct 30 13:25:21.915172 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:21.915141    4364 scope.go:122] "RemoveContainer" containerID="892f73e4ec65ed02bf1f273204b3c5baeb79940b7ac2e3cdfb03d8d30f3abe2d"
Oct 30 13:25:21.918909 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:21.918880    4364 generic.go:356] "Generic (PLEG): container finished" podID="ea8df410-6452-4d39-966f-feda1430956c" containerID="3e34b90a68dbbcf23877636968714111cb0e1352daf666dcbacc99053b06d538" exitCode=0
Oct 30 13:25:21.919089 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:21.919071    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-306f377f-5pqdt" event={"ID":"ea8df410-6452-4d39-966f-feda1430956c","Type":"ContainerDied","Data":"3e34b90a68dbbcf23877636968714111cb0e1352daf666dcbacc99053b06d538"}
Oct 30 13:25:21.940352 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:21.940327    4364 scope.go:122] "RemoveContainer" containerID="892f73e4ec65ed02bf1f273204b3c5baeb79940b7ac2e3cdfb03d8d30f3abe2d"
Oct 30 13:25:21.940966 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:25:21.940740    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"892f73e4ec65ed02bf1f273204b3c5baeb79940b7ac2e3cdfb03d8d30f3abe2d\": container with ID starting with 892f73e4ec65ed02bf1f273204b3c5baeb79940b7ac2e3cdfb03d8d30f3abe2d not found: ID does not exist" containerID="892f73e4ec65ed02bf1f273204b3c5baeb79940b7ac2e3cdfb03d8d30f3abe2d"
Oct 30 13:25:21.940966 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:21.940769    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"892f73e4ec65ed02bf1f273204b3c5baeb79940b7ac2e3cdfb03d8d30f3abe2d"} err="failed to get container status \"892f73e4ec65ed02bf1f273204b3c5baeb79940b7ac2e3cdfb03d8d30f3abe2d\": rpc error: code = NotFound desc = could not find container \"892f73e4ec65ed02bf1f273204b3c5baeb79940b7ac2e3cdfb03d8d30f3abe2d\": container with ID starting with 892f73e4ec65ed02bf1f273204b3c5baeb79940b7ac2e3cdfb03d8d30f3abe2d not found: ID does not exist"
Oct 30 13:25:21.971790 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:21.971763    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp"]
Oct 30 13:25:21.977112 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:21.977091    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-77254d-openshift-default-6ccff6c797-nlzjp"]
Oct 30 13:25:22.106348 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:22.105904    4364 patch_prober.go:29] interesting pod/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Oct 30 13:25:22.106348 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:22.105969    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw" podUID="617447eb-2414-439d-9923-c47b809de841" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Oct 30 13:25:22.975250 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:22.975213    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9ee49f0-1a2e-4b0b-8046-36b55a049fad" path="/var/lib/kubelet/pods/a9ee49f0-1a2e-4b0b-8046-36b55a049fad/volumes"
Oct 30 13:25:23.110993 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:23.110392    4364 patch_prober.go:29] interesting pod/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Oct 30 13:25:23.110993 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:23.110447    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw" podUID="617447eb-2414-439d-9923-c47b809de841" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Oct 30 13:25:23.157703 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:23.157680    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-306f377f-5pqdt"
Oct 30 13:25:23.230785 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:23.230701    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea8df410-6452-4d39-966f-feda1430956c-kube-api-access-bz6vx\" (UniqueName: \"kubernetes.io/projected/ea8df410-6452-4d39-966f-feda1430956c-kube-api-access-bz6vx\") pod \"ea8df410-6452-4d39-966f-feda1430956c\" (UID: \"ea8df410-6452-4d39-966f-feda1430956c\") "
Oct 30 13:25:23.230962 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:23.230818    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea8df410-6452-4d39-966f-feda1430956c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea8df410-6452-4d39-966f-feda1430956c-maas-api-openshift-ca\") pod \"ea8df410-6452-4d39-966f-feda1430956c\" (UID: \"ea8df410-6452-4d39-966f-feda1430956c\") "
Oct 30 13:25:23.231236 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:23.231211    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea8df410-6452-4d39-966f-feda1430956c-maas-api-openshift-ca" pod "ea8df410-6452-4d39-966f-feda1430956c" (UID: "ea8df410-6452-4d39-966f-feda1430956c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:25:23.231508 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:23.231488    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea8df410-6452-4d39-966f-feda1430956c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea8df410-6452-4d39-966f-feda1430956c-maas-api-xks-ca\") pod \"ea8df410-6452-4d39-966f-feda1430956c\" (UID: \"ea8df410-6452-4d39-966f-feda1430956c\") "
Oct 30 13:25:23.232746 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:23.232724    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea8df410-6452-4d39-966f-feda1430956c-maas-api-openshift-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:25:23.233710 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:23.233641    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea8df410-6452-4d39-966f-feda1430956c-kube-api-access-bz6vx" pod "ea8df410-6452-4d39-966f-feda1430956c" (UID: "ea8df410-6452-4d39-966f-feda1430956c"). InnerVolumeSpecName "kube-api-access-bz6vx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:25:23.234199 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:23.234164    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea8df410-6452-4d39-966f-feda1430956c-maas-api-xks-ca" pod "ea8df410-6452-4d39-966f-feda1430956c" (UID: "ea8df410-6452-4d39-966f-feda1430956c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:25:23.333903 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:23.333874    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bz6vx\" (UniqueName: \"kubernetes.io/projected/ea8df410-6452-4d39-966f-feda1430956c-kube-api-access-bz6vx\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:25:23.334095 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:23.334084    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea8df410-6452-4d39-966f-feda1430956c-maas-api-xks-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:25:23.952307 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:23.952119    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-306f377f-5pqdt"
Oct 30 13:25:23.952307 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:23.952152    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-306f377f-5pqdt" event={"ID":"ea8df410-6452-4d39-966f-feda1430956c","Type":"ContainerDied","Data":"ab24a42aa3ac268da6a07ae3a3090effb54d7199ea257c58234ad3933b3c39e8"}
Oct 30 13:25:23.952307 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:23.952229    4364 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab24a42aa3ac268da6a07ae3a3090effb54d7199ea257c58234ad3933b3c39e8"
Oct 30 13:25:24.103625 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:24.102961    4364 patch_prober.go:29] interesting pod/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Oct 30 13:25:24.103625 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:24.103024    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw" podUID="617447eb-2414-439d-9923-c47b809de841" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Oct 30 13:25:25.087665 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:25.087634    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw"
Oct 30 13:25:25.088116 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:25.088096    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw"
Oct 30 13:25:25.088770 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:25.088753    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw"
Oct 30 13:25:26.880348 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:26.880314    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4"]
Oct 30 13:25:26.880857 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:26.880577    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4" podUID="b96befc8-d310-489e-a714-2a06ccd4455c" containerName="istio-proxy" containerID="cri-o://60e747d0babe56a27a485269a80fe2afb8a6a61328f0366e890dddd2656dd332" gracePeriod=30
Oct 30 13:25:27.376429 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:27.376397    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ddb271-789bc5d4c9-p97mb"]
Oct 30 13:25:30.931300 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:30.931252    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-31721c/test-model-31721c-kserve-64c44bfc-dmnrj"]
Oct 30 13:25:32.185086 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:32.185055    4364 generic.go:356] "Generic (PLEG): container finished" podID="b96befc8-d310-489e-a714-2a06ccd4455c" containerID="60e747d0babe56a27a485269a80fe2afb8a6a61328f0366e890dddd2656dd332" exitCode=0
Oct 30 13:25:32.185428 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:32.185127    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4" event={"ID":"b96befc8-d310-489e-a714-2a06ccd4455c","Type":"ContainerDied","Data":"60e747d0babe56a27a485269a80fe2afb8a6a61328f0366e890dddd2656dd332"}
Oct 30 13:25:32.185428 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:32.185153    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4"
Oct 30 13:25:32.185428 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:32.185213    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4" event={"ID":"b96befc8-d310-489e-a714-2a06ccd4455c","Type":"ContainerDied","Data":"207f96bf6c3b1141fc4088f4a4b9dbd4b0e9f735bac871e53eadd43b06e1c4bc"}
Oct 30 13:25:32.185428 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:32.185243    4364 scope.go:122] "RemoveContainer" containerID="60e747d0babe56a27a485269a80fe2afb8a6a61328f0366e890dddd2656dd332"
Oct 30 13:25:32.196144 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:32.196123    4364 scope.go:122] "RemoveContainer" containerID="60e747d0babe56a27a485269a80fe2afb8a6a61328f0366e890dddd2656dd332"
Oct 30 13:25:32.196427 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:25:32.196408    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60e747d0babe56a27a485269a80fe2afb8a6a61328f0366e890dddd2656dd332\": container with ID starting with 60e747d0babe56a27a485269a80fe2afb8a6a61328f0366e890dddd2656dd332 not found: ID does not exist" containerID="60e747d0babe56a27a485269a80fe2afb8a6a61328f0366e890dddd2656dd332"
Oct 30 13:25:32.196466 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:32.196437    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60e747d0babe56a27a485269a80fe2afb8a6a61328f0366e890dddd2656dd332"} err="failed to get container status \"60e747d0babe56a27a485269a80fe2afb8a6a61328f0366e890dddd2656dd332\": rpc error: code = NotFound desc = could not find container \"60e747d0babe56a27a485269a80fe2afb8a6a61328f0366e890dddd2656dd332\": container with ID starting with 60e747d0babe56a27a485269a80fe2afb8a6a61328f0366e890dddd2656dd332 not found: ID does not exist"
Oct 30 13:25:32.274923 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:32.274895    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b96befc8-d310-489e-a714-2a06ccd4455c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b96befc8-d310-489e-a714-2a06ccd4455c-istio-envoy\") pod \"b96befc8-d310-489e-a714-2a06ccd4455c\" (UID: \"b96befc8-d310-489e-a714-2a06ccd4455c\") "
Oct 30 13:25:32.275121 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:32.274963    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b96befc8-d310-489e-a714-2a06ccd4455c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b96befc8-d310-489e-a714-2a06ccd4455c-credential-socket\") pod \"b96befc8-d310-489e-a714-2a06ccd4455c\" (UID: \"b96befc8-d310-489e-a714-2a06ccd4455c\") "
Oct 30 13:25:32.275121 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:32.275076    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b96befc8-d310-489e-a714-2a06ccd4455c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b96befc8-d310-489e-a714-2a06ccd4455c-workload-socket\") pod \"b96befc8-d310-489e-a714-2a06ccd4455c\" (UID: \"b96befc8-d310-489e-a714-2a06ccd4455c\") "
Oct 30 13:25:32.275264 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:32.275186    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b96befc8-d310-489e-a714-2a06ccd4455c-kube-api-access-n4gp2\" (UniqueName: \"kubernetes.io/projected/b96befc8-d310-489e-a714-2a06ccd4455c-kube-api-access-n4gp2\") pod \"b96befc8-d310-489e-a714-2a06ccd4455c\" (UID: \"b96befc8-d310-489e-a714-2a06ccd4455c\") "
Oct 30 13:25:32.275264 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:32.275237    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b96befc8-d310-489e-a714-2a06ccd4455c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b96befc8-d310-489e-a714-2a06ccd4455c-istiod-ca-cert\") pod \"b96befc8-d310-489e-a714-2a06ccd4455c\" (UID: \"b96befc8-d310-489e-a714-2a06ccd4455c\") "
Oct 30 13:25:32.275371 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:32.275319    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b96befc8-d310-489e-a714-2a06ccd4455c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b96befc8-d310-489e-a714-2a06ccd4455c-istio-data\") pod \"b96befc8-d310-489e-a714-2a06ccd4455c\" (UID: \"b96befc8-d310-489e-a714-2a06ccd4455c\") "
Oct 30 13:25:32.275463 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:32.275423    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b96befc8-d310-489e-a714-2a06ccd4455c-workload-socket" pod "b96befc8-d310-489e-a714-2a06ccd4455c" (UID: "b96befc8-d310-489e-a714-2a06ccd4455c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:25:32.275463 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:32.275440    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b96befc8-d310-489e-a714-2a06ccd4455c-istio-token\" (UniqueName: \"kubernetes.io/projected/b96befc8-d310-489e-a714-2a06ccd4455c-istio-token\") pod \"b96befc8-d310-489e-a714-2a06ccd4455c\" (UID: \"b96befc8-d310-489e-a714-2a06ccd4455c\") "
Oct 30 13:25:32.275587 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:32.275511    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b96befc8-d310-489e-a714-2a06ccd4455c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b96befc8-d310-489e-a714-2a06ccd4455c-istio-podinfo\") pod \"b96befc8-d310-489e-a714-2a06ccd4455c\" (UID: \"b96befc8-d310-489e-a714-2a06ccd4455c\") "
Oct 30 13:25:32.275687 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:32.275619    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b96befc8-d310-489e-a714-2a06ccd4455c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b96befc8-d310-489e-a714-2a06ccd4455c-workload-certs\") pod \"b96befc8-d310-489e-a714-2a06ccd4455c\" (UID: \"b96befc8-d310-489e-a714-2a06ccd4455c\") "
Oct 30 13:25:32.275747 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:32.275708    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b96befc8-d310-489e-a714-2a06ccd4455c-istiod-ca-cert" pod "b96befc8-d310-489e-a714-2a06ccd4455c" (UID: "b96befc8-d310-489e-a714-2a06ccd4455c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:25:32.275900 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:32.275879    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b96befc8-d310-489e-a714-2a06ccd4455c-credential-socket" pod "b96befc8-d310-489e-a714-2a06ccd4455c" (UID: "b96befc8-d310-489e-a714-2a06ccd4455c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:25:32.276222 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:32.276198    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b96befc8-d310-489e-a714-2a06ccd4455c-workload-certs" pod "b96befc8-d310-489e-a714-2a06ccd4455c" (UID: "b96befc8-d310-489e-a714-2a06ccd4455c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:25:32.276406 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:32.276385    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b96befc8-d310-489e-a714-2a06ccd4455c-istio-data" pod "b96befc8-d310-489e-a714-2a06ccd4455c" (UID: "b96befc8-d310-489e-a714-2a06ccd4455c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:25:32.276908 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:32.276884    4364 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b96befc8-d310-489e-a714-2a06ccd4455c-credential-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:25:32.277033 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:32.277019    4364 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b96befc8-d310-489e-a714-2a06ccd4455c-workload-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:25:32.277133 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:32.277122    4364 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b96befc8-d310-489e-a714-2a06ccd4455c-istiod-ca-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:25:32.277225 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:32.277214    4364 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b96befc8-d310-489e-a714-2a06ccd4455c-istio-data\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:25:32.277313 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:32.277302    4364 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b96befc8-d310-489e-a714-2a06ccd4455c-workload-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:25:32.278293 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:32.278233    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b96befc8-d310-489e-a714-2a06ccd4455c-istio-envoy" pod "b96befc8-d310-489e-a714-2a06ccd4455c" (UID: "b96befc8-d310-489e-a714-2a06ccd4455c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:25:32.278389 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:32.278336    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b96befc8-d310-489e-a714-2a06ccd4455c-istio-token" pod "b96befc8-d310-489e-a714-2a06ccd4455c" (UID: "b96befc8-d310-489e-a714-2a06ccd4455c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:25:32.278808 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:32.278781    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b96befc8-d310-489e-a714-2a06ccd4455c-istio-podinfo" pod "b96befc8-d310-489e-a714-2a06ccd4455c" (UID: "b96befc8-d310-489e-a714-2a06ccd4455c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:25:32.279147 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:32.279122    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b96befc8-d310-489e-a714-2a06ccd4455c-kube-api-access-n4gp2" pod "b96befc8-d310-489e-a714-2a06ccd4455c" (UID: "b96befc8-d310-489e-a714-2a06ccd4455c"). InnerVolumeSpecName "kube-api-access-n4gp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:25:32.377713 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:32.377640    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ddb271-789bc5d4c9-p97mb" podUID="b2bde24e-a303-4111-adf3-123d6df17c00" containerName="payload-pre-processing" containerID="cri-o://9852aa57b3462a6707f95d9d51449df20f381de671eb9faa42d6d79d66b0702b" gracePeriod=25
Oct 30 13:25:32.379034 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:32.379005    4364 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b96befc8-d310-489e-a714-2a06ccd4455c-istio-envoy\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:25:32.379034 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:32.379033    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4gp2\" (UniqueName: \"kubernetes.io/projected/b96befc8-d310-489e-a714-2a06ccd4455c-kube-api-access-n4gp2\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:25:32.379276 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:32.379043    4364 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b96befc8-d310-489e-a714-2a06ccd4455c-istio-token\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:25:32.379276 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:32.379053    4364 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b96befc8-d310-489e-a714-2a06ccd4455c-istio-podinfo\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:25:33.195518 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:33.195483    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4"
Oct 30 13:25:33.262552 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:33.262470    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4"]
Oct 30 13:25:33.273706 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:33.273678    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-gvkj4"]
Oct 30 13:25:34.974298 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:34.974269    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b96befc8-d310-489e-a714-2a06ccd4455c" path="/var/lib/kubelet/pods/b96befc8-d310-489e-a714-2a06ccd4455c/volumes"
Oct 30 13:25:34.981547 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:34.981521    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40bd9b0b-54kfk"]
Oct 30 13:25:34.982525 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:34.982502    4364 cpu_manager.go:414] "containerMap: removing container" podUID="a9ee49f0-1a2e-4b0b-8046-36b55a049fad" containerName="istio-proxy"
Oct 30 13:25:34.982525 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:34.982523    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9ee49f0-1a2e-4b0b-8046-36b55a049fad" containerName="istio-proxy"
Oct 30 13:25:34.983180 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:34.982559    4364 cpu_manager.go:414] "containerMap: removing container" podUID="ea8df410-6452-4d39-966f-feda1430956c" containerName="revoke-keys"
Oct 30 13:25:34.983180 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:34.982567    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea8df410-6452-4d39-966f-feda1430956c" containerName="revoke-keys"
Oct 30 13:25:34.983180 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:34.982686    4364 cpu_manager.go:414] "containerMap: removing container" podUID="b96befc8-d310-489e-a714-2a06ccd4455c" containerName="istio-proxy"
Oct 30 13:25:34.983180 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:34.982700    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b96befc8-d310-489e-a714-2a06ccd4455c" containerName="istio-proxy"
Oct 30 13:25:35.002621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:34.999051    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="b96befc8-d310-489e-a714-2a06ccd4455c" containerName="istio-proxy"
Oct 30 13:25:35.002621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:34.999104    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9ee49f0-1a2e-4b0b-8046-36b55a049fad" containerName="istio-proxy"
Oct 30 13:25:35.002621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:34.999127    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea8df410-6452-4d39-966f-feda1430956c" containerName="revoke-keys"
Oct 30 13:25:35.032568 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:35.029815    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40bd9b0b-54kfk"
Oct 30 13:25:35.045739 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:35.045324    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:25:35.058450 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:35.057697    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\""
Oct 30 13:25:35.077683 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:35.071639    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40bd9b0b-54kfk"]
Oct 30 13:25:35.131752 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:35.131715    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73a2c7d3-f20c-4c98-8416-eec166f20c48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40bd9b0b-54kfk\" (UID: \"73a2c7d3-f20c-4c98-8416-eec166f20c48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40bd9b0b-54kfk"
Oct 30 13:25:35.131939 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:35.131840    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xwv2\" (UniqueName: \"kubernetes.io/projected/73a2c7d3-f20c-4c98-8416-eec166f20c48-kube-api-access-9xwv2\") pod \"maas-api-revoke-sub-40bd9b0b-54kfk\" (UID: \"73a2c7d3-f20c-4c98-8416-eec166f20c48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40bd9b0b-54kfk"
Oct 30 13:25:35.132063 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:35.132047    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73a2c7d3-f20c-4c98-8416-eec166f20c48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40bd9b0b-54kfk\" (UID: \"73a2c7d3-f20c-4c98-8416-eec166f20c48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40bd9b0b-54kfk"
Oct 30 13:25:35.234352 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:35.234318    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73a2c7d3-f20c-4c98-8416-eec166f20c48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40bd9b0b-54kfk\" (UID: \"73a2c7d3-f20c-4c98-8416-eec166f20c48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40bd9b0b-54kfk"
Oct 30 13:25:35.234552 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:35.234383    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xwv2\" (UniqueName: \"kubernetes.io/projected/73a2c7d3-f20c-4c98-8416-eec166f20c48-kube-api-access-9xwv2\") pod \"maas-api-revoke-sub-40bd9b0b-54kfk\" (UID: \"73a2c7d3-f20c-4c98-8416-eec166f20c48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40bd9b0b-54kfk"
Oct 30 13:25:35.234552 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:35.234493    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73a2c7d3-f20c-4c98-8416-eec166f20c48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40bd9b0b-54kfk\" (UID: \"73a2c7d3-f20c-4c98-8416-eec166f20c48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40bd9b0b-54kfk"
Oct 30 13:25:35.235205 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:35.235172    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73a2c7d3-f20c-4c98-8416-eec166f20c48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40bd9b0b-54kfk\" (UID: \"73a2c7d3-f20c-4c98-8416-eec166f20c48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40bd9b0b-54kfk"
Oct 30 13:25:35.236935 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:35.236911    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73a2c7d3-f20c-4c98-8416-eec166f20c48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40bd9b0b-54kfk\" (UID: \"73a2c7d3-f20c-4c98-8416-eec166f20c48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40bd9b0b-54kfk"
Oct 30 13:25:35.257945 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:35.257920    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xwv2\" (UniqueName: \"kubernetes.io/projected/73a2c7d3-f20c-4c98-8416-eec166f20c48-kube-api-access-9xwv2\") pod \"maas-api-revoke-sub-40bd9b0b-54kfk\" (UID: \"73a2c7d3-f20c-4c98-8416-eec166f20c48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40bd9b0b-54kfk"
Oct 30 13:25:35.392458 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:35.392377    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40bd9b0b-54kfk"
Oct 30 13:25:35.719953 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:35.719923    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40bd9b0b-54kfk"]
Oct 30 13:25:35.720323 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:25:35.720294    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73a2c7d3_f20c_4c98_8416_eec166f20c48.slice/crio-cef1c829c69788b7250de41e9938134e7131f339699207e7700d8c0a2ffc83e9 WatchSource:0}: Error finding container cef1c829c69788b7250de41e9938134e7131f339699207e7700d8c0a2ffc83e9: Status 404 returned error can't find the container with id cef1c829c69788b7250de41e9938134e7131f339699207e7700d8c0a2ffc83e9
Oct 30 13:25:35.919418 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:35.919372    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ddb271-789bc5d4c9-p97mb" podUID="b2bde24e-a303-4111-adf3-123d6df17c00" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:25:36.224370 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:36.224324    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40bd9b0b-54kfk" event={"ID":"73a2c7d3-f20c-4c98-8416-eec166f20c48","Type":"ContainerStarted","Data":"41a7b49c2fdd70344ac64bb7bd9f750517b497c6dadd7de00758025db0692658"}
Oct 30 13:25:36.224852 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:36.224396    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40bd9b0b-54kfk" event={"ID":"73a2c7d3-f20c-4c98-8416-eec166f20c48","Type":"ContainerStarted","Data":"cef1c829c69788b7250de41e9938134e7131f339699207e7700d8c0a2ffc83e9"}
Oct 30 13:25:36.273162 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:36.273111    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40bd9b0b-54kfk" podStartSLOduration=2.2730982060000002 podStartE2EDuration="2.273098206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:25:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:25:36.270048275 +0000 UTC m=+2035.436412923" watchObservedRunningTime="2026-10-30 13:25:36.273098206 +0000 UTC m=+2035.439462854"
Oct 30 13:25:37.236364 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:37.236324    4364 generic.go:356] "Generic (PLEG): container finished" podID="73a2c7d3-f20c-4c98-8416-eec166f20c48" containerID="41a7b49c2fdd70344ac64bb7bd9f750517b497c6dadd7de00758025db0692658" exitCode=0
Oct 30 13:25:37.236834 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:37.236408    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40bd9b0b-54kfk" event={"ID":"73a2c7d3-f20c-4c98-8416-eec166f20c48","Type":"ContainerDied","Data":"41a7b49c2fdd70344ac64bb7bd9f750517b497c6dadd7de00758025db0692658"}
Oct 30 13:25:37.702495 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:37.702457    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-pjfhj"]
Oct 30 13:25:37.702806 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:37.702698    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-pjfhj" podUID="aab4aabb-55a4-412c-85c9-bf76db9513b3" containerName="istio-proxy" containerID="cri-o://916993b4c62dd3962a5343045bbbfacbdc027c3eea9f349526aae17df2e30a91" gracePeriod=30
Oct 30 13:25:38.436968 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:38.436936    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40bd9b0b-54kfk"
Oct 30 13:25:38.502916 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:38.502821    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73a2c7d3-f20c-4c98-8416-eec166f20c48-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73a2c7d3-f20c-4c98-8416-eec166f20c48-maas-api-xks-ca\") pod \"73a2c7d3-f20c-4c98-8416-eec166f20c48\" (UID: \"73a2c7d3-f20c-4c98-8416-eec166f20c48\") "
Oct 30 13:25:38.503104 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:38.502957    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73a2c7d3-f20c-4c98-8416-eec166f20c48-kube-api-access-9xwv2\" (UniqueName: \"kubernetes.io/projected/73a2c7d3-f20c-4c98-8416-eec166f20c48-kube-api-access-9xwv2\") pod \"73a2c7d3-f20c-4c98-8416-eec166f20c48\" (UID: \"73a2c7d3-f20c-4c98-8416-eec166f20c48\") "
Oct 30 13:25:38.503178 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:38.503139    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73a2c7d3-f20c-4c98-8416-eec166f20c48-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73a2c7d3-f20c-4c98-8416-eec166f20c48-maas-api-openshift-ca\") pod \"73a2c7d3-f20c-4c98-8416-eec166f20c48\" (UID: \"73a2c7d3-f20c-4c98-8416-eec166f20c48\") "
Oct 30 13:25:38.503572 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:38.503536    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73a2c7d3-f20c-4c98-8416-eec166f20c48-maas-api-openshift-ca" pod "73a2c7d3-f20c-4c98-8416-eec166f20c48" (UID: "73a2c7d3-f20c-4c98-8416-eec166f20c48"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:25:38.504093 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:38.504068    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73a2c7d3-f20c-4c98-8416-eec166f20c48-maas-api-openshift-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:25:38.505264 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:38.505238    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73a2c7d3-f20c-4c98-8416-eec166f20c48-maas-api-xks-ca" pod "73a2c7d3-f20c-4c98-8416-eec166f20c48" (UID: "73a2c7d3-f20c-4c98-8416-eec166f20c48"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:25:38.505634 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:38.505592    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73a2c7d3-f20c-4c98-8416-eec166f20c48-kube-api-access-9xwv2" pod "73a2c7d3-f20c-4c98-8416-eec166f20c48" (UID: "73a2c7d3-f20c-4c98-8416-eec166f20c48"). InnerVolumeSpecName "kube-api-access-9xwv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:25:38.612782 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:38.611716    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73a2c7d3-f20c-4c98-8416-eec166f20c48-maas-api-xks-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:25:38.612782 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:38.611829    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xwv2\" (UniqueName: \"kubernetes.io/projected/73a2c7d3-f20c-4c98-8416-eec166f20c48-kube-api-access-9xwv2\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:25:39.256791 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:39.256707    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40bd9b0b-54kfk" event={"ID":"73a2c7d3-f20c-4c98-8416-eec166f20c48","Type":"ContainerDied","Data":"cef1c829c69788b7250de41e9938134e7131f339699207e7700d8c0a2ffc83e9"}
Oct 30 13:25:39.256791 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:39.256735    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40bd9b0b-54kfk"
Oct 30 13:25:39.256791 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:39.256757    4364 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cef1c829c69788b7250de41e9938134e7131f339699207e7700d8c0a2ffc83e9"
Oct 30 13:25:41.506046 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:25:41.506007    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:25:42.723262 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:42.723233    4364 scope.go:122] "RemoveContainer" containerID="b146f090c42801191e551e76cfefed623e03a5676c488624db81f6a54666589d"
Oct 30 13:25:43.008902 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:43.008877    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-pjfhj"
Oct 30 13:25:43.110576 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:43.110539    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aab4aabb-55a4-412c-85c9-bf76db9513b3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aab4aabb-55a4-412c-85c9-bf76db9513b3-istiod-ca-cert\") pod \"aab4aabb-55a4-412c-85c9-bf76db9513b3\" (UID: \"aab4aabb-55a4-412c-85c9-bf76db9513b3\") "
Oct 30 13:25:43.110828 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:43.110716    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aab4aabb-55a4-412c-85c9-bf76db9513b3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aab4aabb-55a4-412c-85c9-bf76db9513b3-workload-certs\") pod \"aab4aabb-55a4-412c-85c9-bf76db9513b3\" (UID: \"aab4aabb-55a4-412c-85c9-bf76db9513b3\") "
Oct 30 13:25:43.110828 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:43.110749    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aab4aabb-55a4-412c-85c9-bf76db9513b3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aab4aabb-55a4-412c-85c9-bf76db9513b3-credential-socket\") pod \"aab4aabb-55a4-412c-85c9-bf76db9513b3\" (UID: \"aab4aabb-55a4-412c-85c9-bf76db9513b3\") "
Oct 30 13:25:43.110828 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:43.110788    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aab4aabb-55a4-412c-85c9-bf76db9513b3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aab4aabb-55a4-412c-85c9-bf76db9513b3-istio-data\") pod \"aab4aabb-55a4-412c-85c9-bf76db9513b3\" (UID: \"aab4aabb-55a4-412c-85c9-bf76db9513b3\") "
Oct 30 13:25:43.110828 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:43.110819    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aab4aabb-55a4-412c-85c9-bf76db9513b3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aab4aabb-55a4-412c-85c9-bf76db9513b3-istio-envoy\") pod \"aab4aabb-55a4-412c-85c9-bf76db9513b3\" (UID: \"aab4aabb-55a4-412c-85c9-bf76db9513b3\") "
Oct 30 13:25:43.111075 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:43.110875    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aab4aabb-55a4-412c-85c9-bf76db9513b3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aab4aabb-55a4-412c-85c9-bf76db9513b3-workload-socket\") pod \"aab4aabb-55a4-412c-85c9-bf76db9513b3\" (UID: \"aab4aabb-55a4-412c-85c9-bf76db9513b3\") "
Oct 30 13:25:43.111075 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:43.110980    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aab4aabb-55a4-412c-85c9-bf76db9513b3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aab4aabb-55a4-412c-85c9-bf76db9513b3-istio-podinfo\") pod \"aab4aabb-55a4-412c-85c9-bf76db9513b3\" (UID: \"aab4aabb-55a4-412c-85c9-bf76db9513b3\") "
Oct 30 13:25:43.111188 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:43.111074    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aab4aabb-55a4-412c-85c9-bf76db9513b3-credential-socket" pod "aab4aabb-55a4-412c-85c9-bf76db9513b3" (UID: "aab4aabb-55a4-412c-85c9-bf76db9513b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:25:43.111188 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:43.111091    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aab4aabb-55a4-412c-85c9-bf76db9513b3-workload-certs" pod "aab4aabb-55a4-412c-85c9-bf76db9513b3" (UID: "aab4aabb-55a4-412c-85c9-bf76db9513b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:25:43.111188 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:43.111085    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aab4aabb-55a4-412c-85c9-bf76db9513b3-istiod-ca-cert" pod "aab4aabb-55a4-412c-85c9-bf76db9513b3" (UID: "aab4aabb-55a4-412c-85c9-bf76db9513b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:25:43.111376 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:43.111250    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aab4aabb-55a4-412c-85c9-bf76db9513b3-kube-api-access-pxqgw\" (UniqueName: \"kubernetes.io/projected/aab4aabb-55a4-412c-85c9-bf76db9513b3-kube-api-access-pxqgw\") pod \"aab4aabb-55a4-412c-85c9-bf76db9513b3\" (UID: \"aab4aabb-55a4-412c-85c9-bf76db9513b3\") "
Oct 30 13:25:43.111376 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:43.111296    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aab4aabb-55a4-412c-85c9-bf76db9513b3-istio-token\" (UniqueName: \"kubernetes.io/projected/aab4aabb-55a4-412c-85c9-bf76db9513b3-istio-token\") pod \"aab4aabb-55a4-412c-85c9-bf76db9513b3\" (UID: \"aab4aabb-55a4-412c-85c9-bf76db9513b3\") "
Oct 30 13:25:43.112457 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:43.112131    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aab4aabb-55a4-412c-85c9-bf76db9513b3-workload-socket" pod "aab4aabb-55a4-412c-85c9-bf76db9513b3" (UID: "aab4aabb-55a4-412c-85c9-bf76db9513b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:25:43.112457 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:43.111400    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aab4aabb-55a4-412c-85c9-bf76db9513b3-istio-data" pod "aab4aabb-55a4-412c-85c9-bf76db9513b3" (UID: "aab4aabb-55a4-412c-85c9-bf76db9513b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:25:43.113388 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:43.113284    4364 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aab4aabb-55a4-412c-85c9-bf76db9513b3-istiod-ca-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:25:43.113388 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:43.113306    4364 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aab4aabb-55a4-412c-85c9-bf76db9513b3-workload-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:25:43.113388 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:43.113321    4364 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aab4aabb-55a4-412c-85c9-bf76db9513b3-credential-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:25:43.113388 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:43.113344    4364 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aab4aabb-55a4-412c-85c9-bf76db9513b3-istio-data\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:25:43.113388 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:43.113360    4364 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aab4aabb-55a4-412c-85c9-bf76db9513b3-workload-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:25:43.141223 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:43.140425    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aab4aabb-55a4-412c-85c9-bf76db9513b3-istio-podinfo" pod "aab4aabb-55a4-412c-85c9-bf76db9513b3" (UID: "aab4aabb-55a4-412c-85c9-bf76db9513b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:25:43.141362 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:43.141288    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aab4aabb-55a4-412c-85c9-bf76db9513b3-istio-envoy" pod "aab4aabb-55a4-412c-85c9-bf76db9513b3" (UID: "aab4aabb-55a4-412c-85c9-bf76db9513b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:25:43.141449 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:43.141419    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aab4aabb-55a4-412c-85c9-bf76db9513b3-istio-token" pod "aab4aabb-55a4-412c-85c9-bf76db9513b3" (UID: "aab4aabb-55a4-412c-85c9-bf76db9513b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:25:43.143150 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:43.143120    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aab4aabb-55a4-412c-85c9-bf76db9513b3-kube-api-access-pxqgw" pod "aab4aabb-55a4-412c-85c9-bf76db9513b3" (UID: "aab4aabb-55a4-412c-85c9-bf76db9513b3"). InnerVolumeSpecName "kube-api-access-pxqgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:25:43.215491 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:43.215454    4364 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aab4aabb-55a4-412c-85c9-bf76db9513b3-istio-envoy\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:25:43.215491 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:43.215487    4364 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aab4aabb-55a4-412c-85c9-bf76db9513b3-istio-podinfo\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:25:43.215491 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:43.215498    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxqgw\" (UniqueName: \"kubernetes.io/projected/aab4aabb-55a4-412c-85c9-bf76db9513b3-kube-api-access-pxqgw\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:25:43.215770 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:43.215507    4364 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aab4aabb-55a4-412c-85c9-bf76db9513b3-istio-token\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:25:43.352373 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:43.352282    4364 generic.go:356] "Generic (PLEG): container finished" podID="aab4aabb-55a4-412c-85c9-bf76db9513b3" containerID="916993b4c62dd3962a5343045bbbfacbdc027c3eea9f349526aae17df2e30a91" exitCode=0
Oct 30 13:25:43.352373 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:43.352358    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-pjfhj"
Oct 30 13:25:43.352373 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:43.352364    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-pjfhj" event={"ID":"aab4aabb-55a4-412c-85c9-bf76db9513b3","Type":"ContainerDied","Data":"916993b4c62dd3962a5343045bbbfacbdc027c3eea9f349526aae17df2e30a91"}
Oct 30 13:25:43.352678 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:43.352451    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-pjfhj" event={"ID":"aab4aabb-55a4-412c-85c9-bf76db9513b3","Type":"ContainerDied","Data":"88f471f2becd01891447d7a096259002c63ba8f92b37fdcb17c5f675a9f61d85"}
Oct 30 13:25:43.352678 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:43.352485    4364 scope.go:122] "RemoveContainer" containerID="916993b4c62dd3962a5343045bbbfacbdc027c3eea9f349526aae17df2e30a91"
Oct 30 13:25:43.366244 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:43.366224    4364 scope.go:122] "RemoveContainer" containerID="916993b4c62dd3962a5343045bbbfacbdc027c3eea9f349526aae17df2e30a91"
Oct 30 13:25:43.366535 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:25:43.366514    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"916993b4c62dd3962a5343045bbbfacbdc027c3eea9f349526aae17df2e30a91\": container with ID starting with 916993b4c62dd3962a5343045bbbfacbdc027c3eea9f349526aae17df2e30a91 not found: ID does not exist" containerID="916993b4c62dd3962a5343045bbbfacbdc027c3eea9f349526aae17df2e30a91"
Oct 30 13:25:43.366644 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:43.366547    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"916993b4c62dd3962a5343045bbbfacbdc027c3eea9f349526aae17df2e30a91"} err="failed to get container status \"916993b4c62dd3962a5343045bbbfacbdc027c3eea9f349526aae17df2e30a91\": rpc error: code = NotFound desc = could not find container \"916993b4c62dd3962a5343045bbbfacbdc027c3eea9f349526aae17df2e30a91\": container with ID starting with 916993b4c62dd3962a5343045bbbfacbdc027c3eea9f349526aae17df2e30a91 not found: ID does not exist"
Oct 30 13:25:43.408162 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:43.408134    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-pjfhj"]
Oct 30 13:25:43.419851 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:43.419826    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-pjfhj"]
Oct 30 13:25:44.973315 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:44.973280    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aab4aabb-55a4-412c-85c9-bf76db9513b3" path="/var/lib/kubelet/pods/aab4aabb-55a4-412c-85c9-bf76db9513b3/volumes"
Oct 30 13:25:45.923434 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:45.923386    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ddb271-789bc5d4c9-p97mb" podUID="b2bde24e-a303-4111-adf3-123d6df17c00" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:25:45.957527 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:45.957487    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-31721c/test-model-31721c-kserve-64c44bfc-dmnrj" podUID="ac457ffd-caeb-419f-b448-1b21942ead0f" containerName="main" containerID="cri-o://15677d30c7a218cfb710b093bd2a792102cd52cadff96f8edcd5bc23c16a1097" gracePeriod=45
Oct 30 13:25:46.319592 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:46.319564    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-31721c/test-model-31721c-kserve-64c44bfc-dmnrj"
Oct 30 13:25:46.387225 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:46.387184    4364 generic.go:356] "Generic (PLEG): container finished" podID="ac457ffd-caeb-419f-b448-1b21942ead0f" containerID="15677d30c7a218cfb710b093bd2a792102cd52cadff96f8edcd5bc23c16a1097" exitCode=0
Oct 30 13:25:46.387423 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:46.387235    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-31721c/test-model-31721c-kserve-64c44bfc-dmnrj"
Oct 30 13:25:46.387423 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:46.387242    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-31721c/test-model-31721c-kserve-64c44bfc-dmnrj" event={"ID":"ac457ffd-caeb-419f-b448-1b21942ead0f","Type":"ContainerDied","Data":"15677d30c7a218cfb710b093bd2a792102cd52cadff96f8edcd5bc23c16a1097"}
Oct 30 13:25:46.387423 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:46.387300    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-31721c/test-model-31721c-kserve-64c44bfc-dmnrj" event={"ID":"ac457ffd-caeb-419f-b448-1b21942ead0f","Type":"ContainerDied","Data":"763f85de5cadb71a16518c209e5fe71346ba02b5298107b618186d12e62f096d"}
Oct 30 13:25:46.387423 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:46.387327    4364 scope.go:122] "RemoveContainer" containerID="15677d30c7a218cfb710b093bd2a792102cd52cadff96f8edcd5bc23c16a1097"
Oct 30 13:25:46.400383 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:46.400359    4364 scope.go:122] "RemoveContainer" containerID="15677d30c7a218cfb710b093bd2a792102cd52cadff96f8edcd5bc23c16a1097"
Oct 30 13:25:46.400654 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:25:46.400631    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15677d30c7a218cfb710b093bd2a792102cd52cadff96f8edcd5bc23c16a1097\": container with ID starting with 15677d30c7a218cfb710b093bd2a792102cd52cadff96f8edcd5bc23c16a1097 not found: ID does not exist" containerID="15677d30c7a218cfb710b093bd2a792102cd52cadff96f8edcd5bc23c16a1097"
Oct 30 13:25:46.400739 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:46.400661    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15677d30c7a218cfb710b093bd2a792102cd52cadff96f8edcd5bc23c16a1097"} err="failed to get container status \"15677d30c7a218cfb710b093bd2a792102cd52cadff96f8edcd5bc23c16a1097\": rpc error: code = NotFound desc = could not find container \"15677d30c7a218cfb710b093bd2a792102cd52cadff96f8edcd5bc23c16a1097\": container with ID starting with 15677d30c7a218cfb710b093bd2a792102cd52cadff96f8edcd5bc23c16a1097 not found: ID does not exist"
Oct 30 13:25:46.507424 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:46.507389    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac457ffd-caeb-419f-b448-1b21942ead0f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac457ffd-caeb-419f-b448-1b21942ead0f-tmp-dir\") pod \"ac457ffd-caeb-419f-b448-1b21942ead0f\" (UID: \"ac457ffd-caeb-419f-b448-1b21942ead0f\") "
Oct 30 13:25:46.507661 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:46.507500    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac457ffd-caeb-419f-b448-1b21942ead0f-home\" (UniqueName: \"kubernetes.io/empty-dir/ac457ffd-caeb-419f-b448-1b21942ead0f-home\") pod \"ac457ffd-caeb-419f-b448-1b21942ead0f\" (UID: \"ac457ffd-caeb-419f-b448-1b21942ead0f\") "
Oct 30 13:25:46.507661 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:46.507561    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac457ffd-caeb-419f-b448-1b21942ead0f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac457ffd-caeb-419f-b448-1b21942ead0f-model-cache\") pod \"ac457ffd-caeb-419f-b448-1b21942ead0f\" (UID: \"ac457ffd-caeb-419f-b448-1b21942ead0f\") "
Oct 30 13:25:46.507661 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:46.507594    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac457ffd-caeb-419f-b448-1b21942ead0f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac457ffd-caeb-419f-b448-1b21942ead0f-dshm\") pod \"ac457ffd-caeb-419f-b448-1b21942ead0f\" (UID: \"ac457ffd-caeb-419f-b448-1b21942ead0f\") "
Oct 30 13:25:46.507860 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:46.507685    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac457ffd-caeb-419f-b448-1b21942ead0f-tls-certs\" (UniqueName: \"kubernetes.io/secret/ac457ffd-caeb-419f-b448-1b21942ead0f-tls-certs\") pod \"ac457ffd-caeb-419f-b448-1b21942ead0f\" (UID: \"ac457ffd-caeb-419f-b448-1b21942ead0f\") "
Oct 30 13:25:46.507860 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:46.507696    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac457ffd-caeb-419f-b448-1b21942ead0f-tmp-dir" pod "ac457ffd-caeb-419f-b448-1b21942ead0f" (UID: "ac457ffd-caeb-419f-b448-1b21942ead0f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:25:46.507860 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:46.507791    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac457ffd-caeb-419f-b448-1b21942ead0f-kube-api-access-tt984\" (UniqueName: \"kubernetes.io/projected/ac457ffd-caeb-419f-b448-1b21942ead0f-kube-api-access-tt984\") pod \"ac457ffd-caeb-419f-b448-1b21942ead0f\" (UID: \"ac457ffd-caeb-419f-b448-1b21942ead0f\") "
Oct 30 13:25:46.507860 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:46.507840    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac457ffd-caeb-419f-b448-1b21942ead0f-model-cache" pod "ac457ffd-caeb-419f-b448-1b21942ead0f" (UID: "ac457ffd-caeb-419f-b448-1b21942ead0f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:25:46.508028 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:46.507855    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac457ffd-caeb-419f-b448-1b21942ead0f-home" pod "ac457ffd-caeb-419f-b448-1b21942ead0f" (UID: "ac457ffd-caeb-419f-b448-1b21942ead0f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:25:46.508571 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:46.508546    4364 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac457ffd-caeb-419f-b448-1b21942ead0f-home\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:25:46.508571 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:46.508569    4364 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac457ffd-caeb-419f-b448-1b21942ead0f-model-cache\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:25:46.508571 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:46.508578    4364 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac457ffd-caeb-419f-b448-1b21942ead0f-tmp-dir\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:25:46.510059 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:46.510033    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac457ffd-caeb-419f-b448-1b21942ead0f-tls-certs" pod "ac457ffd-caeb-419f-b448-1b21942ead0f" (UID: "ac457ffd-caeb-419f-b448-1b21942ead0f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:25:46.510059 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:46.510034    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac457ffd-caeb-419f-b448-1b21942ead0f-kube-api-access-tt984" pod "ac457ffd-caeb-419f-b448-1b21942ead0f" (UID: "ac457ffd-caeb-419f-b448-1b21942ead0f"). InnerVolumeSpecName "kube-api-access-tt984". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:25:46.510172 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:46.510119    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac457ffd-caeb-419f-b448-1b21942ead0f-dshm" pod "ac457ffd-caeb-419f-b448-1b21942ead0f" (UID: "ac457ffd-caeb-419f-b448-1b21942ead0f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:25:46.610214 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:46.610177    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tt984\" (UniqueName: \"kubernetes.io/projected/ac457ffd-caeb-419f-b448-1b21942ead0f-kube-api-access-tt984\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:25:46.610214 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:46.610205    4364 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac457ffd-caeb-419f-b448-1b21942ead0f-dshm\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:25:46.610214 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:46.610214    4364 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac457ffd-caeb-419f-b448-1b21942ead0f-tls-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:25:46.735974 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:46.735941    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-31721c/test-model-31721c-kserve-64c44bfc-dmnrj"]
Oct 30 13:25:46.747322 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:46.747291    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-31721c/test-model-31721c-kserve-64c44bfc-dmnrj"]
Oct 30 13:25:46.972899 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:46.972855    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac457ffd-caeb-419f-b448-1b21942ead0f" path="/var/lib/kubelet/pods/ac457ffd-caeb-419f-b448-1b21942ead0f/volumes"
Oct 30 13:25:49.685490 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:49.685454    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:25:52.554235 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:52.553912    4364 generic.go:356] "Generic (PLEG): container finished" podID="b2bde24e-a303-4111-adf3-123d6df17c00" containerID="9852aa57b3462a6707f95d9d51449df20f381de671eb9faa42d6d79d66b0702b" exitCode=0
Oct 30 13:25:52.554235 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:52.554115    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ddb271-789bc5d4c9-p97mb" event={"ID":"b2bde24e-a303-4111-adf3-123d6df17c00","Type":"ContainerDied","Data":"9852aa57b3462a6707f95d9d51449df20f381de671eb9faa42d6d79d66b0702b"}
Oct 30 13:25:52.800774 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:52.800743    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ddb271-789bc5d4c9-p97mb"
Oct 30 13:25:52.920648 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:52.915183    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2bde24e-a303-4111-adf3-123d6df17c00-kube-api-access-pls2t\" (UniqueName: \"kubernetes.io/projected/b2bde24e-a303-4111-adf3-123d6df17c00-kube-api-access-pls2t\") pod \"b2bde24e-a303-4111-adf3-123d6df17c00\" (UID: \"b2bde24e-a303-4111-adf3-123d6df17c00\") "
Oct 30 13:25:52.920648 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:52.915255    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2bde24e-a303-4111-adf3-123d6df17c00-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2bde24e-a303-4111-adf3-123d6df17c00-plugins-config-volume\") pod \"b2bde24e-a303-4111-adf3-123d6df17c00\" (UID: \"b2bde24e-a303-4111-adf3-123d6df17c00\") "
Oct 30 13:25:52.920648 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:52.917145    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2bde24e-a303-4111-adf3-123d6df17c00-plugins-config-volume" pod "b2bde24e-a303-4111-adf3-123d6df17c00" (UID: "b2bde24e-a303-4111-adf3-123d6df17c00"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:25:52.920648 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:52.919813    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2bde24e-a303-4111-adf3-123d6df17c00-kube-api-access-pls2t" pod "b2bde24e-a303-4111-adf3-123d6df17c00" (UID: "b2bde24e-a303-4111-adf3-123d6df17c00"). InnerVolumeSpecName "kube-api-access-pls2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:25:53.018174 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:53.018087    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pls2t\" (UniqueName: \"kubernetes.io/projected/b2bde24e-a303-4111-adf3-123d6df17c00-kube-api-access-pls2t\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:25:53.018174 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:53.018125    4364 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2bde24e-a303-4111-adf3-123d6df17c00-plugins-config-volume\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:25:53.563941 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:53.563912    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ddb271-789bc5d4c9-p97mb"
Oct 30 13:25:53.564411 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:53.563942    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ddb271-789bc5d4c9-p97mb" event={"ID":"b2bde24e-a303-4111-adf3-123d6df17c00","Type":"ContainerDied","Data":"0c4a8b9aad98ba42d7e3586f03d202c6a01f552c3cea850a27ec0d2a18c6cfd0"}
Oct 30 13:25:53.564411 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:53.564006    4364 scope.go:122] "RemoveContainer" containerID="9852aa57b3462a6707f95d9d51449df20f381de671eb9faa42d6d79d66b0702b"
Oct 30 13:25:53.618069 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:53.618035    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ddb271-789bc5d4c9-p97mb"]
Oct 30 13:25:53.624887 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:53.624859    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ddb271-789bc5d4c9-p97mb"]
Oct 30 13:25:54.973954 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:25:54.973921    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2bde24e-a303-4111-adf3-123d6df17c00" path="/var/lib/kubelet/pods/b2bde24e-a303-4111-adf3-123d6df17c00/volumes"
Oct 30 13:26:06.046263 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:06.046213    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-31721c-kcb4z"]
Oct 30 13:26:06.047572 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:06.047545    4364 cpu_manager.go:414] "containerMap: removing container" podUID="aab4aabb-55a4-412c-85c9-bf76db9513b3" containerName="istio-proxy"
Oct 30 13:26:06.047572 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:06.047572    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aab4aabb-55a4-412c-85c9-bf76db9513b3" containerName="istio-proxy"
Oct 30 13:26:06.047775 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:06.047619    4364 cpu_manager.go:414] "containerMap: removing container" podUID="b2bde24e-a303-4111-adf3-123d6df17c00" containerName="payload-pre-processing"
Oct 30 13:26:06.047775 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:06.047629    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2bde24e-a303-4111-adf3-123d6df17c00" containerName="payload-pre-processing"
Oct 30 13:26:06.047775 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:06.047666    4364 cpu_manager.go:414] "containerMap: removing container" podUID="ac457ffd-caeb-419f-b448-1b21942ead0f" containerName="main"
Oct 30 13:26:06.047775 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:06.047674    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac457ffd-caeb-419f-b448-1b21942ead0f" containerName="main"
Oct 30 13:26:06.047775 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:06.047687    4364 cpu_manager.go:414] "containerMap: removing container" podUID="73a2c7d3-f20c-4c98-8416-eec166f20c48" containerName="revoke-keys"
Oct 30 13:26:06.047775 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:06.047694    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73a2c7d3-f20c-4c98-8416-eec166f20c48" containerName="revoke-keys"
Oct 30 13:26:06.048114 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:06.047901    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2bde24e-a303-4111-adf3-123d6df17c00" containerName="payload-pre-processing"
Oct 30 13:26:06.048114 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:06.047920    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac457ffd-caeb-419f-b448-1b21942ead0f" containerName="main"
Oct 30 13:26:06.048114 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:06.047930    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="73a2c7d3-f20c-4c98-8416-eec166f20c48" containerName="revoke-keys"
Oct 30 13:26:06.048114 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:06.047947    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="aab4aabb-55a4-412c-85c9-bf76db9513b3" containerName="istio-proxy"
Oct 30 13:26:06.054001 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:06.053979    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-31721c-kcb4z"
Oct 30 13:26:06.068670 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:06.068527    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-31721c-kcb4z"]
Oct 30 13:26:06.073756 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:06.073036    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:26:06.073756 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:06.073143    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\""
Oct 30 13:26:06.185801 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:06.185766    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6041e6f1-3ee9-45b8-bbc5-f6c73436f7f5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-31721c-kcb4z\" (UID: \"6041e6f1-3ee9-45b8-bbc5-f6c73436f7f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-31721c-kcb4z"
Oct 30 13:26:06.186058 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:06.186042    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzbs2\" (UniqueName: \"kubernetes.io/projected/6041e6f1-3ee9-45b8-bbc5-f6c73436f7f5-kube-api-access-mzbs2\") pod \"maas-api-revoke-keys-e2e-inf-a-31721c-kcb4z\" (UID: \"6041e6f1-3ee9-45b8-bbc5-f6c73436f7f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-31721c-kcb4z"
Oct 30 13:26:06.186152 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:06.186139    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6041e6f1-3ee9-45b8-bbc5-f6c73436f7f5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-31721c-kcb4z\" (UID: \"6041e6f1-3ee9-45b8-bbc5-f6c73436f7f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-31721c-kcb4z"
Oct 30 13:26:06.255225 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:06.255188    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-b5a369/test-model-b5a369-kserve-6bfb684495-gckcs"]
Oct 30 13:26:06.287618 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:06.287575    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6041e6f1-3ee9-45b8-bbc5-f6c73436f7f5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-31721c-kcb4z\" (UID: \"6041e6f1-3ee9-45b8-bbc5-f6c73436f7f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-31721c-kcb4z"
Oct 30 13:26:06.287812 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:06.287685    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6041e6f1-3ee9-45b8-bbc5-f6c73436f7f5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-31721c-kcb4z\" (UID: \"6041e6f1-3ee9-45b8-bbc5-f6c73436f7f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-31721c-kcb4z"
Oct 30 13:26:06.288045 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:06.288021    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzbs2\" (UniqueName: \"kubernetes.io/projected/6041e6f1-3ee9-45b8-bbc5-f6c73436f7f5-kube-api-access-mzbs2\") pod \"maas-api-revoke-keys-e2e-inf-a-31721c-kcb4z\" (UID: \"6041e6f1-3ee9-45b8-bbc5-f6c73436f7f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-31721c-kcb4z"
Oct 30 13:26:06.288404 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:06.288386    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6041e6f1-3ee9-45b8-bbc5-f6c73436f7f5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-31721c-kcb4z\" (UID: \"6041e6f1-3ee9-45b8-bbc5-f6c73436f7f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-31721c-kcb4z"
Oct 30 13:26:06.289843 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:06.289820    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6041e6f1-3ee9-45b8-bbc5-f6c73436f7f5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-31721c-kcb4z\" (UID: \"6041e6f1-3ee9-45b8-bbc5-f6c73436f7f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-31721c-kcb4z"
Oct 30 13:26:06.311796 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:06.311735    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzbs2\" (UniqueName: \"kubernetes.io/projected/6041e6f1-3ee9-45b8-bbc5-f6c73436f7f5-kube-api-access-mzbs2\") pod \"maas-api-revoke-keys-e2e-inf-a-31721c-kcb4z\" (UID: \"6041e6f1-3ee9-45b8-bbc5-f6c73436f7f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-31721c-kcb4z"
Oct 30 13:26:06.382950 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:06.382910    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-31721c-kcb4z"
Oct 30 13:26:06.847904 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:06.847817    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-280238be-w76vf"]
Oct 30 13:26:06.852896 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:06.852876    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-280238be-w76vf"
Oct 30 13:26:06.870717 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:06.870691    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-31721c-kcb4z"]
Oct 30 13:26:06.879959 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:26:06.879933    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6041e6f1_3ee9_45b8_bbc5_f6c73436f7f5.slice/crio-fab785e947758552d3966cb00022a7f80d3a2c39ea0f540571405a4feb7b6ab5 WatchSource:0}: Error finding container fab785e947758552d3966cb00022a7f80d3a2c39ea0f540571405a4feb7b6ab5: Status 404 returned error can't find the container with id fab785e947758552d3966cb00022a7f80d3a2c39ea0f540571405a4feb7b6ab5
Oct 30 13:26:06.880192 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:06.880168    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-280238be-w76vf"]
Oct 30 13:26:07.019299 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:07.019269    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5895871b-7b5c-4ba0-8b6e-e761ca5b1adf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-280238be-w76vf\" (UID: \"5895871b-7b5c-4ba0-8b6e-e761ca5b1adf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-280238be-w76vf"
Oct 30 13:26:07.019457 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:07.019425    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cppx\" (UniqueName: \"kubernetes.io/projected/5895871b-7b5c-4ba0-8b6e-e761ca5b1adf-kube-api-access-5cppx\") pod \"maas-api-revoke-sub-280238be-w76vf\" (UID: \"5895871b-7b5c-4ba0-8b6e-e761ca5b1adf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-280238be-w76vf"
Oct 30 13:26:07.019524 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:07.019483    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5895871b-7b5c-4ba0-8b6e-e761ca5b1adf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-280238be-w76vf\" (UID: \"5895871b-7b5c-4ba0-8b6e-e761ca5b1adf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-280238be-w76vf"
Oct 30 13:26:07.122045 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:07.122010    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5895871b-7b5c-4ba0-8b6e-e761ca5b1adf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-280238be-w76vf\" (UID: \"5895871b-7b5c-4ba0-8b6e-e761ca5b1adf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-280238be-w76vf"
Oct 30 13:26:07.122560 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:07.122302    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cppx\" (UniqueName: \"kubernetes.io/projected/5895871b-7b5c-4ba0-8b6e-e761ca5b1adf-kube-api-access-5cppx\") pod \"maas-api-revoke-sub-280238be-w76vf\" (UID: \"5895871b-7b5c-4ba0-8b6e-e761ca5b1adf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-280238be-w76vf"
Oct 30 13:26:07.122560 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:07.122394    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5895871b-7b5c-4ba0-8b6e-e761ca5b1adf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-280238be-w76vf\" (UID: \"5895871b-7b5c-4ba0-8b6e-e761ca5b1adf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-280238be-w76vf"
Oct 30 13:26:07.123112 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:07.123082    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5895871b-7b5c-4ba0-8b6e-e761ca5b1adf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-280238be-w76vf\" (UID: \"5895871b-7b5c-4ba0-8b6e-e761ca5b1adf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-280238be-w76vf"
Oct 30 13:26:07.124569 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:07.124546    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5895871b-7b5c-4ba0-8b6e-e761ca5b1adf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-280238be-w76vf\" (UID: \"5895871b-7b5c-4ba0-8b6e-e761ca5b1adf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-280238be-w76vf"
Oct 30 13:26:07.147254 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:07.147222    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cppx\" (UniqueName: \"kubernetes.io/projected/5895871b-7b5c-4ba0-8b6e-e761ca5b1adf-kube-api-access-5cppx\") pod \"maas-api-revoke-sub-280238be-w76vf\" (UID: \"5895871b-7b5c-4ba0-8b6e-e761ca5b1adf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-280238be-w76vf"
Oct 30 13:26:07.172552 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:07.172521    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-280238be-w76vf"
Oct 30 13:26:07.496757 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:26:07.496722    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5895871b_7b5c_4ba0_8b6e_e761ca5b1adf.slice/crio-2bd95212f82a46dd01204f3ad0ad715ecdcf3f2d9889a5883b5c04ea54fa066e WatchSource:0}: Error finding container 2bd95212f82a46dd01204f3ad0ad715ecdcf3f2d9889a5883b5c04ea54fa066e: Status 404 returned error can't find the container with id 2bd95212f82a46dd01204f3ad0ad715ecdcf3f2d9889a5883b5c04ea54fa066e
Oct 30 13:26:07.496965 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:07.496939    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-280238be-w76vf"]
Oct 30 13:26:07.770933 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:07.770898    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-280238be-w76vf" event={"ID":"5895871b-7b5c-4ba0-8b6e-e761ca5b1adf","Type":"ContainerStarted","Data":"2bd95212f82a46dd01204f3ad0ad715ecdcf3f2d9889a5883b5c04ea54fa066e"}
Oct 30 13:26:07.773002 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:07.772965    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-31721c-kcb4z" event={"ID":"6041e6f1-3ee9-45b8-bbc5-f6c73436f7f5","Type":"ContainerStarted","Data":"7c9b0f2c2fa0e22e9f34203bda8d399630f5023140bb530bd45d6b63709156f9"}
Oct 30 13:26:07.773132 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:07.773025    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-31721c-kcb4z" event={"ID":"6041e6f1-3ee9-45b8-bbc5-f6c73436f7f5","Type":"ContainerStarted","Data":"fab785e947758552d3966cb00022a7f80d3a2c39ea0f540571405a4feb7b6ab5"}
Oct 30 13:26:07.825725 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:07.825677    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-31721c-kcb4z" podStartSLOduration=1.8256634360000001 podStartE2EDuration="1.825663436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:26:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:26:07.812128897 +0000 UTC m=+2066.978493545" watchObservedRunningTime="2026-10-30 13:26:07.825663436 +0000 UTC m=+2066.992028084"
Oct 30 13:26:08.784330 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:08.784241    4364 generic.go:356] "Generic (PLEG): container finished" podID="5895871b-7b5c-4ba0-8b6e-e761ca5b1adf" containerID="d4bd800a0da8f0df9a85a23307d387b612e2445c1c55bd147ac03d9a7ac9147e" exitCode=0
Oct 30 13:26:08.784772 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:08.784329    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-280238be-w76vf" event={"ID":"5895871b-7b5c-4ba0-8b6e-e761ca5b1adf","Type":"ContainerDied","Data":"d4bd800a0da8f0df9a85a23307d387b612e2445c1c55bd147ac03d9a7ac9147e"}
Oct 30 13:26:08.786202 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:08.786183    4364 generic.go:356] "Generic (PLEG): container finished" podID="6041e6f1-3ee9-45b8-bbc5-f6c73436f7f5" containerID="7c9b0f2c2fa0e22e9f34203bda8d399630f5023140bb530bd45d6b63709156f9" exitCode=0
Oct 30 13:26:08.786297 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:08.786212    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-31721c-kcb4z" event={"ID":"6041e6f1-3ee9-45b8-bbc5-f6c73436f7f5","Type":"ContainerDied","Data":"7c9b0f2c2fa0e22e9f34203bda8d399630f5023140bb530bd45d6b63709156f9"}
Oct 30 13:26:10.014577 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:10.014543    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-31721c-kcb4z"
Oct 30 13:26:10.057829 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:10.057765    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-280238be-w76vf"
Oct 30 13:26:10.085353 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:10.085313    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6041e6f1-3ee9-45b8-bbc5-f6c73436f7f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6041e6f1-3ee9-45b8-bbc5-f6c73436f7f5-maas-api-openshift-ca\") pod \"6041e6f1-3ee9-45b8-bbc5-f6c73436f7f5\" (UID: \"6041e6f1-3ee9-45b8-bbc5-f6c73436f7f5\") "
Oct 30 13:26:10.085515 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:10.085448    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6041e6f1-3ee9-45b8-bbc5-f6c73436f7f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6041e6f1-3ee9-45b8-bbc5-f6c73436f7f5-maas-api-xks-ca\") pod \"6041e6f1-3ee9-45b8-bbc5-f6c73436f7f5\" (UID: \"6041e6f1-3ee9-45b8-bbc5-f6c73436f7f5\") "
Oct 30 13:26:10.085617 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:10.085517    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5895871b-7b5c-4ba0-8b6e-e761ca5b1adf-kube-api-access-5cppx\" (UniqueName: \"kubernetes.io/projected/5895871b-7b5c-4ba0-8b6e-e761ca5b1adf-kube-api-access-5cppx\") pod \"5895871b-7b5c-4ba0-8b6e-e761ca5b1adf\" (UID: \"5895871b-7b5c-4ba0-8b6e-e761ca5b1adf\") "
Oct 30 13:26:10.085688 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:10.085627    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5895871b-7b5c-4ba0-8b6e-e761ca5b1adf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5895871b-7b5c-4ba0-8b6e-e761ca5b1adf-maas-api-xks-ca\") pod \"5895871b-7b5c-4ba0-8b6e-e761ca5b1adf\" (UID: \"5895871b-7b5c-4ba0-8b6e-e761ca5b1adf\") "
Oct 30 13:26:10.085754 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:10.085698    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6041e6f1-3ee9-45b8-bbc5-f6c73436f7f5-kube-api-access-mzbs2\" (UniqueName: \"kubernetes.io/projected/6041e6f1-3ee9-45b8-bbc5-f6c73436f7f5-kube-api-access-mzbs2\") pod \"6041e6f1-3ee9-45b8-bbc5-f6c73436f7f5\" (UID: \"6041e6f1-3ee9-45b8-bbc5-f6c73436f7f5\") "
Oct 30 13:26:10.085754 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:10.085715    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6041e6f1-3ee9-45b8-bbc5-f6c73436f7f5-maas-api-openshift-ca" pod "6041e6f1-3ee9-45b8-bbc5-f6c73436f7f5" (UID: "6041e6f1-3ee9-45b8-bbc5-f6c73436f7f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:26:10.085933 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:10.085912    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5895871b-7b5c-4ba0-8b6e-e761ca5b1adf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5895871b-7b5c-4ba0-8b6e-e761ca5b1adf-maas-api-openshift-ca\") pod \"5895871b-7b5c-4ba0-8b6e-e761ca5b1adf\" (UID: \"5895871b-7b5c-4ba0-8b6e-e761ca5b1adf\") "
Oct 30 13:26:10.086516 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:10.086470    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5895871b-7b5c-4ba0-8b6e-e761ca5b1adf-maas-api-openshift-ca" pod "5895871b-7b5c-4ba0-8b6e-e761ca5b1adf" (UID: "5895871b-7b5c-4ba0-8b6e-e761ca5b1adf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:26:10.088511 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:10.088482    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6041e6f1-3ee9-45b8-bbc5-f6c73436f7f5-kube-api-access-mzbs2" pod "6041e6f1-3ee9-45b8-bbc5-f6c73436f7f5" (UID: "6041e6f1-3ee9-45b8-bbc5-f6c73436f7f5"). InnerVolumeSpecName "kube-api-access-mzbs2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:26:10.089035 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:10.089007    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5895871b-7b5c-4ba0-8b6e-e761ca5b1adf-maas-api-xks-ca" pod "5895871b-7b5c-4ba0-8b6e-e761ca5b1adf" (UID: "5895871b-7b5c-4ba0-8b6e-e761ca5b1adf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:26:10.089397 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:10.089374    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5895871b-7b5c-4ba0-8b6e-e761ca5b1adf-maas-api-xks-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:26:10.089477 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:10.089405    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzbs2\" (UniqueName: \"kubernetes.io/projected/6041e6f1-3ee9-45b8-bbc5-f6c73436f7f5-kube-api-access-mzbs2\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:26:10.089477 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:10.089419    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5895871b-7b5c-4ba0-8b6e-e761ca5b1adf-maas-api-openshift-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:26:10.089477 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:10.089433    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6041e6f1-3ee9-45b8-bbc5-f6c73436f7f5-maas-api-openshift-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:26:10.089477 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:10.089459    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6041e6f1-3ee9-45b8-bbc5-f6c73436f7f5-maas-api-xks-ca" pod "6041e6f1-3ee9-45b8-bbc5-f6c73436f7f5" (UID: "6041e6f1-3ee9-45b8-bbc5-f6c73436f7f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:26:10.093808 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:10.093775    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5895871b-7b5c-4ba0-8b6e-e761ca5b1adf-kube-api-access-5cppx" pod "5895871b-7b5c-4ba0-8b6e-e761ca5b1adf" (UID: "5895871b-7b5c-4ba0-8b6e-e761ca5b1adf"). InnerVolumeSpecName "kube-api-access-5cppx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:26:10.191458 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:10.191425    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cppx\" (UniqueName: \"kubernetes.io/projected/5895871b-7b5c-4ba0-8b6e-e761ca5b1adf-kube-api-access-5cppx\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:26:10.191458 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:10.191456    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6041e6f1-3ee9-45b8-bbc5-f6c73436f7f5-maas-api-xks-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:26:10.809679 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:10.809650    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-31721c-kcb4z"
Oct 30 13:26:10.809887 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:10.809707    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-31721c-kcb4z" event={"ID":"6041e6f1-3ee9-45b8-bbc5-f6c73436f7f5","Type":"ContainerDied","Data":"fab785e947758552d3966cb00022a7f80d3a2c39ea0f540571405a4feb7b6ab5"}
Oct 30 13:26:10.809887 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:10.809767    4364 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fab785e947758552d3966cb00022a7f80d3a2c39ea0f540571405a4feb7b6ab5"
Oct 30 13:26:10.811585 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:10.811564    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-280238be-w76vf"
Oct 30 13:26:10.811750 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:10.811549    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-280238be-w76vf" event={"ID":"5895871b-7b5c-4ba0-8b6e-e761ca5b1adf","Type":"ContainerDied","Data":"2bd95212f82a46dd01204f3ad0ad715ecdcf3f2d9889a5883b5c04ea54fa066e"}
Oct 30 13:26:10.811750 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:10.811690    4364 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2bd95212f82a46dd01204f3ad0ad715ecdcf3f2d9889a5883b5c04ea54fa066e"
Oct 30 13:26:11.507100 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:26:11.507056    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:26:16.609751 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:16.609718    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-31721c-6cc5477859-4pglh"]
Oct 30 13:26:19.216591 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:19.216549    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-31721c-kcb4z"]
Oct 30 13:26:19.228128 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:19.228099    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-31721c-kcb4z"]
Oct 30 13:26:20.974239 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:20.974202    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6041e6f1-3ee9-45b8-bbc5-f6c73436f7f5" path="/var/lib/kubelet/pods/6041e6f1-3ee9-45b8-bbc5-f6c73436f7f5/volumes"
Oct 30 13:26:21.287988 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:21.287888    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-b5a369/test-model-b5a369-kserve-6bfb684495-gckcs" podUID="d52aab51-7fdf-4be6-8893-b5fe88dbe341" containerName="main" containerID="cri-o://fc806d441740c27c7d1faa15cca81d6050ee11934ff3d6733efdeccb24cadeef" gracePeriod=45
Oct 30 13:26:21.359499 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:21.359456    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-b5a369/test-model-b5a369-kserve-6bfb684495-gckcs" podUID="d52aab51-7fdf-4be6-8893-b5fe88dbe341" containerName="main" probeResult="failure" output="Get \"https://10.133.0.188:8000/ready\": dial tcp 10.133.0.188:8000: connect: connection refused"
Oct 30 13:26:21.618624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:21.617917    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-31721c-6cc5477859-4pglh" podUID="f9acff8a-e701-4ab6-ba2f-9c8484b7bdcb" containerName="payload-pre-processing" containerID="cri-o://724a012bc8418d5b787734379ce78cb0cf2836b453fc3c18c02d73d566878338" gracePeriod=25
Oct 30 13:26:21.824521 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:21.824496    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b5a369/test-model-b5a369-kserve-6bfb684495-gckcs"
Oct 30 13:26:21.881749 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:21.881665    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d52aab51-7fdf-4be6-8893-b5fe88dbe341-tls-certs\" (UniqueName: \"kubernetes.io/secret/d52aab51-7fdf-4be6-8893-b5fe88dbe341-tls-certs\") pod \"d52aab51-7fdf-4be6-8893-b5fe88dbe341\" (UID: \"d52aab51-7fdf-4be6-8893-b5fe88dbe341\") "
Oct 30 13:26:21.882291 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:21.882237    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d52aab51-7fdf-4be6-8893-b5fe88dbe341-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d52aab51-7fdf-4be6-8893-b5fe88dbe341-model-cache\") pod \"d52aab51-7fdf-4be6-8893-b5fe88dbe341\" (UID: \"d52aab51-7fdf-4be6-8893-b5fe88dbe341\") "
Oct 30 13:26:21.882412 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:21.882370    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d52aab51-7fdf-4be6-8893-b5fe88dbe341-kube-api-access-f4w4g\" (UniqueName: \"kubernetes.io/projected/d52aab51-7fdf-4be6-8893-b5fe88dbe341-kube-api-access-f4w4g\") pod \"d52aab51-7fdf-4be6-8893-b5fe88dbe341\" (UID: \"d52aab51-7fdf-4be6-8893-b5fe88dbe341\") "
Oct 30 13:26:21.882484 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:21.882413    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d52aab51-7fdf-4be6-8893-b5fe88dbe341-home\" (UniqueName: \"kubernetes.io/empty-dir/d52aab51-7fdf-4be6-8893-b5fe88dbe341-home\") pod \"d52aab51-7fdf-4be6-8893-b5fe88dbe341\" (UID: \"d52aab51-7fdf-4be6-8893-b5fe88dbe341\") "
Oct 30 13:26:21.882543 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:21.882520    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d52aab51-7fdf-4be6-8893-b5fe88dbe341-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d52aab51-7fdf-4be6-8893-b5fe88dbe341-dshm\") pod \"d52aab51-7fdf-4be6-8893-b5fe88dbe341\" (UID: \"d52aab51-7fdf-4be6-8893-b5fe88dbe341\") "
Oct 30 13:26:21.882902 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:21.882871    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d52aab51-7fdf-4be6-8893-b5fe88dbe341-home" pod "d52aab51-7fdf-4be6-8893-b5fe88dbe341" (UID: "d52aab51-7fdf-4be6-8893-b5fe88dbe341"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:26:21.883874 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:21.883361    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d52aab51-7fdf-4be6-8893-b5fe88dbe341-model-cache" pod "d52aab51-7fdf-4be6-8893-b5fe88dbe341" (UID: "d52aab51-7fdf-4be6-8893-b5fe88dbe341"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:26:21.884084 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:21.884060    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d52aab51-7fdf-4be6-8893-b5fe88dbe341-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d52aab51-7fdf-4be6-8893-b5fe88dbe341-tmp-dir\") pod \"d52aab51-7fdf-4be6-8893-b5fe88dbe341\" (UID: \"d52aab51-7fdf-4be6-8893-b5fe88dbe341\") "
Oct 30 13:26:21.892649 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:21.885714    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d52aab51-7fdf-4be6-8893-b5fe88dbe341-dshm" pod "d52aab51-7fdf-4be6-8893-b5fe88dbe341" (UID: "d52aab51-7fdf-4be6-8893-b5fe88dbe341"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:26:21.892649 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:21.886138    4364 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d52aab51-7fdf-4be6-8893-b5fe88dbe341-home\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:26:21.892649 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:21.886159    4364 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d52aab51-7fdf-4be6-8893-b5fe88dbe341-dshm\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:26:21.892649 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:21.886173    4364 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d52aab51-7fdf-4be6-8893-b5fe88dbe341-model-cache\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:26:21.892649 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:21.886728    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d52aab51-7fdf-4be6-8893-b5fe88dbe341-tmp-dir" pod "d52aab51-7fdf-4be6-8893-b5fe88dbe341" (UID: "d52aab51-7fdf-4be6-8893-b5fe88dbe341"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:26:21.892649 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:21.888301    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d52aab51-7fdf-4be6-8893-b5fe88dbe341-tls-certs" pod "d52aab51-7fdf-4be6-8893-b5fe88dbe341" (UID: "d52aab51-7fdf-4be6-8893-b5fe88dbe341"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:26:21.892649 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:21.891508    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d52aab51-7fdf-4be6-8893-b5fe88dbe341-kube-api-access-f4w4g" pod "d52aab51-7fdf-4be6-8893-b5fe88dbe341" (UID: "d52aab51-7fdf-4be6-8893-b5fe88dbe341"). InnerVolumeSpecName "kube-api-access-f4w4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:26:21.988225 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:21.988187    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4w4g\" (UniqueName: \"kubernetes.io/projected/d52aab51-7fdf-4be6-8893-b5fe88dbe341-kube-api-access-f4w4g\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:26:21.988225 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:21.988220    4364 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d52aab51-7fdf-4be6-8893-b5fe88dbe341-tmp-dir\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:26:21.988225 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:21.988230    4364 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d52aab51-7fdf-4be6-8893-b5fe88dbe341-tls-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:26:22.070779 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:22.070738    4364 generic.go:356] "Generic (PLEG): container finished" podID="d52aab51-7fdf-4be6-8893-b5fe88dbe341" containerID="fc806d441740c27c7d1faa15cca81d6050ee11934ff3d6733efdeccb24cadeef" exitCode=0
Oct 30 13:26:22.070993 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:22.070792    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b5a369/test-model-b5a369-kserve-6bfb684495-gckcs" event={"ID":"d52aab51-7fdf-4be6-8893-b5fe88dbe341","Type":"ContainerDied","Data":"fc806d441740c27c7d1faa15cca81d6050ee11934ff3d6733efdeccb24cadeef"}
Oct 30 13:26:22.070993 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:22.070826    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b5a369/test-model-b5a369-kserve-6bfb684495-gckcs"
Oct 30 13:26:22.070993 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:22.070848    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b5a369/test-model-b5a369-kserve-6bfb684495-gckcs" event={"ID":"d52aab51-7fdf-4be6-8893-b5fe88dbe341","Type":"ContainerDied","Data":"3942eae0bcda38852a2326dc5be15070eb404d2c68786b6681ad046d05b1a979"}
Oct 30 13:26:22.070993 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:22.070880    4364 scope.go:122] "RemoveContainer" containerID="fc806d441740c27c7d1faa15cca81d6050ee11934ff3d6733efdeccb24cadeef"
Oct 30 13:26:22.084045 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:22.084027    4364 scope.go:122] "RemoveContainer" containerID="fc806d441740c27c7d1faa15cca81d6050ee11934ff3d6733efdeccb24cadeef"
Oct 30 13:26:22.084306 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:26:22.084286    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc806d441740c27c7d1faa15cca81d6050ee11934ff3d6733efdeccb24cadeef\": container with ID starting with fc806d441740c27c7d1faa15cca81d6050ee11934ff3d6733efdeccb24cadeef not found: ID does not exist" containerID="fc806d441740c27c7d1faa15cca81d6050ee11934ff3d6733efdeccb24cadeef"
Oct 30 13:26:22.084351 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:22.084316    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc806d441740c27c7d1faa15cca81d6050ee11934ff3d6733efdeccb24cadeef"} err="failed to get container status \"fc806d441740c27c7d1faa15cca81d6050ee11934ff3d6733efdeccb24cadeef\": rpc error: code = NotFound desc = could not find container \"fc806d441740c27c7d1faa15cca81d6050ee11934ff3d6733efdeccb24cadeef\": container with ID starting with fc806d441740c27c7d1faa15cca81d6050ee11934ff3d6733efdeccb24cadeef not found: ID does not exist"
Oct 30 13:26:22.125101 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:22.125061    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-b5a369/test-model-b5a369-kserve-6bfb684495-gckcs"]
Oct 30 13:26:22.132401 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:22.132309    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-b5a369/test-model-b5a369-kserve-6bfb684495-gckcs"]
Oct 30 13:26:22.255673 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:22.255639    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd"]
Oct 30 13:26:22.255983 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:22.255954    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd" podUID="0f77c6bd-36f6-45cc-92ed-b22fa718700a" containerName="istio-proxy" containerID="cri-o://62dc970ba4997d047018b450364fb355469c1de8020f1513bd1986102e16542f" gracePeriod=30
Oct 30 13:26:22.987443 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:22.986724    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d52aab51-7fdf-4be6-8893-b5fe88dbe341" path="/var/lib/kubelet/pods/d52aab51-7fdf-4be6-8893-b5fe88dbe341/volumes"
Oct 30 13:26:26.896053 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:26.896009    4364 patch_prober.go:29] interesting pod/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:26:26.896053 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:26.896098    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd" podUID="0f77c6bd-36f6-45cc-92ed-b22fa718700a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:26:28.022691 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:28.022661    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd"
Oct 30 13:26:28.125146 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:28.125109    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f77c6bd-36f6-45cc-92ed-b22fa718700a-kube-api-access-bj29s\" (UniqueName: \"kubernetes.io/projected/0f77c6bd-36f6-45cc-92ed-b22fa718700a-kube-api-access-bj29s\") pod \"0f77c6bd-36f6-45cc-92ed-b22fa718700a\" (UID: \"0f77c6bd-36f6-45cc-92ed-b22fa718700a\") "
Oct 30 13:26:28.125315 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:28.125186    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f77c6bd-36f6-45cc-92ed-b22fa718700a-istio-token\" (UniqueName: \"kubernetes.io/projected/0f77c6bd-36f6-45cc-92ed-b22fa718700a-istio-token\") pod \"0f77c6bd-36f6-45cc-92ed-b22fa718700a\" (UID: \"0f77c6bd-36f6-45cc-92ed-b22fa718700a\") "
Oct 30 13:26:28.125315 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:28.125223    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f77c6bd-36f6-45cc-92ed-b22fa718700a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f77c6bd-36f6-45cc-92ed-b22fa718700a-credential-socket\") pod \"0f77c6bd-36f6-45cc-92ed-b22fa718700a\" (UID: \"0f77c6bd-36f6-45cc-92ed-b22fa718700a\") "
Oct 30 13:26:28.125315 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:28.125248    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f77c6bd-36f6-45cc-92ed-b22fa718700a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f77c6bd-36f6-45cc-92ed-b22fa718700a-workload-socket\") pod \"0f77c6bd-36f6-45cc-92ed-b22fa718700a\" (UID: \"0f77c6bd-36f6-45cc-92ed-b22fa718700a\") "
Oct 30 13:26:28.125508 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:28.125326    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f77c6bd-36f6-45cc-92ed-b22fa718700a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f77c6bd-36f6-45cc-92ed-b22fa718700a-istio-envoy\") pod \"0f77c6bd-36f6-45cc-92ed-b22fa718700a\" (UID: \"0f77c6bd-36f6-45cc-92ed-b22fa718700a\") "
Oct 30 13:26:28.125508 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:28.125398    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f77c6bd-36f6-45cc-92ed-b22fa718700a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f77c6bd-36f6-45cc-92ed-b22fa718700a-istio-data\") pod \"0f77c6bd-36f6-45cc-92ed-b22fa718700a\" (UID: \"0f77c6bd-36f6-45cc-92ed-b22fa718700a\") "
Oct 30 13:26:28.125508 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:28.125446    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f77c6bd-36f6-45cc-92ed-b22fa718700a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f77c6bd-36f6-45cc-92ed-b22fa718700a-workload-certs\") pod \"0f77c6bd-36f6-45cc-92ed-b22fa718700a\" (UID: \"0f77c6bd-36f6-45cc-92ed-b22fa718700a\") "
Oct 30 13:26:28.125684 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:28.125534    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f77c6bd-36f6-45cc-92ed-b22fa718700a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f77c6bd-36f6-45cc-92ed-b22fa718700a-istiod-ca-cert\") pod \"0f77c6bd-36f6-45cc-92ed-b22fa718700a\" (UID: \"0f77c6bd-36f6-45cc-92ed-b22fa718700a\") "
Oct 30 13:26:28.125684 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:28.125575    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f77c6bd-36f6-45cc-92ed-b22fa718700a-credential-socket" pod "0f77c6bd-36f6-45cc-92ed-b22fa718700a" (UID: "0f77c6bd-36f6-45cc-92ed-b22fa718700a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:26:28.126278 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:28.125931    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0f77c6bd-36f6-45cc-92ed-b22fa718700a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f77c6bd-36f6-45cc-92ed-b22fa718700a-istio-podinfo\") pod \"0f77c6bd-36f6-45cc-92ed-b22fa718700a\" (UID: \"0f77c6bd-36f6-45cc-92ed-b22fa718700a\") "
Oct 30 13:26:28.126278 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:28.125965    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f77c6bd-36f6-45cc-92ed-b22fa718700a-istiod-ca-cert" pod "0f77c6bd-36f6-45cc-92ed-b22fa718700a" (UID: "0f77c6bd-36f6-45cc-92ed-b22fa718700a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:26:28.126278 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:28.126043    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f77c6bd-36f6-45cc-92ed-b22fa718700a-workload-socket" pod "0f77c6bd-36f6-45cc-92ed-b22fa718700a" (UID: "0f77c6bd-36f6-45cc-92ed-b22fa718700a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:26:28.126278 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:28.126224    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f77c6bd-36f6-45cc-92ed-b22fa718700a-workload-certs" pod "0f77c6bd-36f6-45cc-92ed-b22fa718700a" (UID: "0f77c6bd-36f6-45cc-92ed-b22fa718700a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:26:28.126990 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:28.126788    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f77c6bd-36f6-45cc-92ed-b22fa718700a-istio-data" pod "0f77c6bd-36f6-45cc-92ed-b22fa718700a" (UID: "0f77c6bd-36f6-45cc-92ed-b22fa718700a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:26:28.126990 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:28.126935    4364 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f77c6bd-36f6-45cc-92ed-b22fa718700a-credential-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:26:28.126990 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:28.126949    4364 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f77c6bd-36f6-45cc-92ed-b22fa718700a-workload-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:26:28.126990 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:28.126958    4364 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f77c6bd-36f6-45cc-92ed-b22fa718700a-istio-data\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:26:28.126990 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:28.126966    4364 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f77c6bd-36f6-45cc-92ed-b22fa718700a-workload-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:26:28.126990 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:28.126974    4364 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f77c6bd-36f6-45cc-92ed-b22fa718700a-istiod-ca-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:26:28.128225 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:28.128191    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f77c6bd-36f6-45cc-92ed-b22fa718700a-kube-api-access-bj29s" pod "0f77c6bd-36f6-45cc-92ed-b22fa718700a" (UID: "0f77c6bd-36f6-45cc-92ed-b22fa718700a"). InnerVolumeSpecName "kube-api-access-bj29s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:26:28.128459 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:28.128438    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f77c6bd-36f6-45cc-92ed-b22fa718700a-istio-envoy" pod "0f77c6bd-36f6-45cc-92ed-b22fa718700a" (UID: "0f77c6bd-36f6-45cc-92ed-b22fa718700a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:26:28.128545 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:28.128452    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f77c6bd-36f6-45cc-92ed-b22fa718700a-istio-token" pod "0f77c6bd-36f6-45cc-92ed-b22fa718700a" (UID: "0f77c6bd-36f6-45cc-92ed-b22fa718700a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:26:28.128791 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:28.128771    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f77c6bd-36f6-45cc-92ed-b22fa718700a-istio-podinfo" pod "0f77c6bd-36f6-45cc-92ed-b22fa718700a" (UID: "0f77c6bd-36f6-45cc-92ed-b22fa718700a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:26:28.228492 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:28.228456    4364 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f77c6bd-36f6-45cc-92ed-b22fa718700a-istio-podinfo\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:26:28.228492 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:28.228488    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bj29s\" (UniqueName: \"kubernetes.io/projected/0f77c6bd-36f6-45cc-92ed-b22fa718700a-kube-api-access-bj29s\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:26:28.228492 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:28.228498    4364 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f77c6bd-36f6-45cc-92ed-b22fa718700a-istio-token\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:26:28.228845 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:28.228507    4364 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f77c6bd-36f6-45cc-92ed-b22fa718700a-istio-envoy\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:26:28.463209 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:28.463117    4364 generic.go:356] "Generic (PLEG): container finished" podID="0f77c6bd-36f6-45cc-92ed-b22fa718700a" containerID="62dc970ba4997d047018b450364fb355469c1de8020f1513bd1986102e16542f" exitCode=0
Oct 30 13:26:28.463209 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:28.463173    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd"
Oct 30 13:26:28.463209 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:28.463200    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd" event={"ID":"0f77c6bd-36f6-45cc-92ed-b22fa718700a","Type":"ContainerDied","Data":"62dc970ba4997d047018b450364fb355469c1de8020f1513bd1986102e16542f"}
Oct 30 13:26:28.463463 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:28.463257    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd" event={"ID":"0f77c6bd-36f6-45cc-92ed-b22fa718700a","Type":"ContainerDied","Data":"1556a0ead9df68d09cf4d9504f6cb8b6b5485ceeccab6b25b1c88accaf98bf01"}
Oct 30 13:26:28.463463 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:28.463281    4364 scope.go:122] "RemoveContainer" containerID="62dc970ba4997d047018b450364fb355469c1de8020f1513bd1986102e16542f"
Oct 30 13:26:28.479748 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:28.479484    4364 scope.go:122] "RemoveContainer" containerID="844465097f0200cdee28e6245e758d4eccfe729bfcb546db88580734c0d748d6"
Oct 30 13:26:28.492210 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:28.492184    4364 scope.go:122] "RemoveContainer" containerID="62dc970ba4997d047018b450364fb355469c1de8020f1513bd1986102e16542f"
Oct 30 13:26:28.492501 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:26:28.492483    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62dc970ba4997d047018b450364fb355469c1de8020f1513bd1986102e16542f\": container with ID starting with 62dc970ba4997d047018b450364fb355469c1de8020f1513bd1986102e16542f not found: ID does not exist" containerID="62dc970ba4997d047018b450364fb355469c1de8020f1513bd1986102e16542f"
Oct 30 13:26:28.492554 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:28.492510    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62dc970ba4997d047018b450364fb355469c1de8020f1513bd1986102e16542f"} err="failed to get container status \"62dc970ba4997d047018b450364fb355469c1de8020f1513bd1986102e16542f\": rpc error: code = NotFound desc = could not find container \"62dc970ba4997d047018b450364fb355469c1de8020f1513bd1986102e16542f\": container with ID starting with 62dc970ba4997d047018b450364fb355469c1de8020f1513bd1986102e16542f not found: ID does not exist"
Oct 30 13:26:28.492554 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:28.492530    4364 scope.go:122] "RemoveContainer" containerID="844465097f0200cdee28e6245e758d4eccfe729bfcb546db88580734c0d748d6"
Oct 30 13:26:28.492820 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:26:28.492804    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"844465097f0200cdee28e6245e758d4eccfe729bfcb546db88580734c0d748d6\": container with ID starting with 844465097f0200cdee28e6245e758d4eccfe729bfcb546db88580734c0d748d6 not found: ID does not exist" containerID="844465097f0200cdee28e6245e758d4eccfe729bfcb546db88580734c0d748d6"
Oct 30 13:26:28.492866 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:28.492824    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"844465097f0200cdee28e6245e758d4eccfe729bfcb546db88580734c0d748d6"} err="failed to get container status \"844465097f0200cdee28e6245e758d4eccfe729bfcb546db88580734c0d748d6\": rpc error: code = NotFound desc = could not find container \"844465097f0200cdee28e6245e758d4eccfe729bfcb546db88580734c0d748d6\": container with ID starting with 844465097f0200cdee28e6245e758d4eccfe729bfcb546db88580734c0d748d6 not found: ID does not exist"
Oct 30 13:26:28.521855 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:28.521823    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd"]
Oct 30 13:26:28.533242 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:28.533209    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-ddb271-openshift-default-7c7557c9bf-pwcjd"]
Oct 30 13:26:28.974034 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:28.974000    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f77c6bd-36f6-45cc-92ed-b22fa718700a" path="/var/lib/kubelet/pods/0f77c6bd-36f6-45cc-92ed-b22fa718700a/volumes"
Oct 30 13:26:29.100123 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:29.100081    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-31721c-6cc5477859-4pglh" podUID="f9acff8a-e701-4ab6-ba2f-9c8484b7bdcb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:26:33.462137 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:33.462101    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-0ad384/test-model-0ad384-kserve-6875d6fcff-9mwp9"]
Oct 30 13:26:38.126783 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:38.126741    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5dd8cc-6c7f8464c9-2jtlq"]
Oct 30 13:26:39.099378 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:39.099342    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-31721c-6cc5477859-4pglh" podUID="f9acff8a-e701-4ab6-ba2f-9c8484b7bdcb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:26:41.515062 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:26:41.515013    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:26:41.854169 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:41.854143    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-31721c-6cc5477859-4pglh"
Oct 30 13:26:41.955185 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:41.955143    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9acff8a-e701-4ab6-ba2f-9c8484b7bdcb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9acff8a-e701-4ab6-ba2f-9c8484b7bdcb-plugins-config-volume\") pod \"f9acff8a-e701-4ab6-ba2f-9c8484b7bdcb\" (UID: \"f9acff8a-e701-4ab6-ba2f-9c8484b7bdcb\") "
Oct 30 13:26:41.955385 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:41.955334    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9acff8a-e701-4ab6-ba2f-9c8484b7bdcb-kube-api-access-5qm5m\" (UniqueName: \"kubernetes.io/projected/f9acff8a-e701-4ab6-ba2f-9c8484b7bdcb-kube-api-access-5qm5m\") pod \"f9acff8a-e701-4ab6-ba2f-9c8484b7bdcb\" (UID: \"f9acff8a-e701-4ab6-ba2f-9c8484b7bdcb\") "
Oct 30 13:26:41.955621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:41.955570    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9acff8a-e701-4ab6-ba2f-9c8484b7bdcb-plugins-config-volume" pod "f9acff8a-e701-4ab6-ba2f-9c8484b7bdcb" (UID: "f9acff8a-e701-4ab6-ba2f-9c8484b7bdcb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:26:41.956393 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:41.956362    4364 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9acff8a-e701-4ab6-ba2f-9c8484b7bdcb-plugins-config-volume\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:26:41.957841 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:41.957806    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9acff8a-e701-4ab6-ba2f-9c8484b7bdcb-kube-api-access-5qm5m" pod "f9acff8a-e701-4ab6-ba2f-9c8484b7bdcb" (UID: "f9acff8a-e701-4ab6-ba2f-9c8484b7bdcb"). InnerVolumeSpecName "kube-api-access-5qm5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:26:42.058786 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:42.058757    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qm5m\" (UniqueName: \"kubernetes.io/projected/f9acff8a-e701-4ab6-ba2f-9c8484b7bdcb-kube-api-access-5qm5m\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:26:42.617049 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:42.617004    4364 generic.go:356] "Generic (PLEG): container finished" podID="f9acff8a-e701-4ab6-ba2f-9c8484b7bdcb" containerID="724a012bc8418d5b787734379ce78cb0cf2836b453fc3c18c02d73d566878338" exitCode=0
Oct 30 13:26:42.617049 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:42.617052    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-31721c-6cc5477859-4pglh"
Oct 30 13:26:42.617679 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:42.617088    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-31721c-6cc5477859-4pglh" event={"ID":"f9acff8a-e701-4ab6-ba2f-9c8484b7bdcb","Type":"ContainerDied","Data":"724a012bc8418d5b787734379ce78cb0cf2836b453fc3c18c02d73d566878338"}
Oct 30 13:26:42.617679 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:42.617145    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-31721c-6cc5477859-4pglh" event={"ID":"f9acff8a-e701-4ab6-ba2f-9c8484b7bdcb","Type":"ContainerDied","Data":"55793fdb9b2b1a25a724900a99d843c1a36cd31a008c574bb3c53434662ea8c7"}
Oct 30 13:26:42.617679 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:42.617173    4364 scope.go:122] "RemoveContainer" containerID="724a012bc8418d5b787734379ce78cb0cf2836b453fc3c18c02d73d566878338"
Oct 30 13:26:42.629532 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:42.629513    4364 scope.go:122] "RemoveContainer" containerID="724a012bc8418d5b787734379ce78cb0cf2836b453fc3c18c02d73d566878338"
Oct 30 13:26:42.629842 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:26:42.629820    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"724a012bc8418d5b787734379ce78cb0cf2836b453fc3c18c02d73d566878338\": container with ID starting with 724a012bc8418d5b787734379ce78cb0cf2836b453fc3c18c02d73d566878338 not found: ID does not exist" containerID="724a012bc8418d5b787734379ce78cb0cf2836b453fc3c18c02d73d566878338"
Oct 30 13:26:42.629924 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:42.629850    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"724a012bc8418d5b787734379ce78cb0cf2836b453fc3c18c02d73d566878338"} err="failed to get container status \"724a012bc8418d5b787734379ce78cb0cf2836b453fc3c18c02d73d566878338\": rpc error: code = NotFound desc = could not find container \"724a012bc8418d5b787734379ce78cb0cf2836b453fc3c18c02d73d566878338\": container with ID starting with 724a012bc8418d5b787734379ce78cb0cf2836b453fc3c18c02d73d566878338 not found: ID does not exist"
Oct 30 13:26:42.675555 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:42.675524    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-31721c-6cc5477859-4pglh"]
Oct 30 13:26:42.687653 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:42.687623    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-31721c-6cc5477859-4pglh"]
Oct 30 13:26:42.981990 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:42.981950    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9acff8a-e701-4ab6-ba2f-9c8484b7bdcb" path="/var/lib/kubelet/pods/f9acff8a-e701-4ab6-ba2f-9c8484b7bdcb/volumes"
Oct 30 13:26:43.127734 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:43.127637    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5dd8cc-6c7f8464c9-2jtlq" podUID="a9c7d05a-ebeb-4506-88f9-5932b1730822" containerName="payload-pre-processing" containerID="cri-o://60b8d619a5c60919a59a4ab2d505befec14fd56c434e06bcb233a7272ca88444" gracePeriod=25
Oct 30 13:26:45.367454 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:45.367415    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5dd8cc-6c7f8464c9-2jtlq" podUID="a9c7d05a-ebeb-4506-88f9-5932b1730822" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:26:48.485859 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:48.485789    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-0ad384/test-model-0ad384-kserve-6875d6fcff-9mwp9" podUID="517cf09e-dcec-4fe5-ad42-d7e15cd1e48a" containerName="main" containerID="cri-o://b49c78a98579e4a9234c02cda834e8bb197fca0c15650d31b923daf8b2f1be61" gracePeriod=45
Oct 30 13:26:48.691833 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:48.691613    4364 generic.go:356] "Generic (PLEG): container finished" podID="517cf09e-dcec-4fe5-ad42-d7e15cd1e48a" containerID="b49c78a98579e4a9234c02cda834e8bb197fca0c15650d31b923daf8b2f1be61" exitCode=0
Oct 30 13:26:48.691833 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:48.691722    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0ad384/test-model-0ad384-kserve-6875d6fcff-9mwp9" event={"ID":"517cf09e-dcec-4fe5-ad42-d7e15cd1e48a","Type":"ContainerDied","Data":"b49c78a98579e4a9234c02cda834e8bb197fca0c15650d31b923daf8b2f1be61"}
Oct 30 13:26:48.853187 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:48.853157    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0ad384/test-model-0ad384-kserve-6875d6fcff-9mwp9"
Oct 30 13:26:48.897759 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:48.897727    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/517cf09e-dcec-4fe5-ad42-d7e15cd1e48a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/517cf09e-dcec-4fe5-ad42-d7e15cd1e48a-model-cache\") pod \"517cf09e-dcec-4fe5-ad42-d7e15cd1e48a\" (UID: \"517cf09e-dcec-4fe5-ad42-d7e15cd1e48a\") "
Oct 30 13:26:48.897940 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:48.897788    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/517cf09e-dcec-4fe5-ad42-d7e15cd1e48a-home\" (UniqueName: \"kubernetes.io/empty-dir/517cf09e-dcec-4fe5-ad42-d7e15cd1e48a-home\") pod \"517cf09e-dcec-4fe5-ad42-d7e15cd1e48a\" (UID: \"517cf09e-dcec-4fe5-ad42-d7e15cd1e48a\") "
Oct 30 13:26:48.897940 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:48.897914    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/517cf09e-dcec-4fe5-ad42-d7e15cd1e48a-tls-certs\" (UniqueName: \"kubernetes.io/secret/517cf09e-dcec-4fe5-ad42-d7e15cd1e48a-tls-certs\") pod \"517cf09e-dcec-4fe5-ad42-d7e15cd1e48a\" (UID: \"517cf09e-dcec-4fe5-ad42-d7e15cd1e48a\") "
Oct 30 13:26:48.898049 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:48.898019    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/517cf09e-dcec-4fe5-ad42-d7e15cd1e48a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/517cf09e-dcec-4fe5-ad42-d7e15cd1e48a-tmp-dir\") pod \"517cf09e-dcec-4fe5-ad42-d7e15cd1e48a\" (UID: \"517cf09e-dcec-4fe5-ad42-d7e15cd1e48a\") "
Oct 30 13:26:48.898112 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:48.898033    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/517cf09e-dcec-4fe5-ad42-d7e15cd1e48a-model-cache" pod "517cf09e-dcec-4fe5-ad42-d7e15cd1e48a" (UID: "517cf09e-dcec-4fe5-ad42-d7e15cd1e48a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:26:48.898182 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:48.898113    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/517cf09e-dcec-4fe5-ad42-d7e15cd1e48a-home" pod "517cf09e-dcec-4fe5-ad42-d7e15cd1e48a" (UID: "517cf09e-dcec-4fe5-ad42-d7e15cd1e48a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:26:48.898182 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:48.898124    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/517cf09e-dcec-4fe5-ad42-d7e15cd1e48a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/517cf09e-dcec-4fe5-ad42-d7e15cd1e48a-dshm\") pod \"517cf09e-dcec-4fe5-ad42-d7e15cd1e48a\" (UID: \"517cf09e-dcec-4fe5-ad42-d7e15cd1e48a\") "
Oct 30 13:26:48.898182 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:48.898174    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/517cf09e-dcec-4fe5-ad42-d7e15cd1e48a-kube-api-access-x6kf7\" (UniqueName: \"kubernetes.io/projected/517cf09e-dcec-4fe5-ad42-d7e15cd1e48a-kube-api-access-x6kf7\") pod \"517cf09e-dcec-4fe5-ad42-d7e15cd1e48a\" (UID: \"517cf09e-dcec-4fe5-ad42-d7e15cd1e48a\") "
Oct 30 13:26:48.898359 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:48.898317    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/517cf09e-dcec-4fe5-ad42-d7e15cd1e48a-tmp-dir" pod "517cf09e-dcec-4fe5-ad42-d7e15cd1e48a" (UID: "517cf09e-dcec-4fe5-ad42-d7e15cd1e48a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:26:48.898812 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:48.898732    4364 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/517cf09e-dcec-4fe5-ad42-d7e15cd1e48a-tmp-dir\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:26:48.898812 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:48.898760    4364 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/517cf09e-dcec-4fe5-ad42-d7e15cd1e48a-model-cache\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:26:48.898812 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:48.898773    4364 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/517cf09e-dcec-4fe5-ad42-d7e15cd1e48a-home\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:26:48.901024 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:48.900995    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/517cf09e-dcec-4fe5-ad42-d7e15cd1e48a-dshm" pod "517cf09e-dcec-4fe5-ad42-d7e15cd1e48a" (UID: "517cf09e-dcec-4fe5-ad42-d7e15cd1e48a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:26:48.901787 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:48.901760    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/517cf09e-dcec-4fe5-ad42-d7e15cd1e48a-tls-certs" pod "517cf09e-dcec-4fe5-ad42-d7e15cd1e48a" (UID: "517cf09e-dcec-4fe5-ad42-d7e15cd1e48a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:26:48.901898 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:48.901827    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/517cf09e-dcec-4fe5-ad42-d7e15cd1e48a-kube-api-access-x6kf7" pod "517cf09e-dcec-4fe5-ad42-d7e15cd1e48a" (UID: "517cf09e-dcec-4fe5-ad42-d7e15cd1e48a"). InnerVolumeSpecName "kube-api-access-x6kf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:26:48.999826 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:48.999791    4364 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/517cf09e-dcec-4fe5-ad42-d7e15cd1e48a-tls-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:26:48.999826 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:48.999821    4364 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/517cf09e-dcec-4fe5-ad42-d7e15cd1e48a-dshm\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:26:48.999826 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:48.999830    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6kf7\" (UniqueName: \"kubernetes.io/projected/517cf09e-dcec-4fe5-ad42-d7e15cd1e48a-kube-api-access-x6kf7\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:26:49.701958 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:49.701928    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0ad384/test-model-0ad384-kserve-6875d6fcff-9mwp9"
Oct 30 13:26:49.702396 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:49.701960    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0ad384/test-model-0ad384-kserve-6875d6fcff-9mwp9" event={"ID":"517cf09e-dcec-4fe5-ad42-d7e15cd1e48a","Type":"ContainerDied","Data":"dd69532c6b35791f302c4c3b42ee153469eae4e0772926af6e6a8907253e36d6"}
Oct 30 13:26:49.702396 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:49.702013    4364 scope.go:122] "RemoveContainer" containerID="b49c78a98579e4a9234c02cda834e8bb197fca0c15650d31b923daf8b2f1be61"
Oct 30 13:26:49.756532 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:49.756497    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-0ad384/test-model-0ad384-kserve-6875d6fcff-9mwp9"]
Oct 30 13:26:49.767765 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:49.767738    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-0ad384/test-model-0ad384-kserve-6875d6fcff-9mwp9"]
Oct 30 13:26:50.973646 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:50.973610    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="517cf09e-dcec-4fe5-ad42-d7e15cd1e48a" path="/var/lib/kubelet/pods/517cf09e-dcec-4fe5-ad42-d7e15cd1e48a/volumes"
Oct 30 13:26:55.367000 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:55.366908    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5dd8cc-6c7f8464c9-2jtlq" podUID="a9c7d05a-ebeb-4506-88f9-5932b1730822" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:26:55.716497 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:26:55.716412    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-09e4c0/test-model-09e4c0-kserve-56cdb9c5b5-nrjtk"]
Oct 30 13:27:03.432760 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:03.432728    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5dd8cc-6c7f8464c9-2jtlq"
Oct 30 13:27:03.474983 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:03.474283    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9c7d05a-ebeb-4506-88f9-5932b1730822-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9c7d05a-ebeb-4506-88f9-5932b1730822-plugins-config-volume\") pod \"a9c7d05a-ebeb-4506-88f9-5932b1730822\" (UID: \"a9c7d05a-ebeb-4506-88f9-5932b1730822\") "
Oct 30 13:27:03.474983 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:03.474458    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9c7d05a-ebeb-4506-88f9-5932b1730822-kube-api-access-ggqjs\" (UniqueName: \"kubernetes.io/projected/a9c7d05a-ebeb-4506-88f9-5932b1730822-kube-api-access-ggqjs\") pod \"a9c7d05a-ebeb-4506-88f9-5932b1730822\" (UID: \"a9c7d05a-ebeb-4506-88f9-5932b1730822\") "
Oct 30 13:27:03.477218 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:03.477194    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9c7d05a-ebeb-4506-88f9-5932b1730822-plugins-config-volume" pod "a9c7d05a-ebeb-4506-88f9-5932b1730822" (UID: "a9c7d05a-ebeb-4506-88f9-5932b1730822"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:27:03.479910 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:03.479882    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9c7d05a-ebeb-4506-88f9-5932b1730822-kube-api-access-ggqjs" pod "a9c7d05a-ebeb-4506-88f9-5932b1730822" (UID: "a9c7d05a-ebeb-4506-88f9-5932b1730822"). InnerVolumeSpecName "kube-api-access-ggqjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:27:03.577173 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:03.577141    4364 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9c7d05a-ebeb-4506-88f9-5932b1730822-plugins-config-volume\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:27:03.577173 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:03.577168    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggqjs\" (UniqueName: \"kubernetes.io/projected/a9c7d05a-ebeb-4506-88f9-5932b1730822-kube-api-access-ggqjs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:27:03.825365 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:03.825275    4364 generic.go:356] "Generic (PLEG): container finished" podID="a9c7d05a-ebeb-4506-88f9-5932b1730822" containerID="60b8d619a5c60919a59a4ab2d505befec14fd56c434e06bcb233a7272ca88444" exitCode=0
Oct 30 13:27:03.825365 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:03.825325    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5dd8cc-6c7f8464c9-2jtlq"
Oct 30 13:27:03.825365 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:03.825355    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5dd8cc-6c7f8464c9-2jtlq" event={"ID":"a9c7d05a-ebeb-4506-88f9-5932b1730822","Type":"ContainerDied","Data":"60b8d619a5c60919a59a4ab2d505befec14fd56c434e06bcb233a7272ca88444"}
Oct 30 13:27:03.825686 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:03.825410    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5dd8cc-6c7f8464c9-2jtlq" event={"ID":"a9c7d05a-ebeb-4506-88f9-5932b1730822","Type":"ContainerDied","Data":"812b790e517fae219de8f3666a7635929a4f8265eb343af5ded6e0994bf9b4c3"}
Oct 30 13:27:03.825686 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:03.825440    4364 scope.go:122] "RemoveContainer" containerID="60b8d619a5c60919a59a4ab2d505befec14fd56c434e06bcb233a7272ca88444"
Oct 30 13:27:03.838966 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:03.838943    4364 scope.go:122] "RemoveContainer" containerID="60b8d619a5c60919a59a4ab2d505befec14fd56c434e06bcb233a7272ca88444"
Oct 30 13:27:03.839247 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:27:03.839226    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60b8d619a5c60919a59a4ab2d505befec14fd56c434e06bcb233a7272ca88444\": container with ID starting with 60b8d619a5c60919a59a4ab2d505befec14fd56c434e06bcb233a7272ca88444 not found: ID does not exist" containerID="60b8d619a5c60919a59a4ab2d505befec14fd56c434e06bcb233a7272ca88444"
Oct 30 13:27:03.839323 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:03.839257    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60b8d619a5c60919a59a4ab2d505befec14fd56c434e06bcb233a7272ca88444"} err="failed to get container status \"60b8d619a5c60919a59a4ab2d505befec14fd56c434e06bcb233a7272ca88444\": rpc error: code = NotFound desc = could not find container \"60b8d619a5c60919a59a4ab2d505befec14fd56c434e06bcb233a7272ca88444\": container with ID starting with 60b8d619a5c60919a59a4ab2d505befec14fd56c434e06bcb233a7272ca88444 not found: ID does not exist"
Oct 30 13:27:03.881239 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:03.881210    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5dd8cc-6c7f8464c9-2jtlq"]
Oct 30 13:27:03.892565 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:03.892537    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5dd8cc-6c7f8464c9-2jtlq"]
Oct 30 13:27:04.972352 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:04.972320    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9c7d05a-ebeb-4506-88f9-5932b1730822" path="/var/lib/kubelet/pods/a9c7d05a-ebeb-4506-88f9-5932b1730822/volumes"
Oct 30 13:27:06.293046 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:06.293012    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0ad384-554b74dc84-486b2"]
Oct 30 13:27:10.741902 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:10.741835    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-09e4c0/test-model-09e4c0-kserve-56cdb9c5b5-nrjtk" podUID="3fd28ec8-f8ba-400b-940a-9c6b5c516936" containerName="main" containerID="cri-o://d08f515d537a59787af0dfcbcf5c8a7c14fe7410b9f270e28c68edba48817f25" gracePeriod=45
Oct 30 13:27:10.999565 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:10.999480    4364 generic.go:356] "Generic (PLEG): container finished" podID="3fd28ec8-f8ba-400b-940a-9c6b5c516936" containerID="d08f515d537a59787af0dfcbcf5c8a7c14fe7410b9f270e28c68edba48817f25" exitCode=0
Oct 30 13:27:10.999783 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:10.999756    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-09e4c0/test-model-09e4c0-kserve-56cdb9c5b5-nrjtk" event={"ID":"3fd28ec8-f8ba-400b-940a-9c6b5c516936","Type":"ContainerDied","Data":"d08f515d537a59787af0dfcbcf5c8a7c14fe7410b9f270e28c68edba48817f25"}
Oct 30 13:27:11.083337 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:11.083309    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-09e4c0/test-model-09e4c0-kserve-56cdb9c5b5-nrjtk"
Oct 30 13:27:11.250573 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:11.250487    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fd28ec8-f8ba-400b-940a-9c6b5c516936-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fd28ec8-f8ba-400b-940a-9c6b5c516936-model-cache\") pod \"3fd28ec8-f8ba-400b-940a-9c6b5c516936\" (UID: \"3fd28ec8-f8ba-400b-940a-9c6b5c516936\") "
Oct 30 13:27:11.250761 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:11.250664    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fd28ec8-f8ba-400b-940a-9c6b5c516936-home\" (UniqueName: \"kubernetes.io/empty-dir/3fd28ec8-f8ba-400b-940a-9c6b5c516936-home\") pod \"3fd28ec8-f8ba-400b-940a-9c6b5c516936\" (UID: \"3fd28ec8-f8ba-400b-940a-9c6b5c516936\") "
Oct 30 13:27:11.250761 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:11.250717    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fd28ec8-f8ba-400b-940a-9c6b5c516936-kube-api-access-9p5nz\" (UniqueName: \"kubernetes.io/projected/3fd28ec8-f8ba-400b-940a-9c6b5c516936-kube-api-access-9p5nz\") pod \"3fd28ec8-f8ba-400b-940a-9c6b5c516936\" (UID: \"3fd28ec8-f8ba-400b-940a-9c6b5c516936\") "
Oct 30 13:27:11.250848 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:11.250774    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fd28ec8-f8ba-400b-940a-9c6b5c516936-model-cache" pod "3fd28ec8-f8ba-400b-940a-9c6b5c516936" (UID: "3fd28ec8-f8ba-400b-940a-9c6b5c516936"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:27:11.250848 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:11.250794    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fd28ec8-f8ba-400b-940a-9c6b5c516936-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fd28ec8-f8ba-400b-940a-9c6b5c516936-dshm\") pod \"3fd28ec8-f8ba-400b-940a-9c6b5c516936\" (UID: \"3fd28ec8-f8ba-400b-940a-9c6b5c516936\") "
Oct 30 13:27:11.250934 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:11.250918    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fd28ec8-f8ba-400b-940a-9c6b5c516936-tls-certs\" (UniqueName: \"kubernetes.io/secret/3fd28ec8-f8ba-400b-940a-9c6b5c516936-tls-certs\") pod \"3fd28ec8-f8ba-400b-940a-9c6b5c516936\" (UID: \"3fd28ec8-f8ba-400b-940a-9c6b5c516936\") "
Oct 30 13:27:11.250968 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:11.250945    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fd28ec8-f8ba-400b-940a-9c6b5c516936-home" pod "3fd28ec8-f8ba-400b-940a-9c6b5c516936" (UID: "3fd28ec8-f8ba-400b-940a-9c6b5c516936"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:27:11.251082 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:11.251070    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fd28ec8-f8ba-400b-940a-9c6b5c516936-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fd28ec8-f8ba-400b-940a-9c6b5c516936-tmp-dir\") pod \"3fd28ec8-f8ba-400b-940a-9c6b5c516936\" (UID: \"3fd28ec8-f8ba-400b-940a-9c6b5c516936\") "
Oct 30 13:27:11.251348 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:11.251326    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fd28ec8-f8ba-400b-940a-9c6b5c516936-tmp-dir" pod "3fd28ec8-f8ba-400b-940a-9c6b5c516936" (UID: "3fd28ec8-f8ba-400b-940a-9c6b5c516936"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:27:11.251840 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:11.251821    4364 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fd28ec8-f8ba-400b-940a-9c6b5c516936-tmp-dir\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:27:11.251939 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:11.251842    4364 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fd28ec8-f8ba-400b-940a-9c6b5c516936-model-cache\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:27:11.251939 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:11.251855    4364 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fd28ec8-f8ba-400b-940a-9c6b5c516936-home\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:27:11.253179 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:11.253148    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fd28ec8-f8ba-400b-940a-9c6b5c516936-tls-certs" pod "3fd28ec8-f8ba-400b-940a-9c6b5c516936" (UID: "3fd28ec8-f8ba-400b-940a-9c6b5c516936"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:27:11.253270 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:11.253174    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fd28ec8-f8ba-400b-940a-9c6b5c516936-dshm" pod "3fd28ec8-f8ba-400b-940a-9c6b5c516936" (UID: "3fd28ec8-f8ba-400b-940a-9c6b5c516936"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:27:11.253270 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:11.253251    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fd28ec8-f8ba-400b-940a-9c6b5c516936-kube-api-access-9p5nz" pod "3fd28ec8-f8ba-400b-940a-9c6b5c516936" (UID: "3fd28ec8-f8ba-400b-940a-9c6b5c516936"). InnerVolumeSpecName "kube-api-access-9p5nz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:27:11.294719 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:11.294677    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-0ad384-554b74dc84-486b2" podUID="0cd7b23d-f2dc-49be-b0a5-0dd233c8faa9" containerName="payload-processing" containerID="cri-o://65dd0bb5a787194b4f374511bd09d7eac49760c506055e24073b53b19bed8148" gracePeriod=25
Oct 30 13:27:11.353973 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:11.353937    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9p5nz\" (UniqueName: \"kubernetes.io/projected/3fd28ec8-f8ba-400b-940a-9c6b5c516936-kube-api-access-9p5nz\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:27:11.353973 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:11.353971    4364 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fd28ec8-f8ba-400b-940a-9c6b5c516936-dshm\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:27:11.354179 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:11.353984    4364 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fd28ec8-f8ba-400b-940a-9c6b5c516936-tls-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:27:11.520157 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:27:11.517487    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:27:11.565233 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:11.565204    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0ad384-554b74dc84-486b2"
Oct 30 13:27:11.659105 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:11.659061    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cd7b23d-f2dc-49be-b0a5-0dd233c8faa9-kube-api-access-dbqvq\" (UniqueName: \"kubernetes.io/projected/0cd7b23d-f2dc-49be-b0a5-0dd233c8faa9-kube-api-access-dbqvq\") pod \"0cd7b23d-f2dc-49be-b0a5-0dd233c8faa9\" (UID: \"0cd7b23d-f2dc-49be-b0a5-0dd233c8faa9\") "
Oct 30 13:27:11.659312 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:11.659219    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cd7b23d-f2dc-49be-b0a5-0dd233c8faa9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cd7b23d-f2dc-49be-b0a5-0dd233c8faa9-plugins-config-volume\") pod \"0cd7b23d-f2dc-49be-b0a5-0dd233c8faa9\" (UID: \"0cd7b23d-f2dc-49be-b0a5-0dd233c8faa9\") "
Oct 30 13:27:11.659771 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:11.659739    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cd7b23d-f2dc-49be-b0a5-0dd233c8faa9-plugins-config-volume" pod "0cd7b23d-f2dc-49be-b0a5-0dd233c8faa9" (UID: "0cd7b23d-f2dc-49be-b0a5-0dd233c8faa9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:27:11.660097 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:11.660074    4364 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cd7b23d-f2dc-49be-b0a5-0dd233c8faa9-plugins-config-volume\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:27:11.661217 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:11.661195    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cd7b23d-f2dc-49be-b0a5-0dd233c8faa9-kube-api-access-dbqvq" pod "0cd7b23d-f2dc-49be-b0a5-0dd233c8faa9" (UID: "0cd7b23d-f2dc-49be-b0a5-0dd233c8faa9"). InnerVolumeSpecName "kube-api-access-dbqvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:27:11.761502 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:11.761462    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbqvq\" (UniqueName: \"kubernetes.io/projected/0cd7b23d-f2dc-49be-b0a5-0dd233c8faa9-kube-api-access-dbqvq\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:27:12.009392 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:12.009359    4364 generic.go:356] "Generic (PLEG): container finished" podID="0cd7b23d-f2dc-49be-b0a5-0dd233c8faa9" containerID="65dd0bb5a787194b4f374511bd09d7eac49760c506055e24073b53b19bed8148" exitCode=0
Oct 30 13:27:12.009589 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:12.009404    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0ad384-554b74dc84-486b2"
Oct 30 13:27:12.009589 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:12.009441    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0ad384-554b74dc84-486b2" event={"ID":"0cd7b23d-f2dc-49be-b0a5-0dd233c8faa9","Type":"ContainerDied","Data":"65dd0bb5a787194b4f374511bd09d7eac49760c506055e24073b53b19bed8148"}
Oct 30 13:27:12.009589 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:12.009496    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0ad384-554b74dc84-486b2" event={"ID":"0cd7b23d-f2dc-49be-b0a5-0dd233c8faa9","Type":"ContainerDied","Data":"cfcc600c59903272c6b37a0544e65cf0f70672ac4121c88c36aa0f2c958a4c39"}
Oct 30 13:27:12.009589 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:12.009519    4364 scope.go:122] "RemoveContainer" containerID="65dd0bb5a787194b4f374511bd09d7eac49760c506055e24073b53b19bed8148"
Oct 30 13:27:12.011216 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:12.011185    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-09e4c0/test-model-09e4c0-kserve-56cdb9c5b5-nrjtk"
Oct 30 13:27:12.011345 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:12.011226    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-09e4c0/test-model-09e4c0-kserve-56cdb9c5b5-nrjtk" event={"ID":"3fd28ec8-f8ba-400b-940a-9c6b5c516936","Type":"ContainerDied","Data":"02189c6407c7d7affa92c2172e8b44b042c4b5a2c99224f2f2a02907afee90d7"}
Oct 30 13:27:12.022631 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:12.022610    4364 scope.go:122] "RemoveContainer" containerID="65dd0bb5a787194b4f374511bd09d7eac49760c506055e24073b53b19bed8148"
Oct 30 13:27:12.022914 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:27:12.022895    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65dd0bb5a787194b4f374511bd09d7eac49760c506055e24073b53b19bed8148\": container with ID starting with 65dd0bb5a787194b4f374511bd09d7eac49760c506055e24073b53b19bed8148 not found: ID does not exist" containerID="65dd0bb5a787194b4f374511bd09d7eac49760c506055e24073b53b19bed8148"
Oct 30 13:27:12.022991 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:12.022929    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65dd0bb5a787194b4f374511bd09d7eac49760c506055e24073b53b19bed8148"} err="failed to get container status \"65dd0bb5a787194b4f374511bd09d7eac49760c506055e24073b53b19bed8148\": rpc error: code = NotFound desc = could not find container \"65dd0bb5a787194b4f374511bd09d7eac49760c506055e24073b53b19bed8148\": container with ID starting with 65dd0bb5a787194b4f374511bd09d7eac49760c506055e24073b53b19bed8148 not found: ID does not exist"
Oct 30 13:27:12.022991 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:12.022954    4364 scope.go:122] "RemoveContainer" containerID="d08f515d537a59787af0dfcbcf5c8a7c14fe7410b9f270e28c68edba48817f25"
Oct 30 13:27:12.033966 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:12.033939    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-09e4c0/test-model-09e4c0-kserve-56cdb9c5b5-nrjtk" podUID="3fd28ec8-f8ba-400b-940a-9c6b5c516936" containerName="main" probeResult="failure" output="Get \"https://10.133.0.196:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:27:12.078454 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:12.078423    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-09e4c0/test-model-09e4c0-kserve-56cdb9c5b5-nrjtk"]
Oct 30 13:27:12.085851 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:12.085826    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-09e4c0/test-model-09e4c0-kserve-56cdb9c5b5-nrjtk"]
Oct 30 13:27:12.132792 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:12.132764    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0ad384-554b74dc84-486b2"]
Oct 30 13:27:12.141542 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:12.141515    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0ad384-554b74dc84-486b2"]
Oct 30 13:27:12.972619 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:12.972569    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cd7b23d-f2dc-49be-b0a5-0dd233c8faa9" path="/var/lib/kubelet/pods/0cd7b23d-f2dc-49be-b0a5-0dd233c8faa9/volumes"
Oct 30 13:27:12.973295 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:12.973271    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fd28ec8-f8ba-400b-940a-9c6b5c516936" path="/var/lib/kubelet/pods/3fd28ec8-f8ba-400b-940a-9c6b5c516936/volumes"
Oct 30 13:27:13.091385 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:13.091344    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz"]
Oct 30 13:27:13.091663 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:13.091634    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz" podUID="52bf85ca-e35d-4a85-85f4-21d6d20a191d" containerName="istio-proxy" containerID="cri-o://83f8744d68777f39cfac6983cf5de47dd6c2a5c57c8234fe41a8e2720924131f" gracePeriod=30
Oct 30 13:27:16.387562 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:16.387524    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5dd8cc-657cd56d9c-wps6v"]
Oct 30 13:27:16.388059 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:16.387788    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5dd8cc-657cd56d9c-wps6v" podUID="873a9c3f-d2a1-4eb8-a413-216e4d5cb48c" containerName="maas-api" containerID="cri-o://a67dfd83613ae6c7334c3475ab447b7ec84f0027de21b7a083084d4b0afe7a06" gracePeriod=30
Oct 30 13:27:16.820292 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:16.820134    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5dd8cc-657cd56d9c-wps6v"
Oct 30 13:27:16.922285 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:16.919691    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/873a9c3f-d2a1-4eb8-a413-216e4d5cb48c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/873a9c3f-d2a1-4eb8-a413-216e4d5cb48c-metrics-certs\") pod \"873a9c3f-d2a1-4eb8-a413-216e4d5cb48c\" (UID: \"873a9c3f-d2a1-4eb8-a413-216e4d5cb48c\") "
Oct 30 13:27:16.922285 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:16.919873    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/873a9c3f-d2a1-4eb8-a413-216e4d5cb48c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/873a9c3f-d2a1-4eb8-a413-216e4d5cb48c-openshift-service-ca\") pod \"873a9c3f-d2a1-4eb8-a413-216e4d5cb48c\" (UID: \"873a9c3f-d2a1-4eb8-a413-216e4d5cb48c\") "
Oct 30 13:27:16.922285 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:16.919977    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/873a9c3f-d2a1-4eb8-a413-216e4d5cb48c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/873a9c3f-d2a1-4eb8-a413-216e4d5cb48c-maas-api-tls\") pod \"873a9c3f-d2a1-4eb8-a413-216e4d5cb48c\" (UID: \"873a9c3f-d2a1-4eb8-a413-216e4d5cb48c\") "
Oct 30 13:27:16.922285 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:16.920118    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/873a9c3f-d2a1-4eb8-a413-216e4d5cb48c-kube-api-access-r8p6k\" (UniqueName: \"kubernetes.io/projected/873a9c3f-d2a1-4eb8-a413-216e4d5cb48c-kube-api-access-r8p6k\") pod \"873a9c3f-d2a1-4eb8-a413-216e4d5cb48c\" (UID: \"873a9c3f-d2a1-4eb8-a413-216e4d5cb48c\") "
Oct 30 13:27:16.922285 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:16.921982    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/873a9c3f-d2a1-4eb8-a413-216e4d5cb48c-openshift-service-ca" pod "873a9c3f-d2a1-4eb8-a413-216e4d5cb48c" (UID: "873a9c3f-d2a1-4eb8-a413-216e4d5cb48c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:27:17.100696 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:17.020102    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/873a9c3f-d2a1-4eb8-a413-216e4d5cb48c-kube-api-access-r8p6k" pod "873a9c3f-d2a1-4eb8-a413-216e4d5cb48c" (UID: "873a9c3f-d2a1-4eb8-a413-216e4d5cb48c"). InnerVolumeSpecName "kube-api-access-r8p6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:27:17.100696 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:17.028659    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/873a9c3f-d2a1-4eb8-a413-216e4d5cb48c-metrics-certs" pod "873a9c3f-d2a1-4eb8-a413-216e4d5cb48c" (UID: "873a9c3f-d2a1-4eb8-a413-216e4d5cb48c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:27:17.100696 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:17.035475    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/873a9c3f-d2a1-4eb8-a413-216e4d5cb48c-maas-api-tls" pod "873a9c3f-d2a1-4eb8-a413-216e4d5cb48c" (UID: "873a9c3f-d2a1-4eb8-a413-216e4d5cb48c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:27:17.109410 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:17.106497    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8p6k\" (UniqueName: \"kubernetes.io/projected/873a9c3f-d2a1-4eb8-a413-216e4d5cb48c-kube-api-access-r8p6k\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:27:17.109410 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:17.107663    4364 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/873a9c3f-d2a1-4eb8-a413-216e4d5cb48c-metrics-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:27:17.109410 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:17.107691    4364 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/873a9c3f-d2a1-4eb8-a413-216e4d5cb48c-openshift-service-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:27:17.109410 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:17.107706    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/873a9c3f-d2a1-4eb8-a413-216e4d5cb48c-maas-api-tls\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:27:17.127712 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:17.127680    4364 generic.go:356] "Generic (PLEG): container finished" podID="873a9c3f-d2a1-4eb8-a413-216e4d5cb48c" containerID="a67dfd83613ae6c7334c3475ab447b7ec84f0027de21b7a083084d4b0afe7a06" exitCode=0
Oct 30 13:27:17.127902 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:17.127837    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5dd8cc-657cd56d9c-wps6v"
Oct 30 13:27:17.127988 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:17.127963    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5dd8cc-657cd56d9c-wps6v" event={"ID":"873a9c3f-d2a1-4eb8-a413-216e4d5cb48c","Type":"ContainerDied","Data":"a67dfd83613ae6c7334c3475ab447b7ec84f0027de21b7a083084d4b0afe7a06"}
Oct 30 13:27:17.128065 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:17.128026    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5dd8cc-657cd56d9c-wps6v" event={"ID":"873a9c3f-d2a1-4eb8-a413-216e4d5cb48c","Type":"ContainerDied","Data":"c4f20b8f9310ac26974a515521ebd80be28afe9a4fdaa0a42286494175ab5ac9"}
Oct 30 13:27:17.128065 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:17.128061    4364 scope.go:122] "RemoveContainer" containerID="a67dfd83613ae6c7334c3475ab447b7ec84f0027de21b7a083084d4b0afe7a06"
Oct 30 13:27:17.163318 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:17.163292    4364 scope.go:122] "RemoveContainer" containerID="a67dfd83613ae6c7334c3475ab447b7ec84f0027de21b7a083084d4b0afe7a06"
Oct 30 13:27:17.163712 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:27:17.163689    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a67dfd83613ae6c7334c3475ab447b7ec84f0027de21b7a083084d4b0afe7a06\": container with ID starting with a67dfd83613ae6c7334c3475ab447b7ec84f0027de21b7a083084d4b0afe7a06 not found: ID does not exist" containerID="a67dfd83613ae6c7334c3475ab447b7ec84f0027de21b7a083084d4b0afe7a06"
Oct 30 13:27:17.163802 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:17.163719    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a67dfd83613ae6c7334c3475ab447b7ec84f0027de21b7a083084d4b0afe7a06"} err="failed to get container status \"a67dfd83613ae6c7334c3475ab447b7ec84f0027de21b7a083084d4b0afe7a06\": rpc error: code = NotFound desc = could not find container \"a67dfd83613ae6c7334c3475ab447b7ec84f0027de21b7a083084d4b0afe7a06\": container with ID starting with a67dfd83613ae6c7334c3475ab447b7ec84f0027de21b7a083084d4b0afe7a06 not found: ID does not exist"
Oct 30 13:27:17.198838 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:17.198802    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5dd8cc-657cd56d9c-wps6v"]
Oct 30 13:27:17.203575 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:17.203551    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5dd8cc-657cd56d9c-wps6v"]
Oct 30 13:27:18.383136 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:18.383107    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz"
Oct 30 13:27:18.432451 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:18.432419    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52bf85ca-e35d-4a85-85f4-21d6d20a191d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52bf85ca-e35d-4a85-85f4-21d6d20a191d-credential-socket\") pod \"52bf85ca-e35d-4a85-85f4-21d6d20a191d\" (UID: \"52bf85ca-e35d-4a85-85f4-21d6d20a191d\") "
Oct 30 13:27:18.432647 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:18.432503    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52bf85ca-e35d-4a85-85f4-21d6d20a191d-kube-api-access-lwf9s\" (UniqueName: \"kubernetes.io/projected/52bf85ca-e35d-4a85-85f4-21d6d20a191d-kube-api-access-lwf9s\") pod \"52bf85ca-e35d-4a85-85f4-21d6d20a191d\" (UID: \"52bf85ca-e35d-4a85-85f4-21d6d20a191d\") "
Oct 30 13:27:18.432647 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:18.432576    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52bf85ca-e35d-4a85-85f4-21d6d20a191d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52bf85ca-e35d-4a85-85f4-21d6d20a191d-istiod-ca-cert\") pod \"52bf85ca-e35d-4a85-85f4-21d6d20a191d\" (UID: \"52bf85ca-e35d-4a85-85f4-21d6d20a191d\") "
Oct 30 13:27:18.432785 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:18.432680    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52bf85ca-e35d-4a85-85f4-21d6d20a191d-credential-socket" pod "52bf85ca-e35d-4a85-85f4-21d6d20a191d" (UID: "52bf85ca-e35d-4a85-85f4-21d6d20a191d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:27:18.432785 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:18.432695    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52bf85ca-e35d-4a85-85f4-21d6d20a191d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52bf85ca-e35d-4a85-85f4-21d6d20a191d-istio-data\") pod \"52bf85ca-e35d-4a85-85f4-21d6d20a191d\" (UID: \"52bf85ca-e35d-4a85-85f4-21d6d20a191d\") "
Oct 30 13:27:18.432890 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:18.432796    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52bf85ca-e35d-4a85-85f4-21d6d20a191d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52bf85ca-e35d-4a85-85f4-21d6d20a191d-istio-envoy\") pod \"52bf85ca-e35d-4a85-85f4-21d6d20a191d\" (UID: \"52bf85ca-e35d-4a85-85f4-21d6d20a191d\") "
Oct 30 13:27:18.432890 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:18.432847    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52bf85ca-e35d-4a85-85f4-21d6d20a191d-istio-data" pod "52bf85ca-e35d-4a85-85f4-21d6d20a191d" (UID: "52bf85ca-e35d-4a85-85f4-21d6d20a191d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:27:18.432998 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:18.432913    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52bf85ca-e35d-4a85-85f4-21d6d20a191d-istiod-ca-cert" pod "52bf85ca-e35d-4a85-85f4-21d6d20a191d" (UID: "52bf85ca-e35d-4a85-85f4-21d6d20a191d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:27:18.432998 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:18.432984    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52bf85ca-e35d-4a85-85f4-21d6d20a191d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52bf85ca-e35d-4a85-85f4-21d6d20a191d-workload-certs\") pod \"52bf85ca-e35d-4a85-85f4-21d6d20a191d\" (UID: \"52bf85ca-e35d-4a85-85f4-21d6d20a191d\") "
Oct 30 13:27:18.433101 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:18.433045    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52bf85ca-e35d-4a85-85f4-21d6d20a191d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52bf85ca-e35d-4a85-85f4-21d6d20a191d-workload-socket\") pod \"52bf85ca-e35d-4a85-85f4-21d6d20a191d\" (UID: \"52bf85ca-e35d-4a85-85f4-21d6d20a191d\") "
Oct 30 13:27:18.433159 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:18.433142    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/52bf85ca-e35d-4a85-85f4-21d6d20a191d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52bf85ca-e35d-4a85-85f4-21d6d20a191d-istio-podinfo\") pod \"52bf85ca-e35d-4a85-85f4-21d6d20a191d\" (UID: \"52bf85ca-e35d-4a85-85f4-21d6d20a191d\") "
Oct 30 13:27:18.433211 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:18.433181    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52bf85ca-e35d-4a85-85f4-21d6d20a191d-istio-token\" (UniqueName: \"kubernetes.io/projected/52bf85ca-e35d-4a85-85f4-21d6d20a191d-istio-token\") pod \"52bf85ca-e35d-4a85-85f4-21d6d20a191d\" (UID: \"52bf85ca-e35d-4a85-85f4-21d6d20a191d\") "
Oct 30 13:27:18.433272 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:18.433220    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52bf85ca-e35d-4a85-85f4-21d6d20a191d-workload-certs" pod "52bf85ca-e35d-4a85-85f4-21d6d20a191d" (UID: "52bf85ca-e35d-4a85-85f4-21d6d20a191d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:27:18.433353 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:18.433320    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52bf85ca-e35d-4a85-85f4-21d6d20a191d-workload-socket" pod "52bf85ca-e35d-4a85-85f4-21d6d20a191d" (UID: "52bf85ca-e35d-4a85-85f4-21d6d20a191d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:27:18.433875 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:18.433850    4364 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52bf85ca-e35d-4a85-85f4-21d6d20a191d-workload-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:27:18.433875 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:18.433877    4364 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52bf85ca-e35d-4a85-85f4-21d6d20a191d-credential-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:27:18.434093 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:18.433891    4364 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52bf85ca-e35d-4a85-85f4-21d6d20a191d-istiod-ca-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:27:18.434093 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:18.433905    4364 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52bf85ca-e35d-4a85-85f4-21d6d20a191d-istio-data\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:27:18.434093 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:18.433917    4364 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52bf85ca-e35d-4a85-85f4-21d6d20a191d-workload-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:27:18.434908 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:18.434881    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52bf85ca-e35d-4a85-85f4-21d6d20a191d-istio-envoy" pod "52bf85ca-e35d-4a85-85f4-21d6d20a191d" (UID: "52bf85ca-e35d-4a85-85f4-21d6d20a191d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:27:18.435154 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:18.435131    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52bf85ca-e35d-4a85-85f4-21d6d20a191d-istio-podinfo" pod "52bf85ca-e35d-4a85-85f4-21d6d20a191d" (UID: "52bf85ca-e35d-4a85-85f4-21d6d20a191d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:27:18.435154 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:18.435136    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52bf85ca-e35d-4a85-85f4-21d6d20a191d-kube-api-access-lwf9s" pod "52bf85ca-e35d-4a85-85f4-21d6d20a191d" (UID: "52bf85ca-e35d-4a85-85f4-21d6d20a191d"). InnerVolumeSpecName "kube-api-access-lwf9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:27:18.435283 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:18.435252    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52bf85ca-e35d-4a85-85f4-21d6d20a191d-istio-token" pod "52bf85ca-e35d-4a85-85f4-21d6d20a191d" (UID: "52bf85ca-e35d-4a85-85f4-21d6d20a191d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:27:18.535863 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:18.535830    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwf9s\" (UniqueName: \"kubernetes.io/projected/52bf85ca-e35d-4a85-85f4-21d6d20a191d-kube-api-access-lwf9s\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:27:18.535863 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:18.535860    4364 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52bf85ca-e35d-4a85-85f4-21d6d20a191d-istio-envoy\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:27:18.535863 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:18.535870    4364 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52bf85ca-e35d-4a85-85f4-21d6d20a191d-istio-podinfo\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:27:18.536125 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:18.535878    4364 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52bf85ca-e35d-4a85-85f4-21d6d20a191d-istio-token\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:27:18.973237 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:18.973200    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="873a9c3f-d2a1-4eb8-a413-216e4d5cb48c" path="/var/lib/kubelet/pods/873a9c3f-d2a1-4eb8-a413-216e4d5cb48c/volumes"
Oct 30 13:27:19.148426 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:19.148392    4364 generic.go:356] "Generic (PLEG): container finished" podID="52bf85ca-e35d-4a85-85f4-21d6d20a191d" containerID="83f8744d68777f39cfac6983cf5de47dd6c2a5c57c8234fe41a8e2720924131f" exitCode=0
Oct 30 13:27:19.148612 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:19.148443    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz"
Oct 30 13:27:19.148612 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:19.148477    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz" event={"ID":"52bf85ca-e35d-4a85-85f4-21d6d20a191d","Type":"ContainerDied","Data":"83f8744d68777f39cfac6983cf5de47dd6c2a5c57c8234fe41a8e2720924131f"}
Oct 30 13:27:19.148612 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:19.148535    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz" event={"ID":"52bf85ca-e35d-4a85-85f4-21d6d20a191d","Type":"ContainerDied","Data":"613e7dbf5fe99263df85ebd2519cbd4c7f6c7d3133bc542f1dccc85f54a11aaa"}
Oct 30 13:27:19.148612 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:19.148559    4364 scope.go:122] "RemoveContainer" containerID="83f8744d68777f39cfac6983cf5de47dd6c2a5c57c8234fe41a8e2720924131f"
Oct 30 13:27:19.161652 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:19.161630    4364 scope.go:122] "RemoveContainer" containerID="383280d0ea2ed966b15a3e751091cf11b0522a50adeef0edab2295f885e8120d"
Oct 30 13:27:19.174939 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:19.174916    4364 scope.go:122] "RemoveContainer" containerID="83f8744d68777f39cfac6983cf5de47dd6c2a5c57c8234fe41a8e2720924131f"
Oct 30 13:27:19.175243 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:27:19.175218    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83f8744d68777f39cfac6983cf5de47dd6c2a5c57c8234fe41a8e2720924131f\": container with ID starting with 83f8744d68777f39cfac6983cf5de47dd6c2a5c57c8234fe41a8e2720924131f not found: ID does not exist" containerID="83f8744d68777f39cfac6983cf5de47dd6c2a5c57c8234fe41a8e2720924131f"
Oct 30 13:27:19.175314 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:19.175253    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83f8744d68777f39cfac6983cf5de47dd6c2a5c57c8234fe41a8e2720924131f"} err="failed to get container status \"83f8744d68777f39cfac6983cf5de47dd6c2a5c57c8234fe41a8e2720924131f\": rpc error: code = NotFound desc = could not find container \"83f8744d68777f39cfac6983cf5de47dd6c2a5c57c8234fe41a8e2720924131f\": container with ID starting with 83f8744d68777f39cfac6983cf5de47dd6c2a5c57c8234fe41a8e2720924131f not found: ID does not exist"
Oct 30 13:27:19.175314 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:19.175271    4364 scope.go:122] "RemoveContainer" containerID="383280d0ea2ed966b15a3e751091cf11b0522a50adeef0edab2295f885e8120d"
Oct 30 13:27:19.175533 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:27:19.175510    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"383280d0ea2ed966b15a3e751091cf11b0522a50adeef0edab2295f885e8120d\": container with ID starting with 383280d0ea2ed966b15a3e751091cf11b0522a50adeef0edab2295f885e8120d not found: ID does not exist" containerID="383280d0ea2ed966b15a3e751091cf11b0522a50adeef0edab2295f885e8120d"
Oct 30 13:27:19.175581 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:19.175544    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"383280d0ea2ed966b15a3e751091cf11b0522a50adeef0edab2295f885e8120d"} err="failed to get container status \"383280d0ea2ed966b15a3e751091cf11b0522a50adeef0edab2295f885e8120d\": rpc error: code = NotFound desc = could not find container \"383280d0ea2ed966b15a3e751091cf11b0522a50adeef0edab2295f885e8120d\": container with ID starting with 383280d0ea2ed966b15a3e751091cf11b0522a50adeef0edab2295f885e8120d not found: ID does not exist"
Oct 30 13:27:19.220159 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:19.220129    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz"]
Oct 30 13:27:19.238768 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:19.238694    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-b5a369-openshift-default-86687d9875-fzjxz"]
Oct 30 13:27:19.387994 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:19.387926    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:27:21.010648 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:21.010078    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52bf85ca-e35d-4a85-85f4-21d6d20a191d" path="/var/lib/kubelet/pods/52bf85ca-e35d-4a85-85f4-21d6d20a191d/volumes"
Oct 30 13:27:24.031773 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:24.031731    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8467ed-86997f5d8c-dxslk"]
Oct 30 13:27:29.032960 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:29.032914    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8467ed-86997f5d8c-dxslk" podUID="27c2bf35-ac61-4f5c-80f7-ee4d2b6c4b07" containerName="payload-pre-processing" containerID="cri-o://2e3a41e2ca1e44b4c58c41be9647fdec01b56e9d03a3cd51f974913a123f2544" gracePeriod=25
Oct 30 13:27:32.057979 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:32.057934    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8467ed-86997f5d8c-dxslk" podUID="27c2bf35-ac61-4f5c-80f7-ee4d2b6c4b07" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:27:41.511166 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:27:41.511099    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:27:41.702065 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:41.702021    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx"]
Oct 30 13:27:41.702395 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:41.702250    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx" podUID="945bfd4e-0d0d-499f-8457-5f37a0c48e28" containerName="istio-proxy" containerID="cri-o://ddfca0d35579eff6abf7a1ce3551a4bfc65fbcb296aa7d88317e07b21358ea88" gracePeriod=30
Oct 30 13:27:42.058844 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:42.058803    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8467ed-86997f5d8c-dxslk" podUID="27c2bf35-ac61-4f5c-80f7-ee4d2b6c4b07" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:27:46.995651 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:46.995616    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx"
Oct 30 13:27:47.099318 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.099281    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/945bfd4e-0d0d-499f-8457-5f37a0c48e28-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/945bfd4e-0d0d-499f-8457-5f37a0c48e28-istiod-ca-cert\") pod \"945bfd4e-0d0d-499f-8457-5f37a0c48e28\" (UID: \"945bfd4e-0d0d-499f-8457-5f37a0c48e28\") "
Oct 30 13:27:47.099526 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.099328    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/945bfd4e-0d0d-499f-8457-5f37a0c48e28-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/945bfd4e-0d0d-499f-8457-5f37a0c48e28-workload-socket\") pod \"945bfd4e-0d0d-499f-8457-5f37a0c48e28\" (UID: \"945bfd4e-0d0d-499f-8457-5f37a0c48e28\") "
Oct 30 13:27:47.099526 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.099458    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/945bfd4e-0d0d-499f-8457-5f37a0c48e28-istio-token\" (UniqueName: \"kubernetes.io/projected/945bfd4e-0d0d-499f-8457-5f37a0c48e28-istio-token\") pod \"945bfd4e-0d0d-499f-8457-5f37a0c48e28\" (UID: \"945bfd4e-0d0d-499f-8457-5f37a0c48e28\") "
Oct 30 13:27:47.099526 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.099492    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/945bfd4e-0d0d-499f-8457-5f37a0c48e28-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/945bfd4e-0d0d-499f-8457-5f37a0c48e28-istio-envoy\") pod \"945bfd4e-0d0d-499f-8457-5f37a0c48e28\" (UID: \"945bfd4e-0d0d-499f-8457-5f37a0c48e28\") "
Oct 30 13:27:47.099755 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.099536    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/945bfd4e-0d0d-499f-8457-5f37a0c48e28-kube-api-access-zpqkw\" (UniqueName: \"kubernetes.io/projected/945bfd4e-0d0d-499f-8457-5f37a0c48e28-kube-api-access-zpqkw\") pod \"945bfd4e-0d0d-499f-8457-5f37a0c48e28\" (UID: \"945bfd4e-0d0d-499f-8457-5f37a0c48e28\") "
Oct 30 13:27:47.099755 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.099642    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/945bfd4e-0d0d-499f-8457-5f37a0c48e28-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/945bfd4e-0d0d-499f-8457-5f37a0c48e28-istio-data\") pod \"945bfd4e-0d0d-499f-8457-5f37a0c48e28\" (UID: \"945bfd4e-0d0d-499f-8457-5f37a0c48e28\") "
Oct 30 13:27:47.099755 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.099740    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/945bfd4e-0d0d-499f-8457-5f37a0c48e28-workload-socket" pod "945bfd4e-0d0d-499f-8457-5f37a0c48e28" (UID: "945bfd4e-0d0d-499f-8457-5f37a0c48e28"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:27:47.099929 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.099746    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/945bfd4e-0d0d-499f-8457-5f37a0c48e28-istiod-ca-cert" pod "945bfd4e-0d0d-499f-8457-5f37a0c48e28" (UID: "945bfd4e-0d0d-499f-8457-5f37a0c48e28"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:27:47.099929 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.099916    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/945bfd4e-0d0d-499f-8457-5f37a0c48e28-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/945bfd4e-0d0d-499f-8457-5f37a0c48e28-workload-certs\") pod \"945bfd4e-0d0d-499f-8457-5f37a0c48e28\" (UID: \"945bfd4e-0d0d-499f-8457-5f37a0c48e28\") "
Oct 30 13:27:47.100048 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.099993    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/945bfd4e-0d0d-499f-8457-5f37a0c48e28-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/945bfd4e-0d0d-499f-8457-5f37a0c48e28-istio-podinfo\") pod \"945bfd4e-0d0d-499f-8457-5f37a0c48e28\" (UID: \"945bfd4e-0d0d-499f-8457-5f37a0c48e28\") "
Oct 30 13:27:47.100048 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.100033    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/945bfd4e-0d0d-499f-8457-5f37a0c48e28-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/945bfd4e-0d0d-499f-8457-5f37a0c48e28-credential-socket\") pod \"945bfd4e-0d0d-499f-8457-5f37a0c48e28\" (UID: \"945bfd4e-0d0d-499f-8457-5f37a0c48e28\") "
Oct 30 13:27:47.100198 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.100173    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/945bfd4e-0d0d-499f-8457-5f37a0c48e28-istio-data" pod "945bfd4e-0d0d-499f-8457-5f37a0c48e28" (UID: "945bfd4e-0d0d-499f-8457-5f37a0c48e28"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:27:47.100436 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.100409    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/945bfd4e-0d0d-499f-8457-5f37a0c48e28-workload-certs" pod "945bfd4e-0d0d-499f-8457-5f37a0c48e28" (UID: "945bfd4e-0d0d-499f-8457-5f37a0c48e28"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:27:47.100681 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.100655    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/945bfd4e-0d0d-499f-8457-5f37a0c48e28-credential-socket" pod "945bfd4e-0d0d-499f-8457-5f37a0c48e28" (UID: "945bfd4e-0d0d-499f-8457-5f37a0c48e28"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:27:47.100898 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.100862    4364 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/945bfd4e-0d0d-499f-8457-5f37a0c48e28-workload-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:27:47.101016 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.101003    4364 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/945bfd4e-0d0d-499f-8457-5f37a0c48e28-istiod-ca-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:27:47.101104 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.101093    4364 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/945bfd4e-0d0d-499f-8457-5f37a0c48e28-workload-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:27:47.101190 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.101179    4364 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/945bfd4e-0d0d-499f-8457-5f37a0c48e28-istio-data\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:27:47.102261 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.102217    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/945bfd4e-0d0d-499f-8457-5f37a0c48e28-istio-envoy" pod "945bfd4e-0d0d-499f-8457-5f37a0c48e28" (UID: "945bfd4e-0d0d-499f-8457-5f37a0c48e28"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:27:47.102445 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.102417    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/945bfd4e-0d0d-499f-8457-5f37a0c48e28-kube-api-access-zpqkw" pod "945bfd4e-0d0d-499f-8457-5f37a0c48e28" (UID: "945bfd4e-0d0d-499f-8457-5f37a0c48e28"). InnerVolumeSpecName "kube-api-access-zpqkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:27:47.102715 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.102684    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/945bfd4e-0d0d-499f-8457-5f37a0c48e28-istio-token" pod "945bfd4e-0d0d-499f-8457-5f37a0c48e28" (UID: "945bfd4e-0d0d-499f-8457-5f37a0c48e28"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:27:47.103340 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.103317    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/945bfd4e-0d0d-499f-8457-5f37a0c48e28-istio-podinfo" pod "945bfd4e-0d0d-499f-8457-5f37a0c48e28" (UID: "945bfd4e-0d0d-499f-8457-5f37a0c48e28"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:27:47.203239 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.203202    4364 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/945bfd4e-0d0d-499f-8457-5f37a0c48e28-istio-token\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:27:47.203239 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.203237    4364 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/945bfd4e-0d0d-499f-8457-5f37a0c48e28-istio-envoy\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:27:47.203450 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.203251    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpqkw\" (UniqueName: \"kubernetes.io/projected/945bfd4e-0d0d-499f-8457-5f37a0c48e28-kube-api-access-zpqkw\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:27:47.203450 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.203262    4364 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/945bfd4e-0d0d-499f-8457-5f37a0c48e28-istio-podinfo\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:27:47.203450 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.203276    4364 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/945bfd4e-0d0d-499f-8457-5f37a0c48e28-credential-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:27:47.268925 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.268849    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b97b9de-kvsj7"]
Oct 30 13:27:47.269707 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.269688    4364 cpu_manager.go:414] "containerMap: removing container" podUID="52bf85ca-e35d-4a85-85f4-21d6d20a191d" containerName="istio-proxy"
Oct 30 13:27:47.269769 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.269709    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52bf85ca-e35d-4a85-85f4-21d6d20a191d" containerName="istio-proxy"
Oct 30 13:27:47.269769 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.269721    4364 cpu_manager.go:414] "containerMap: removing container" podUID="0cd7b23d-f2dc-49be-b0a5-0dd233c8faa9" containerName="payload-processing"
Oct 30 13:27:47.269769 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.269727    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cd7b23d-f2dc-49be-b0a5-0dd233c8faa9" containerName="payload-processing"
Oct 30 13:27:47.269769 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.269739    4364 cpu_manager.go:414] "containerMap: removing container" podUID="945bfd4e-0d0d-499f-8457-5f37a0c48e28" containerName="istio-proxy"
Oct 30 13:27:47.269769 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.269744    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="945bfd4e-0d0d-499f-8457-5f37a0c48e28" containerName="istio-proxy"
Oct 30 13:27:47.269769 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.269755    4364 cpu_manager.go:414] "containerMap: removing container" podUID="d52aab51-7fdf-4be6-8893-b5fe88dbe341" containerName="main"
Oct 30 13:27:47.269769 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.269760    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d52aab51-7fdf-4be6-8893-b5fe88dbe341" containerName="main"
Oct 30 13:27:47.269769 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.269767    4364 cpu_manager.go:414] "containerMap: removing container" podUID="6041e6f1-3ee9-45b8-bbc5-f6c73436f7f5" containerName="revoke-keys"
Oct 30 13:27:47.270017 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.269773    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6041e6f1-3ee9-45b8-bbc5-f6c73436f7f5" containerName="revoke-keys"
Oct 30 13:27:47.270017 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.269783    4364 cpu_manager.go:414] "containerMap: removing container" podUID="f9acff8a-e701-4ab6-ba2f-9c8484b7bdcb" containerName="payload-pre-processing"
Oct 30 13:27:47.270017 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.269787    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9acff8a-e701-4ab6-ba2f-9c8484b7bdcb" containerName="payload-pre-processing"
Oct 30 13:27:47.270017 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.269797    4364 cpu_manager.go:414] "containerMap: removing container" podUID="873a9c3f-d2a1-4eb8-a413-216e4d5cb48c" containerName="maas-api"
Oct 30 13:27:47.270017 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.269802    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="873a9c3f-d2a1-4eb8-a413-216e4d5cb48c" containerName="maas-api"
Oct 30 13:27:47.270017 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.269819    4364 cpu_manager.go:414] "containerMap: removing container" podUID="5895871b-7b5c-4ba0-8b6e-e761ca5b1adf" containerName="revoke-keys"
Oct 30 13:27:47.270017 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.269826    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5895871b-7b5c-4ba0-8b6e-e761ca5b1adf" containerName="revoke-keys"
Oct 30 13:27:47.270017 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.269844    4364 cpu_manager.go:414] "containerMap: removing container" podUID="0f77c6bd-36f6-45cc-92ed-b22fa718700a" containerName="istio-proxy"
Oct 30 13:27:47.270017 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.269849    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f77c6bd-36f6-45cc-92ed-b22fa718700a" containerName="istio-proxy"
Oct 30 13:27:47.270017 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.269855    4364 cpu_manager.go:414] "containerMap: removing container" podUID="517cf09e-dcec-4fe5-ad42-d7e15cd1e48a" containerName="main"
Oct 30 13:27:47.270017 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.269859    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="517cf09e-dcec-4fe5-ad42-d7e15cd1e48a" containerName="main"
Oct 30 13:27:47.270017 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.269873    4364 cpu_manager.go:414] "containerMap: removing container" podUID="0f77c6bd-36f6-45cc-92ed-b22fa718700a" containerName="istio-proxy"
Oct 30 13:27:47.270017 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.269877    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f77c6bd-36f6-45cc-92ed-b22fa718700a" containerName="istio-proxy"
Oct 30 13:27:47.270017 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.269886    4364 cpu_manager.go:414] "containerMap: removing container" podUID="a9c7d05a-ebeb-4506-88f9-5932b1730822" containerName="payload-pre-processing"
Oct 30 13:27:47.270017 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.269891    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9c7d05a-ebeb-4506-88f9-5932b1730822" containerName="payload-pre-processing"
Oct 30 13:27:47.270017 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.269899    4364 cpu_manager.go:414] "containerMap: removing container" podUID="3fd28ec8-f8ba-400b-940a-9c6b5c516936" containerName="main"
Oct 30 13:27:47.270017 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.269904    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fd28ec8-f8ba-400b-940a-9c6b5c516936" containerName="main"
Oct 30 13:27:47.270017 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.270024    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fd28ec8-f8ba-400b-940a-9c6b5c516936" containerName="main"
Oct 30 13:27:47.270638 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.270035    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="945bfd4e-0d0d-499f-8457-5f37a0c48e28" containerName="istio-proxy"
Oct 30 13:27:47.270638 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.270041    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="52bf85ca-e35d-4a85-85f4-21d6d20a191d" containerName="istio-proxy"
Oct 30 13:27:47.270638 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.270048    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="52bf85ca-e35d-4a85-85f4-21d6d20a191d" containerName="istio-proxy"
Oct 30 13:27:47.270638 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.270055    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cd7b23d-f2dc-49be-b0a5-0dd233c8faa9" containerName="payload-processing"
Oct 30 13:27:47.270638 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.270059    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="945bfd4e-0d0d-499f-8457-5f37a0c48e28" containerName="istio-proxy"
Oct 30 13:27:47.270638 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.270069    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="d52aab51-7fdf-4be6-8893-b5fe88dbe341" containerName="main"
Oct 30 13:27:47.270638 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.270077    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="6041e6f1-3ee9-45b8-bbc5-f6c73436f7f5" containerName="revoke-keys"
Oct 30 13:27:47.270638 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.270083    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9acff8a-e701-4ab6-ba2f-9c8484b7bdcb" containerName="payload-pre-processing"
Oct 30 13:27:47.270638 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.270090    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="873a9c3f-d2a1-4eb8-a413-216e4d5cb48c" containerName="maas-api"
Oct 30 13:27:47.270638 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.270097    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="5895871b-7b5c-4ba0-8b6e-e761ca5b1adf" containerName="revoke-keys"
Oct 30 13:27:47.270638 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.270106    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f77c6bd-36f6-45cc-92ed-b22fa718700a" containerName="istio-proxy"
Oct 30 13:27:47.270638 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.270110    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="517cf09e-dcec-4fe5-ad42-d7e15cd1e48a" containerName="main"
Oct 30 13:27:47.270638 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.270118    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f77c6bd-36f6-45cc-92ed-b22fa718700a" containerName="istio-proxy"
Oct 30 13:27:47.270638 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.270124    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9c7d05a-ebeb-4506-88f9-5932b1730822" containerName="payload-pre-processing"
Oct 30 13:27:47.274277 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.274261    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b97b9de-kvsj7"
Oct 30 13:27:47.290137 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.289811    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:27:47.290303 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.290204    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\""
Oct 30 13:27:47.293731 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.293705    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b97b9de-kvsj7"]
Oct 30 13:27:47.304828 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.304797    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4xfw\" (UniqueName: \"kubernetes.io/projected/adc1ed51-9afb-4fe4-80ef-92d0109836e0-kube-api-access-g4xfw\") pod \"maas-api-revoke-sub-6b97b9de-kvsj7\" (UID: \"adc1ed51-9afb-4fe4-80ef-92d0109836e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b97b9de-kvsj7"
Oct 30 13:27:47.305161 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.305140    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adc1ed51-9afb-4fe4-80ef-92d0109836e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b97b9de-kvsj7\" (UID: \"adc1ed51-9afb-4fe4-80ef-92d0109836e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b97b9de-kvsj7"
Oct 30 13:27:47.305384 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.305352    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adc1ed51-9afb-4fe4-80ef-92d0109836e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b97b9de-kvsj7\" (UID: \"adc1ed51-9afb-4fe4-80ef-92d0109836e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b97b9de-kvsj7"
Oct 30 13:27:47.407222 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.407188    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4xfw\" (UniqueName: \"kubernetes.io/projected/adc1ed51-9afb-4fe4-80ef-92d0109836e0-kube-api-access-g4xfw\") pod \"maas-api-revoke-sub-6b97b9de-kvsj7\" (UID: \"adc1ed51-9afb-4fe4-80ef-92d0109836e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b97b9de-kvsj7"
Oct 30 13:27:47.407438 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.407340    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adc1ed51-9afb-4fe4-80ef-92d0109836e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b97b9de-kvsj7\" (UID: \"adc1ed51-9afb-4fe4-80ef-92d0109836e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b97b9de-kvsj7"
Oct 30 13:27:47.407438 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.407415    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adc1ed51-9afb-4fe4-80ef-92d0109836e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b97b9de-kvsj7\" (UID: \"adc1ed51-9afb-4fe4-80ef-92d0109836e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b97b9de-kvsj7"
Oct 30 13:27:47.408011 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.407989    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adc1ed51-9afb-4fe4-80ef-92d0109836e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b97b9de-kvsj7\" (UID: \"adc1ed51-9afb-4fe4-80ef-92d0109836e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b97b9de-kvsj7"
Oct 30 13:27:47.409744 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.409726    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adc1ed51-9afb-4fe4-80ef-92d0109836e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b97b9de-kvsj7\" (UID: \"adc1ed51-9afb-4fe4-80ef-92d0109836e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b97b9de-kvsj7"
Oct 30 13:27:47.445087 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.445054    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4xfw\" (UniqueName: \"kubernetes.io/projected/adc1ed51-9afb-4fe4-80ef-92d0109836e0-kube-api-access-g4xfw\") pod \"maas-api-revoke-sub-6b97b9de-kvsj7\" (UID: \"adc1ed51-9afb-4fe4-80ef-92d0109836e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b97b9de-kvsj7"
Oct 30 13:27:47.589819 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.589731    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b97b9de-kvsj7"
Oct 30 13:27:47.604936 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.604904    4364 generic.go:356] "Generic (PLEG): container finished" podID="945bfd4e-0d0d-499f-8457-5f37a0c48e28" containerID="ddfca0d35579eff6abf7a1ce3551a4bfc65fbcb296aa7d88317e07b21358ea88" exitCode=0
Oct 30 13:27:47.605035 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.604977    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx"
Oct 30 13:27:47.605035 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.604980    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx" event={"ID":"945bfd4e-0d0d-499f-8457-5f37a0c48e28","Type":"ContainerDied","Data":"ddfca0d35579eff6abf7a1ce3551a4bfc65fbcb296aa7d88317e07b21358ea88"}
Oct 30 13:27:47.605128 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.605061    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx" event={"ID":"945bfd4e-0d0d-499f-8457-5f37a0c48e28","Type":"ContainerDied","Data":"66e62a9d96da5d5116d3a5cb176bccd9cc88fb4fa1b2f8f7f8ccf284f9d68771"}
Oct 30 13:27:47.605128 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.605084    4364 scope.go:122] "RemoveContainer" containerID="ddfca0d35579eff6abf7a1ce3551a4bfc65fbcb296aa7d88317e07b21358ea88"
Oct 30 13:27:47.616458 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.616435    4364 scope.go:122] "RemoveContainer" containerID="632629d92e8dc7487e27cd045c846c6f44737ecfe34b41e6779ba9ddb88775af"
Oct 30 13:27:47.627276 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.627255    4364 scope.go:122] "RemoveContainer" containerID="ddfca0d35579eff6abf7a1ce3551a4bfc65fbcb296aa7d88317e07b21358ea88"
Oct 30 13:27:47.627584 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:27:47.627564    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddfca0d35579eff6abf7a1ce3551a4bfc65fbcb296aa7d88317e07b21358ea88\": container with ID starting with ddfca0d35579eff6abf7a1ce3551a4bfc65fbcb296aa7d88317e07b21358ea88 not found: ID does not exist" containerID="ddfca0d35579eff6abf7a1ce3551a4bfc65fbcb296aa7d88317e07b21358ea88"
Oct 30 13:27:47.627690 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.627609    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddfca0d35579eff6abf7a1ce3551a4bfc65fbcb296aa7d88317e07b21358ea88"} err="failed to get container status \"ddfca0d35579eff6abf7a1ce3551a4bfc65fbcb296aa7d88317e07b21358ea88\": rpc error: code = NotFound desc = could not find container \"ddfca0d35579eff6abf7a1ce3551a4bfc65fbcb296aa7d88317e07b21358ea88\": container with ID starting with ddfca0d35579eff6abf7a1ce3551a4bfc65fbcb296aa7d88317e07b21358ea88 not found: ID does not exist"
Oct 30 13:27:47.627690 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.627636    4364 scope.go:122] "RemoveContainer" containerID="632629d92e8dc7487e27cd045c846c6f44737ecfe34b41e6779ba9ddb88775af"
Oct 30 13:27:47.627930 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:27:47.627911    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"632629d92e8dc7487e27cd045c846c6f44737ecfe34b41e6779ba9ddb88775af\": container with ID starting with 632629d92e8dc7487e27cd045c846c6f44737ecfe34b41e6779ba9ddb88775af not found: ID does not exist" containerID="632629d92e8dc7487e27cd045c846c6f44737ecfe34b41e6779ba9ddb88775af"
Oct 30 13:27:47.627994 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.627936    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"632629d92e8dc7487e27cd045c846c6f44737ecfe34b41e6779ba9ddb88775af"} err="failed to get container status \"632629d92e8dc7487e27cd045c846c6f44737ecfe34b41e6779ba9ddb88775af\": rpc error: code = NotFound desc = could not find container \"632629d92e8dc7487e27cd045c846c6f44737ecfe34b41e6779ba9ddb88775af\": container with ID starting with 632629d92e8dc7487e27cd045c846c6f44737ecfe34b41e6779ba9ddb88775af not found: ID does not exist"
Oct 30 13:27:47.687116 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.674327    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx"]
Oct 30 13:27:47.687116 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.683572    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-0ad384-openshift-default-5f7ff54d9f-9g9dx"]
Oct 30 13:27:47.917208 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:47.917175    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b97b9de-kvsj7"]
Oct 30 13:27:47.917491 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:27:47.917469    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadc1ed51_9afb_4fe4_80ef_92d0109836e0.slice/crio-f38f0b6a837e8d5476aab0ad5ab7ed795b015d0a404489859d6d851eeadd32f5 WatchSource:0}: Error finding container f38f0b6a837e8d5476aab0ad5ab7ed795b015d0a404489859d6d851eeadd32f5: Status 404 returned error can't find the container with id f38f0b6a837e8d5476aab0ad5ab7ed795b015d0a404489859d6d851eeadd32f5
Oct 30 13:27:48.616477 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:48.616387    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b97b9de-kvsj7" event={"ID":"adc1ed51-9afb-4fe4-80ef-92d0109836e0","Type":"ContainerStarted","Data":"995a7d010a3b52ede92973828802157c68009bb81a3bca9b6d507cb6db9e77fc"}
Oct 30 13:27:48.616477 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:48.616443    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b97b9de-kvsj7" event={"ID":"adc1ed51-9afb-4fe4-80ef-92d0109836e0","Type":"ContainerStarted","Data":"f38f0b6a837e8d5476aab0ad5ab7ed795b015d0a404489859d6d851eeadd32f5"}
Oct 30 13:27:48.660440 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:48.660385    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b97b9de-kvsj7" podStartSLOduration=1.6603711730000001 podStartE2EDuration="1.660371173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:27:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:27:48.655992917 +0000 UTC m=+2167.822357565" watchObservedRunningTime="2026-10-30 13:27:48.660371173 +0000 UTC m=+2167.826735830"
Oct 30 13:27:48.972162 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:48.972120    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="945bfd4e-0d0d-499f-8457-5f37a0c48e28" path="/var/lib/kubelet/pods/945bfd4e-0d0d-499f-8457-5f37a0c48e28/volumes"
Oct 30 13:27:49.321684 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:49.321655    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8467ed-86997f5d8c-dxslk"
Oct 30 13:27:49.449867 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:49.449835    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27c2bf35-ac61-4f5c-80f7-ee4d2b6c4b07-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27c2bf35-ac61-4f5c-80f7-ee4d2b6c4b07-plugins-config-volume\") pod \"27c2bf35-ac61-4f5c-80f7-ee4d2b6c4b07\" (UID: \"27c2bf35-ac61-4f5c-80f7-ee4d2b6c4b07\") "
Oct 30 13:27:49.450109 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:49.450097    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27c2bf35-ac61-4f5c-80f7-ee4d2b6c4b07-kube-api-access-ngw2k\" (UniqueName: \"kubernetes.io/projected/27c2bf35-ac61-4f5c-80f7-ee4d2b6c4b07-kube-api-access-ngw2k\") pod \"27c2bf35-ac61-4f5c-80f7-ee4d2b6c4b07\" (UID: \"27c2bf35-ac61-4f5c-80f7-ee4d2b6c4b07\") "
Oct 30 13:27:49.450220 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:49.450198    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27c2bf35-ac61-4f5c-80f7-ee4d2b6c4b07-plugins-config-volume" pod "27c2bf35-ac61-4f5c-80f7-ee4d2b6c4b07" (UID: "27c2bf35-ac61-4f5c-80f7-ee4d2b6c4b07"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:27:49.450575 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:49.450561    4364 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27c2bf35-ac61-4f5c-80f7-ee4d2b6c4b07-plugins-config-volume\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:27:49.452175 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:49.452152    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27c2bf35-ac61-4f5c-80f7-ee4d2b6c4b07-kube-api-access-ngw2k" pod "27c2bf35-ac61-4f5c-80f7-ee4d2b6c4b07" (UID: "27c2bf35-ac61-4f5c-80f7-ee4d2b6c4b07"). InnerVolumeSpecName "kube-api-access-ngw2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:27:49.552567 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:49.552488    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngw2k\" (UniqueName: \"kubernetes.io/projected/27c2bf35-ac61-4f5c-80f7-ee4d2b6c4b07-kube-api-access-ngw2k\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:27:49.625839 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:49.625803    4364 generic.go:356] "Generic (PLEG): container finished" podID="27c2bf35-ac61-4f5c-80f7-ee4d2b6c4b07" containerID="2e3a41e2ca1e44b4c58c41be9647fdec01b56e9d03a3cd51f974913a123f2544" exitCode=0
Oct 30 13:27:49.626237 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:49.625848    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8467ed-86997f5d8c-dxslk"
Oct 30 13:27:49.626237 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:49.625881    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8467ed-86997f5d8c-dxslk" event={"ID":"27c2bf35-ac61-4f5c-80f7-ee4d2b6c4b07","Type":"ContainerDied","Data":"2e3a41e2ca1e44b4c58c41be9647fdec01b56e9d03a3cd51f974913a123f2544"}
Oct 30 13:27:49.626237 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:49.625933    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8467ed-86997f5d8c-dxslk" event={"ID":"27c2bf35-ac61-4f5c-80f7-ee4d2b6c4b07","Type":"ContainerDied","Data":"7f62ccea32caa5fa0ce845ed9601c09520ee27032cd0fc0b13547e69654d76db"}
Oct 30 13:27:49.626237 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:49.625956    4364 scope.go:122] "RemoveContainer" containerID="2e3a41e2ca1e44b4c58c41be9647fdec01b56e9d03a3cd51f974913a123f2544"
Oct 30 13:27:49.628023 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:49.627996    4364 generic.go:356] "Generic (PLEG): container finished" podID="adc1ed51-9afb-4fe4-80ef-92d0109836e0" containerID="995a7d010a3b52ede92973828802157c68009bb81a3bca9b6d507cb6db9e77fc" exitCode=0
Oct 30 13:27:49.628172 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:49.628147    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b97b9de-kvsj7" event={"ID":"adc1ed51-9afb-4fe4-80ef-92d0109836e0","Type":"ContainerDied","Data":"995a7d010a3b52ede92973828802157c68009bb81a3bca9b6d507cb6db9e77fc"}
Oct 30 13:27:49.637828 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:49.637811    4364 scope.go:122] "RemoveContainer" containerID="2e3a41e2ca1e44b4c58c41be9647fdec01b56e9d03a3cd51f974913a123f2544"
Oct 30 13:27:49.638070 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:27:49.638052    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e3a41e2ca1e44b4c58c41be9647fdec01b56e9d03a3cd51f974913a123f2544\": container with ID starting with 2e3a41e2ca1e44b4c58c41be9647fdec01b56e9d03a3cd51f974913a123f2544 not found: ID does not exist" containerID="2e3a41e2ca1e44b4c58c41be9647fdec01b56e9d03a3cd51f974913a123f2544"
Oct 30 13:27:49.638111 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:49.638080    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e3a41e2ca1e44b4c58c41be9647fdec01b56e9d03a3cd51f974913a123f2544"} err="failed to get container status \"2e3a41e2ca1e44b4c58c41be9647fdec01b56e9d03a3cd51f974913a123f2544\": rpc error: code = NotFound desc = could not find container \"2e3a41e2ca1e44b4c58c41be9647fdec01b56e9d03a3cd51f974913a123f2544\": container with ID starting with 2e3a41e2ca1e44b4c58c41be9647fdec01b56e9d03a3cd51f974913a123f2544 not found: ID does not exist"
Oct 30 13:27:49.746719 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:49.746681    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8467ed-86997f5d8c-dxslk"]
Oct 30 13:27:49.751425 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:49.751395    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8467ed-86997f5d8c-dxslk"]
Oct 30 13:27:50.812779 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:50.812754    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b97b9de-kvsj7"
Oct 30 13:27:50.888079 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:50.887990    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adc1ed51-9afb-4fe4-80ef-92d0109836e0-kube-api-access-g4xfw\" (UniqueName: \"kubernetes.io/projected/adc1ed51-9afb-4fe4-80ef-92d0109836e0-kube-api-access-g4xfw\") pod \"adc1ed51-9afb-4fe4-80ef-92d0109836e0\" (UID: \"adc1ed51-9afb-4fe4-80ef-92d0109836e0\") "
Oct 30 13:27:50.888079 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:50.888056    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/adc1ed51-9afb-4fe4-80ef-92d0109836e0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adc1ed51-9afb-4fe4-80ef-92d0109836e0-maas-api-xks-ca\") pod \"adc1ed51-9afb-4fe4-80ef-92d0109836e0\" (UID: \"adc1ed51-9afb-4fe4-80ef-92d0109836e0\") "
Oct 30 13:27:50.888335 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:50.888166    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adc1ed51-9afb-4fe4-80ef-92d0109836e0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adc1ed51-9afb-4fe4-80ef-92d0109836e0-maas-api-openshift-ca\") pod \"adc1ed51-9afb-4fe4-80ef-92d0109836e0\" (UID: \"adc1ed51-9afb-4fe4-80ef-92d0109836e0\") "
Oct 30 13:27:50.888627 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:50.888570    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adc1ed51-9afb-4fe4-80ef-92d0109836e0-maas-api-openshift-ca" pod "adc1ed51-9afb-4fe4-80ef-92d0109836e0" (UID: "adc1ed51-9afb-4fe4-80ef-92d0109836e0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:27:50.889025 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:50.889003    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adc1ed51-9afb-4fe4-80ef-92d0109836e0-maas-api-openshift-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:27:50.890307 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:50.890283    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adc1ed51-9afb-4fe4-80ef-92d0109836e0-maas-api-xks-ca" pod "adc1ed51-9afb-4fe4-80ef-92d0109836e0" (UID: "adc1ed51-9afb-4fe4-80ef-92d0109836e0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:27:50.890593 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:50.890573    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adc1ed51-9afb-4fe4-80ef-92d0109836e0-kube-api-access-g4xfw" pod "adc1ed51-9afb-4fe4-80ef-92d0109836e0" (UID: "adc1ed51-9afb-4fe4-80ef-92d0109836e0"). InnerVolumeSpecName "kube-api-access-g4xfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:27:50.972358 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:50.972325    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27c2bf35-ac61-4f5c-80f7-ee4d2b6c4b07" path="/var/lib/kubelet/pods/27c2bf35-ac61-4f5c-80f7-ee4d2b6c4b07/volumes"
Oct 30 13:27:50.990270 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:50.990232    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4xfw\" (UniqueName: \"kubernetes.io/projected/adc1ed51-9afb-4fe4-80ef-92d0109836e0-kube-api-access-g4xfw\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:27:50.990270 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:50.990269    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adc1ed51-9afb-4fe4-80ef-92d0109836e0-maas-api-xks-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:27:51.647315 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:51.647285    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b97b9de-kvsj7"
Oct 30 13:27:51.647510 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:51.647312    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b97b9de-kvsj7" event={"ID":"adc1ed51-9afb-4fe4-80ef-92d0109836e0","Type":"ContainerDied","Data":"f38f0b6a837e8d5476aab0ad5ab7ed795b015d0a404489859d6d851eeadd32f5"}
Oct 30 13:27:51.647510 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:51.647394    4364 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f38f0b6a837e8d5476aab0ad5ab7ed795b015d0a404489859d6d851eeadd32f5"
Oct 30 13:27:52.554460 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:52.554422    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-09e4c0-5454d677c9-bqbgj"]
Oct 30 13:27:56.252860 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:56.252817    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-8d82d5/sub-test-model-8d82d5-kserve-d4df748cf-hqr5x"]
Oct 30 13:27:56.254061 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:56.254027    4364 cpu_manager.go:414] "containerMap: removing container" podUID="27c2bf35-ac61-4f5c-80f7-ee4d2b6c4b07" containerName="payload-pre-processing"
Oct 30 13:27:56.254061 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:56.254058    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27c2bf35-ac61-4f5c-80f7-ee4d2b6c4b07" containerName="payload-pre-processing"
Oct 30 13:27:56.254287 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:56.254107    4364 cpu_manager.go:414] "containerMap: removing container" podUID="52bf85ca-e35d-4a85-85f4-21d6d20a191d" containerName="istio-proxy"
Oct 30 13:27:56.254287 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:56.254116    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52bf85ca-e35d-4a85-85f4-21d6d20a191d" containerName="istio-proxy"
Oct 30 13:27:56.254287 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:56.254137    4364 cpu_manager.go:414] "containerMap: removing container" podUID="945bfd4e-0d0d-499f-8457-5f37a0c48e28" containerName="istio-proxy"
Oct 30 13:27:56.254287 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:56.254166    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="945bfd4e-0d0d-499f-8457-5f37a0c48e28" containerName="istio-proxy"
Oct 30 13:27:56.254287 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:56.254214    4364 cpu_manager.go:414] "containerMap: removing container" podUID="adc1ed51-9afb-4fe4-80ef-92d0109836e0" containerName="revoke-keys"
Oct 30 13:27:56.254287 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:56.254222    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adc1ed51-9afb-4fe4-80ef-92d0109836e0" containerName="revoke-keys"
Oct 30 13:27:56.254625 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:56.254418    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="adc1ed51-9afb-4fe4-80ef-92d0109836e0" containerName="revoke-keys"
Oct 30 13:27:56.254625 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:56.254439    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="27c2bf35-ac61-4f5c-80f7-ee4d2b6c4b07" containerName="payload-pre-processing"
Oct 30 13:27:56.260036 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:56.260004    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-8d82d5/sub-test-model-8d82d5-kserve-d4df748cf-hqr5x"
Oct 30 13:27:56.282960 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:56.282924    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-8d82d5/sub-test-model-8d82d5-kserve-d4df748cf-hqr5x"]
Oct 30 13:27:56.291548 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:56.291519    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-8d82d5\"/\"default-dockercfg-792k5\""
Oct 30 13:27:56.292169 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:56.292140    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74468d4c-64a8-4b40-8d14-e9954caadc22-tls-certs\") pod \"sub-test-model-8d82d5-kserve-d4df748cf-hqr5x\" (UID: \"74468d4c-64a8-4b40-8d14-e9954caadc22\") " pod="ai-tenant-e2e-shared-b-w1-8d82d5/sub-test-model-8d82d5-kserve-d4df748cf-hqr5x"
Oct 30 13:27:56.292294 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:56.292181    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-8d82d5\"/\"openshift-service-ca.crt\""
Oct 30 13:27:56.292294 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:56.292271    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4nnx\" (UniqueName: \"kubernetes.io/projected/74468d4c-64a8-4b40-8d14-e9954caadc22-kube-api-access-s4nnx\") pod \"sub-test-model-8d82d5-kserve-d4df748cf-hqr5x\" (UID: \"74468d4c-64a8-4b40-8d14-e9954caadc22\") " pod="ai-tenant-e2e-shared-b-w1-8d82d5/sub-test-model-8d82d5-kserve-d4df748cf-hqr5x"
Oct 30 13:27:56.292294 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:56.292288    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-8d82d5\"/\"kube-root-ca.crt\""
Oct 30 13:27:56.292456 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:56.292301    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74468d4c-64a8-4b40-8d14-e9954caadc22-model-cache\") pod \"sub-test-model-8d82d5-kserve-d4df748cf-hqr5x\" (UID: \"74468d4c-64a8-4b40-8d14-e9954caadc22\") " pod="ai-tenant-e2e-shared-b-w1-8d82d5/sub-test-model-8d82d5-kserve-d4df748cf-hqr5x"
Oct 30 13:27:56.292456 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:56.292406    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-8d82d5\"/\"sub-test-model-8d82d5-kserve-self-signed-certs\""
Oct 30 13:27:56.292564 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:56.292490    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74468d4c-64a8-4b40-8d14-e9954caadc22-home\") pod \"sub-test-model-8d82d5-kserve-d4df748cf-hqr5x\" (UID: \"74468d4c-64a8-4b40-8d14-e9954caadc22\") " pod="ai-tenant-e2e-shared-b-w1-8d82d5/sub-test-model-8d82d5-kserve-d4df748cf-hqr5x"
Oct 30 13:27:56.292564 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:56.292515    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74468d4c-64a8-4b40-8d14-e9954caadc22-dshm\") pod \"sub-test-model-8d82d5-kserve-d4df748cf-hqr5x\" (UID: \"74468d4c-64a8-4b40-8d14-e9954caadc22\") " pod="ai-tenant-e2e-shared-b-w1-8d82d5/sub-test-model-8d82d5-kserve-d4df748cf-hqr5x"
Oct 30 13:27:56.292719 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:56.292623    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74468d4c-64a8-4b40-8d14-e9954caadc22-tmp-dir\") pod \"sub-test-model-8d82d5-kserve-d4df748cf-hqr5x\" (UID: \"74468d4c-64a8-4b40-8d14-e9954caadc22\") " pod="ai-tenant-e2e-shared-b-w1-8d82d5/sub-test-model-8d82d5-kserve-d4df748cf-hqr5x"
Oct 30 13:27:56.393958 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:56.393922    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74468d4c-64a8-4b40-8d14-e9954caadc22-tls-certs\") pod \"sub-test-model-8d82d5-kserve-d4df748cf-hqr5x\" (UID: \"74468d4c-64a8-4b40-8d14-e9954caadc22\") " pod="ai-tenant-e2e-shared-b-w1-8d82d5/sub-test-model-8d82d5-kserve-d4df748cf-hqr5x"
Oct 30 13:27:56.394167 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:56.394039    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4nnx\" (UniqueName: \"kubernetes.io/projected/74468d4c-64a8-4b40-8d14-e9954caadc22-kube-api-access-s4nnx\") pod \"sub-test-model-8d82d5-kserve-d4df748cf-hqr5x\" (UID: \"74468d4c-64a8-4b40-8d14-e9954caadc22\") " pod="ai-tenant-e2e-shared-b-w1-8d82d5/sub-test-model-8d82d5-kserve-d4df748cf-hqr5x"
Oct 30 13:27:56.394167 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:56.394070    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74468d4c-64a8-4b40-8d14-e9954caadc22-model-cache\") pod \"sub-test-model-8d82d5-kserve-d4df748cf-hqr5x\" (UID: \"74468d4c-64a8-4b40-8d14-e9954caadc22\") " pod="ai-tenant-e2e-shared-b-w1-8d82d5/sub-test-model-8d82d5-kserve-d4df748cf-hqr5x"
Oct 30 13:27:56.394306 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:56.394252    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74468d4c-64a8-4b40-8d14-e9954caadc22-home\") pod \"sub-test-model-8d82d5-kserve-d4df748cf-hqr5x\" (UID: \"74468d4c-64a8-4b40-8d14-e9954caadc22\") " pod="ai-tenant-e2e-shared-b-w1-8d82d5/sub-test-model-8d82d5-kserve-d4df748cf-hqr5x"
Oct 30 13:27:56.394306 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:56.394277    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74468d4c-64a8-4b40-8d14-e9954caadc22-dshm\") pod \"sub-test-model-8d82d5-kserve-d4df748cf-hqr5x\" (UID: \"74468d4c-64a8-4b40-8d14-e9954caadc22\") " pod="ai-tenant-e2e-shared-b-w1-8d82d5/sub-test-model-8d82d5-kserve-d4df748cf-hqr5x"
Oct 30 13:27:56.394422 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:56.394364    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74468d4c-64a8-4b40-8d14-e9954caadc22-tmp-dir\") pod \"sub-test-model-8d82d5-kserve-d4df748cf-hqr5x\" (UID: \"74468d4c-64a8-4b40-8d14-e9954caadc22\") " pod="ai-tenant-e2e-shared-b-w1-8d82d5/sub-test-model-8d82d5-kserve-d4df748cf-hqr5x"
Oct 30 13:27:56.394722 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:56.394698    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74468d4c-64a8-4b40-8d14-e9954caadc22-home\") pod \"sub-test-model-8d82d5-kserve-d4df748cf-hqr5x\" (UID: \"74468d4c-64a8-4b40-8d14-e9954caadc22\") " pod="ai-tenant-e2e-shared-b-w1-8d82d5/sub-test-model-8d82d5-kserve-d4df748cf-hqr5x"
Oct 30 13:27:56.394828 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:56.394737    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74468d4c-64a8-4b40-8d14-e9954caadc22-model-cache\") pod \"sub-test-model-8d82d5-kserve-d4df748cf-hqr5x\" (UID: \"74468d4c-64a8-4b40-8d14-e9954caadc22\") " pod="ai-tenant-e2e-shared-b-w1-8d82d5/sub-test-model-8d82d5-kserve-d4df748cf-hqr5x"
Oct 30 13:27:56.394828 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:56.394782    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74468d4c-64a8-4b40-8d14-e9954caadc22-tmp-dir\") pod \"sub-test-model-8d82d5-kserve-d4df748cf-hqr5x\" (UID: \"74468d4c-64a8-4b40-8d14-e9954caadc22\") " pod="ai-tenant-e2e-shared-b-w1-8d82d5/sub-test-model-8d82d5-kserve-d4df748cf-hqr5x"
Oct 30 13:27:56.396859 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:56.396834    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74468d4c-64a8-4b40-8d14-e9954caadc22-dshm\") pod \"sub-test-model-8d82d5-kserve-d4df748cf-hqr5x\" (UID: \"74468d4c-64a8-4b40-8d14-e9954caadc22\") " pod="ai-tenant-e2e-shared-b-w1-8d82d5/sub-test-model-8d82d5-kserve-d4df748cf-hqr5x"
Oct 30 13:27:56.397018 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:56.397000    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74468d4c-64a8-4b40-8d14-e9954caadc22-tls-certs\") pod \"sub-test-model-8d82d5-kserve-d4df748cf-hqr5x\" (UID: \"74468d4c-64a8-4b40-8d14-e9954caadc22\") " pod="ai-tenant-e2e-shared-b-w1-8d82d5/sub-test-model-8d82d5-kserve-d4df748cf-hqr5x"
Oct 30 13:27:56.417283 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:56.417247    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4nnx\" (UniqueName: \"kubernetes.io/projected/74468d4c-64a8-4b40-8d14-e9954caadc22-kube-api-access-s4nnx\") pod \"sub-test-model-8d82d5-kserve-d4df748cf-hqr5x\" (UID: \"74468d4c-64a8-4b40-8d14-e9954caadc22\") " pod="ai-tenant-e2e-shared-b-w1-8d82d5/sub-test-model-8d82d5-kserve-d4df748cf-hqr5x"
Oct 30 13:27:56.587995 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:56.585979    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-8d82d5/sub-test-model-8d82d5-kserve-d4df748cf-hqr5x"
Oct 30 13:27:56.988001 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:56.987961    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-8d82d5/sub-test-model-8d82d5-kserve-d4df748cf-hqr5x"]
Oct 30 13:27:57.555156 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:57.555117    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-09e4c0-5454d677c9-bqbgj" podUID="79263302-817d-49f0-91a0-43281bff1ebd" containerName="payload-pre-processing" containerID="cri-o://a984316c0e942feb36207be82bcfc69f877d2610eb7a7ae31d1dcbd19d381e16" gracePeriod=25
Oct 30 13:27:57.716924 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:57.716884    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-8d82d5/sub-test-model-8d82d5-kserve-d4df748cf-hqr5x" event={"ID":"74468d4c-64a8-4b40-8d14-e9954caadc22","Type":"ContainerStarted","Data":"924455d80432ad889fadadcb01c34b75d2251f0c9e839c9b1cb1f953cc7865b0"}
Oct 30 13:27:57.717086 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:57.716940    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-8d82d5/sub-test-model-8d82d5-kserve-d4df748cf-hqr5x" event={"ID":"74468d4c-64a8-4b40-8d14-e9954caadc22","Type":"ContainerStarted","Data":"fda0c91a180d604c39b4f3d9b7ecb1b2cff9cec1f4eafc7339dc89aa56381852"}
Oct 30 13:27:57.762761 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:57.762712    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-8d82d5/sub-test-model-8d82d5-kserve-d4df748cf-hqr5x" podStartSLOduration=1.762699783 podStartE2EDuration="1.762699783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:27:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:27:57.758364699 +0000 UTC m=+2176.924729346" watchObservedRunningTime="2026-10-30 13:27:57.762699783 +0000 UTC m=+2176.929064431"
Oct 30 13:27:58.587034 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:27:58.587003    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-8d82d5/sub-test-model-8d82d5-kserve-d4df748cf-hqr5x"
Oct 30 13:28:00.329106 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:00.329069    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8467ed-76766c9c88-cbg5d"]
Oct 30 13:28:00.329568 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:00.329311    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8467ed-76766c9c88-cbg5d" podUID="fc880b1c-4413-4369-950b-a4bb3eb62568" containerName="maas-api" containerID="cri-o://346618995aa7919ab0f0c3800c4c664682b81aa127d076025b041df3533d7b04" gracePeriod=30
Oct 30 13:28:00.976334 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:00.928831    4364 generic.go:356] "Generic (PLEG): container finished" podID="fc880b1c-4413-4369-950b-a4bb3eb62568" containerID="346618995aa7919ab0f0c3800c4c664682b81aa127d076025b041df3533d7b04" exitCode=0
Oct 30 13:28:00.976334 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:00.929067    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8467ed-76766c9c88-cbg5d" event={"ID":"fc880b1c-4413-4369-950b-a4bb3eb62568","Type":"ContainerDied","Data":"346618995aa7919ab0f0c3800c4c664682b81aa127d076025b041df3533d7b04"}
Oct 30 13:28:01.022747 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:01.022719    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8467ed-76766c9c88-cbg5d"
Oct 30 13:28:01.149070 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:01.149037    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc880b1c-4413-4369-950b-a4bb3eb62568-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc880b1c-4413-4369-950b-a4bb3eb62568-maas-api-tls\") pod \"fc880b1c-4413-4369-950b-a4bb3eb62568\" (UID: \"fc880b1c-4413-4369-950b-a4bb3eb62568\") "
Oct 30 13:28:01.149282 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:01.149107    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc880b1c-4413-4369-950b-a4bb3eb62568-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc880b1c-4413-4369-950b-a4bb3eb62568-metrics-certs\") pod \"fc880b1c-4413-4369-950b-a4bb3eb62568\" (UID: \"fc880b1c-4413-4369-950b-a4bb3eb62568\") "
Oct 30 13:28:01.149356 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:01.149292    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc880b1c-4413-4369-950b-a4bb3eb62568-kube-api-access-mdwrt\" (UniqueName: \"kubernetes.io/projected/fc880b1c-4413-4369-950b-a4bb3eb62568-kube-api-access-mdwrt\") pod \"fc880b1c-4413-4369-950b-a4bb3eb62568\" (UID: \"fc880b1c-4413-4369-950b-a4bb3eb62568\") "
Oct 30 13:28:01.149416 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:01.149401    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc880b1c-4413-4369-950b-a4bb3eb62568-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc880b1c-4413-4369-950b-a4bb3eb62568-openshift-service-ca\") pod \"fc880b1c-4413-4369-950b-a4bb3eb62568\" (UID: \"fc880b1c-4413-4369-950b-a4bb3eb62568\") "
Oct 30 13:28:01.149803 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:01.149773    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc880b1c-4413-4369-950b-a4bb3eb62568-openshift-service-ca" pod "fc880b1c-4413-4369-950b-a4bb3eb62568" (UID: "fc880b1c-4413-4369-950b-a4bb3eb62568"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:28:01.150082 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:01.150066    4364 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc880b1c-4413-4369-950b-a4bb3eb62568-openshift-service-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:28:01.151685 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:01.151650    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc880b1c-4413-4369-950b-a4bb3eb62568-kube-api-access-mdwrt" pod "fc880b1c-4413-4369-950b-a4bb3eb62568" (UID: "fc880b1c-4413-4369-950b-a4bb3eb62568"). InnerVolumeSpecName "kube-api-access-mdwrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:28:01.151860 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:01.151835    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc880b1c-4413-4369-950b-a4bb3eb62568-maas-api-tls" pod "fc880b1c-4413-4369-950b-a4bb3eb62568" (UID: "fc880b1c-4413-4369-950b-a4bb3eb62568"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:28:01.151915 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:01.151899    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc880b1c-4413-4369-950b-a4bb3eb62568-metrics-certs" pod "fc880b1c-4413-4369-950b-a4bb3eb62568" (UID: "fc880b1c-4413-4369-950b-a4bb3eb62568"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:28:01.251485 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:01.251402    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc880b1c-4413-4369-950b-a4bb3eb62568-maas-api-tls\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:28:01.251485 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:01.251434    4364 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc880b1c-4413-4369-950b-a4bb3eb62568-metrics-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:28:01.251485 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:01.251445    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdwrt\" (UniqueName: \"kubernetes.io/projected/fc880b1c-4413-4369-950b-a4bb3eb62568-kube-api-access-mdwrt\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:28:01.938207 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:01.938177    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8467ed-76766c9c88-cbg5d"
Oct 30 13:28:01.938693 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:01.938222    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8467ed-76766c9c88-cbg5d" event={"ID":"fc880b1c-4413-4369-950b-a4bb3eb62568","Type":"ContainerDied","Data":"e69ee59df6e7dfe30e0f81594c17c23a57e1a4c6ee3211462559a746dac22499"}
Oct 30 13:28:01.938693 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:01.938289    4364 scope.go:122] "RemoveContainer" containerID="346618995aa7919ab0f0c3800c4c664682b81aa127d076025b041df3533d7b04"
Oct 30 13:28:01.991927 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:01.991896    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8467ed-76766c9c88-cbg5d"]
Oct 30 13:28:02.003454 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:02.003425    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8467ed-76766c9c88-cbg5d"]
Oct 30 13:28:02.040316 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:02.040284    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-09e4c0-5454d677c9-bqbgj" podUID="79263302-817d-49f0-91a0-43281bff1ebd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:28:02.972081 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:02.972044    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc880b1c-4413-4369-950b-a4bb3eb62568" path="/var/lib/kubelet/pods/fc880b1c-4413-4369-950b-a4bb3eb62568/volumes"
Oct 30 13:28:05.200461 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:05.200425    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-4651e8/test-model-4651e8-kserve-5c7fcb7f7-8n9zr"]
Oct 30 13:28:05.390781 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:05.380652    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d25db49-xrw2t"]
Oct 30 13:28:05.390781 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:05.381972    4364 cpu_manager.go:414] "containerMap: removing container" podUID="fc880b1c-4413-4369-950b-a4bb3eb62568" containerName="maas-api"
Oct 30 13:28:05.390781 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:05.381995    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc880b1c-4413-4369-950b-a4bb3eb62568" containerName="maas-api"
Oct 30 13:28:05.390781 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:05.382231    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc880b1c-4413-4369-950b-a4bb3eb62568" containerName="maas-api"
Oct 30 13:28:05.390781 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:05.388349    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d25db49-xrw2t"
Oct 30 13:28:05.413558 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:05.413526    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\""
Oct 30 13:28:05.446536 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:05.446486    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dda6f80-8485-49d9-99cf-82a602c79918-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d25db49-xrw2t\" (UID: \"4dda6f80-8485-49d9-99cf-82a602c79918\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d25db49-xrw2t"
Oct 30 13:28:05.446760 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:05.446697    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4j8z\" (UniqueName: \"kubernetes.io/projected/4dda6f80-8485-49d9-99cf-82a602c79918-kube-api-access-g4j8z\") pod \"maas-api-revoke-sub-1d25db49-xrw2t\" (UID: \"4dda6f80-8485-49d9-99cf-82a602c79918\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d25db49-xrw2t"
Oct 30 13:28:05.446831 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:05.446821    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dda6f80-8485-49d9-99cf-82a602c79918-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d25db49-xrw2t\" (UID: \"4dda6f80-8485-49d9-99cf-82a602c79918\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d25db49-xrw2t"
Oct 30 13:28:05.473874 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:05.473053    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d25db49-xrw2t"]
Oct 30 13:28:05.504503 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:05.503993    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:28:05.548040 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:05.548001    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dda6f80-8485-49d9-99cf-82a602c79918-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d25db49-xrw2t\" (UID: \"4dda6f80-8485-49d9-99cf-82a602c79918\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d25db49-xrw2t"
Oct 30 13:28:05.548191 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:05.548099    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dda6f80-8485-49d9-99cf-82a602c79918-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d25db49-xrw2t\" (UID: \"4dda6f80-8485-49d9-99cf-82a602c79918\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d25db49-xrw2t"
Oct 30 13:28:05.548247 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:05.548191    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4j8z\" (UniqueName: \"kubernetes.io/projected/4dda6f80-8485-49d9-99cf-82a602c79918-kube-api-access-g4j8z\") pod \"maas-api-revoke-sub-1d25db49-xrw2t\" (UID: \"4dda6f80-8485-49d9-99cf-82a602c79918\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d25db49-xrw2t"
Oct 30 13:28:05.548891 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:05.548867    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dda6f80-8485-49d9-99cf-82a602c79918-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d25db49-xrw2t\" (UID: \"4dda6f80-8485-49d9-99cf-82a602c79918\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d25db49-xrw2t"
Oct 30 13:28:05.550496 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:05.550473    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dda6f80-8485-49d9-99cf-82a602c79918-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d25db49-xrw2t\" (UID: \"4dda6f80-8485-49d9-99cf-82a602c79918\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d25db49-xrw2t"
Oct 30 13:28:05.571683 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:05.571659    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4j8z\" (UniqueName: \"kubernetes.io/projected/4dda6f80-8485-49d9-99cf-82a602c79918-kube-api-access-g4j8z\") pod \"maas-api-revoke-sub-1d25db49-xrw2t\" (UID: \"4dda6f80-8485-49d9-99cf-82a602c79918\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d25db49-xrw2t"
Oct 30 13:28:05.800342 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:05.800254    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d25db49-xrw2t"
Oct 30 13:28:06.141580 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:28:06.140151    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dda6f80_8485_49d9_99cf_82a602c79918.slice/crio-547ac684aee3e7c729c043571f176b50442f0cfca988e33c77d58756b87d0b55 WatchSource:0}: Error finding container 547ac684aee3e7c729c043571f176b50442f0cfca988e33c77d58756b87d0b55: Status 404 returned error can't find the container with id 547ac684aee3e7c729c043571f176b50442f0cfca988e33c77d58756b87d0b55
Oct 30 13:28:06.141580 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:06.141487    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d25db49-xrw2t"]
Oct 30 13:28:06.587828 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:06.587799    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-8d82d5/sub-test-model-8d82d5-kserve-d4df748cf-hqr5x"
Oct 30 13:28:06.601646 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:06.601621    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-8d82d5/sub-test-model-8d82d5-kserve-d4df748cf-hqr5x"
Oct 30 13:28:06.988466 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:06.988429    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d25db49-xrw2t" event={"ID":"4dda6f80-8485-49d9-99cf-82a602c79918","Type":"ContainerStarted","Data":"0358195fcc4dc2a31dd1255e0f67eb655f04587a506e63525372bb573686b7cb"}
Oct 30 13:28:06.988466 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:06.988481    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d25db49-xrw2t" event={"ID":"4dda6f80-8485-49d9-99cf-82a602c79918","Type":"ContainerStarted","Data":"547ac684aee3e7c729c043571f176b50442f0cfca988e33c77d58756b87d0b55"}
Oct 30 13:28:07.001174 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:07.001148    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-8d82d5/sub-test-model-8d82d5-kserve-d4df748cf-hqr5x"
Oct 30 13:28:07.056794 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:07.052455    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d25db49-xrw2t" podStartSLOduration=2.052435338 podStartE2EDuration="2.052435338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:28:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:28:07.037438989 +0000 UTC m=+2186.203803678" watchObservedRunningTime="2026-10-30 13:28:07.052435338 +0000 UTC m=+2186.218799988"
Oct 30 13:28:08.000897 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:08.000849    4364 generic.go:356] "Generic (PLEG): container finished" podID="4dda6f80-8485-49d9-99cf-82a602c79918" containerID="0358195fcc4dc2a31dd1255e0f67eb655f04587a506e63525372bb573686b7cb" exitCode=0
Oct 30 13:28:08.001354 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:08.000982    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d25db49-xrw2t" event={"ID":"4dda6f80-8485-49d9-99cf-82a602c79918","Type":"ContainerDied","Data":"0358195fcc4dc2a31dd1255e0f67eb655f04587a506e63525372bb573686b7cb"}
Oct 30 13:28:08.738399 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:08.738362    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn"]
Oct 30 13:28:08.738773 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:08.738720    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn" podUID="ebad4af8-38a2-4da7-a1ec-b56ef4c48a79" containerName="istio-proxy" containerID="cri-o://213ce31fde9bfe9810183291dcd689c66acc715d9865f4382393cea5102b9f28" gracePeriod=30
Oct 30 13:28:09.226666 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:09.226401    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d25db49-xrw2t"
Oct 30 13:28:09.345984 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:09.345903    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4dda6f80-8485-49d9-99cf-82a602c79918-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dda6f80-8485-49d9-99cf-82a602c79918-maas-api-xks-ca\") pod \"4dda6f80-8485-49d9-99cf-82a602c79918\" (UID: \"4dda6f80-8485-49d9-99cf-82a602c79918\") "
Oct 30 13:28:09.346207 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:09.346190    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4dda6f80-8485-49d9-99cf-82a602c79918-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dda6f80-8485-49d9-99cf-82a602c79918-maas-api-openshift-ca\") pod \"4dda6f80-8485-49d9-99cf-82a602c79918\" (UID: \"4dda6f80-8485-49d9-99cf-82a602c79918\") "
Oct 30 13:28:09.346303 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:09.346224    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dda6f80-8485-49d9-99cf-82a602c79918-kube-api-access-g4j8z\" (UniqueName: \"kubernetes.io/projected/4dda6f80-8485-49d9-99cf-82a602c79918-kube-api-access-g4j8z\") pod \"4dda6f80-8485-49d9-99cf-82a602c79918\" (UID: \"4dda6f80-8485-49d9-99cf-82a602c79918\") "
Oct 30 13:28:09.346609 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:09.346569    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dda6f80-8485-49d9-99cf-82a602c79918-maas-api-openshift-ca" pod "4dda6f80-8485-49d9-99cf-82a602c79918" (UID: "4dda6f80-8485-49d9-99cf-82a602c79918"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:28:09.347027 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:09.347009    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dda6f80-8485-49d9-99cf-82a602c79918-maas-api-openshift-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:28:09.348552 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:09.348525    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dda6f80-8485-49d9-99cf-82a602c79918-maas-api-xks-ca" pod "4dda6f80-8485-49d9-99cf-82a602c79918" (UID: "4dda6f80-8485-49d9-99cf-82a602c79918"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:28:09.350792 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:09.350765    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dda6f80-8485-49d9-99cf-82a602c79918-kube-api-access-g4j8z" pod "4dda6f80-8485-49d9-99cf-82a602c79918" (UID: "4dda6f80-8485-49d9-99cf-82a602c79918"). InnerVolumeSpecName "kube-api-access-g4j8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:28:09.448670 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:09.448637    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dda6f80-8485-49d9-99cf-82a602c79918-maas-api-xks-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:28:09.448670 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:09.448665    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4j8z\" (UniqueName: \"kubernetes.io/projected/4dda6f80-8485-49d9-99cf-82a602c79918-kube-api-access-g4j8z\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:28:10.050159 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:10.050128    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d25db49-xrw2t"
Oct 30 13:28:10.050345 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:10.050161    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d25db49-xrw2t" event={"ID":"4dda6f80-8485-49d9-99cf-82a602c79918","Type":"ContainerDied","Data":"547ac684aee3e7c729c043571f176b50442f0cfca988e33c77d58756b87d0b55"}
Oct 30 13:28:10.050345 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:10.050208    4364 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="547ac684aee3e7c729c043571f176b50442f0cfca988e33c77d58756b87d0b55"
Oct 30 13:28:11.506638 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:28:11.506587    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:28:12.044464 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:12.044421    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-09e4c0-5454d677c9-bqbgj" podUID="79263302-817d-49f0-91a0-43281bff1ebd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:28:12.405299 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:12.405260    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c47903-546d9c8468-kwdcc"]
Oct 30 13:28:12.406203 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:12.406177    4364 cpu_manager.go:414] "containerMap: removing container" podUID="4dda6f80-8485-49d9-99cf-82a602c79918" containerName="revoke-keys"
Oct 30 13:28:12.406203 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:12.406198    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dda6f80-8485-49d9-99cf-82a602c79918" containerName="revoke-keys"
Oct 30 13:28:12.406700 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:12.406343    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dda6f80-8485-49d9-99cf-82a602c79918" containerName="revoke-keys"
Oct 30 13:28:12.410405 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:12.410381    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c47903-546d9c8468-kwdcc"
Oct 30 13:28:12.435912 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:12.434867    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c47903-546d9c8468-kwdcc"]
Oct 30 13:28:12.444194 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:12.440104    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-c47903-serving-cert\""
Oct 30 13:28:12.506163 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:12.506122    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-757mt\" (UniqueName: \"kubernetes.io/projected/93fc7bda-b92c-41ce-b567-658a571bc5ba-kube-api-access-757mt\") pod \"maas-api-e2e-shared-a-w3-c47903-546d9c8468-kwdcc\" (UID: \"93fc7bda-b92c-41ce-b567-658a571bc5ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c47903-546d9c8468-kwdcc"
Oct 30 13:28:12.506360 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:12.506192    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93fc7bda-b92c-41ce-b567-658a571bc5ba-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-c47903-546d9c8468-kwdcc\" (UID: \"93fc7bda-b92c-41ce-b567-658a571bc5ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c47903-546d9c8468-kwdcc"
Oct 30 13:28:12.506360 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:12.506273    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93fc7bda-b92c-41ce-b567-658a571bc5ba-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-c47903-546d9c8468-kwdcc\" (UID: \"93fc7bda-b92c-41ce-b567-658a571bc5ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c47903-546d9c8468-kwdcc"
Oct 30 13:28:12.506488 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:12.506446    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93fc7bda-b92c-41ce-b567-658a571bc5ba-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-c47903-546d9c8468-kwdcc\" (UID: \"93fc7bda-b92c-41ce-b567-658a571bc5ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c47903-546d9c8468-kwdcc"
Oct 30 13:28:12.613583 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:12.609998    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-757mt\" (UniqueName: \"kubernetes.io/projected/93fc7bda-b92c-41ce-b567-658a571bc5ba-kube-api-access-757mt\") pod \"maas-api-e2e-shared-a-w3-c47903-546d9c8468-kwdcc\" (UID: \"93fc7bda-b92c-41ce-b567-658a571bc5ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c47903-546d9c8468-kwdcc"
Oct 30 13:28:12.613583 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:12.610064    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93fc7bda-b92c-41ce-b567-658a571bc5ba-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-c47903-546d9c8468-kwdcc\" (UID: \"93fc7bda-b92c-41ce-b567-658a571bc5ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c47903-546d9c8468-kwdcc"
Oct 30 13:28:12.613583 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:12.610139    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93fc7bda-b92c-41ce-b567-658a571bc5ba-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-c47903-546d9c8468-kwdcc\" (UID: \"93fc7bda-b92c-41ce-b567-658a571bc5ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c47903-546d9c8468-kwdcc"
Oct 30 13:28:12.613583 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:12.610272    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93fc7bda-b92c-41ce-b567-658a571bc5ba-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-c47903-546d9c8468-kwdcc\" (UID: \"93fc7bda-b92c-41ce-b567-658a571bc5ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c47903-546d9c8468-kwdcc"
Oct 30 13:28:12.613583 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:12.611430    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93fc7bda-b92c-41ce-b567-658a571bc5ba-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-c47903-546d9c8468-kwdcc\" (UID: \"93fc7bda-b92c-41ce-b567-658a571bc5ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c47903-546d9c8468-kwdcc"
Oct 30 13:28:12.623846 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:12.623810    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93fc7bda-b92c-41ce-b567-658a571bc5ba-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-c47903-546d9c8468-kwdcc\" (UID: \"93fc7bda-b92c-41ce-b567-658a571bc5ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c47903-546d9c8468-kwdcc"
Oct 30 13:28:12.649442 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:12.649405    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93fc7bda-b92c-41ce-b567-658a571bc5ba-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-c47903-546d9c8468-kwdcc\" (UID: \"93fc7bda-b92c-41ce-b567-658a571bc5ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c47903-546d9c8468-kwdcc"
Oct 30 13:28:12.660538 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:12.660452    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-757mt\" (UniqueName: \"kubernetes.io/projected/93fc7bda-b92c-41ce-b567-658a571bc5ba-kube-api-access-757mt\") pod \"maas-api-e2e-shared-a-w3-c47903-546d9c8468-kwdcc\" (UID: \"93fc7bda-b92c-41ce-b567-658a571bc5ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c47903-546d9c8468-kwdcc"
Oct 30 13:28:12.732361 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:12.732326    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c47903-546d9c8468-kwdcc"
Oct 30 13:28:13.051843 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:13.051809    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c47903-546d9c8468-kwdcc"]
Oct 30 13:28:13.052286 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:28:13.052263    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93fc7bda_b92c_41ce_b567_658a571bc5ba.slice/crio-bbd82eb1545fe3c2dff2d57f334da1384b557c52a02fd5f44d4b1e3a9a986ebd WatchSource:0}: Error finding container bbd82eb1545fe3c2dff2d57f334da1384b557c52a02fd5f44d4b1e3a9a986ebd: Status 404 returned error can't find the container with id bbd82eb1545fe3c2dff2d57f334da1384b557c52a02fd5f44d4b1e3a9a986ebd
Oct 30 13:28:13.053765 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:13.053743    4364 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 13:28:13.078829 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:13.078800    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c47903-546d9c8468-kwdcc" event={"ID":"93fc7bda-b92c-41ce-b567-658a571bc5ba","Type":"ContainerStarted","Data":"bbd82eb1545fe3c2dff2d57f334da1384b557c52a02fd5f44d4b1e3a9a986ebd"}
Oct 30 13:28:14.096163 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:14.096136    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn"
Oct 30 13:28:14.097820 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:14.097793    4364 generic.go:356] "Generic (PLEG): container finished" podID="ebad4af8-38a2-4da7-a1ec-b56ef4c48a79" containerID="213ce31fde9bfe9810183291dcd689c66acc715d9865f4382393cea5102b9f28" exitCode=0
Oct 30 13:28:14.097966 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:14.097865    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn" event={"ID":"ebad4af8-38a2-4da7-a1ec-b56ef4c48a79","Type":"ContainerDied","Data":"213ce31fde9bfe9810183291dcd689c66acc715d9865f4382393cea5102b9f28"}
Oct 30 13:28:14.097966 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:14.097921    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn" event={"ID":"ebad4af8-38a2-4da7-a1ec-b56ef4c48a79","Type":"ContainerDied","Data":"c53c3f0f3145151e221e038fe449a4b69378a475f95ae2cb58495d24bc65959c"}
Oct 30 13:28:14.097966 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:14.097944    4364 scope.go:122] "RemoveContainer" containerID="213ce31fde9bfe9810183291dcd689c66acc715d9865f4382393cea5102b9f28"
Oct 30 13:28:14.099404 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:14.099379    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c47903-546d9c8468-kwdcc" event={"ID":"93fc7bda-b92c-41ce-b567-658a571bc5ba","Type":"ContainerStarted","Data":"abbd5d9520c8fd5aa9b863c0999bbd36e684215de3a70273421b525e6ce9c7e9"}
Oct 30 13:28:14.099521 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:14.099513    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c47903-546d9c8468-kwdcc"
Oct 30 13:28:14.109485 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:14.109464    4364 scope.go:122] "RemoveContainer" containerID="b4e8057dbfc71e745c1ec7b36ba257b389a44a362756b1272c6f5253e3d046f1"
Oct 30 13:28:14.119801 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:14.119782    4364 scope.go:122] "RemoveContainer" containerID="213ce31fde9bfe9810183291dcd689c66acc715d9865f4382393cea5102b9f28"
Oct 30 13:28:14.120056 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:28:14.120040    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"213ce31fde9bfe9810183291dcd689c66acc715d9865f4382393cea5102b9f28\": container with ID starting with 213ce31fde9bfe9810183291dcd689c66acc715d9865f4382393cea5102b9f28 not found: ID does not exist" containerID="213ce31fde9bfe9810183291dcd689c66acc715d9865f4382393cea5102b9f28"
Oct 30 13:28:14.120153 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:14.120062    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"213ce31fde9bfe9810183291dcd689c66acc715d9865f4382393cea5102b9f28"} err="failed to get container status \"213ce31fde9bfe9810183291dcd689c66acc715d9865f4382393cea5102b9f28\": rpc error: code = NotFound desc = could not find container \"213ce31fde9bfe9810183291dcd689c66acc715d9865f4382393cea5102b9f28\": container with ID starting with 213ce31fde9bfe9810183291dcd689c66acc715d9865f4382393cea5102b9f28 not found: ID does not exist"
Oct 30 13:28:14.120153 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:14.120078    4364 scope.go:122] "RemoveContainer" containerID="b4e8057dbfc71e745c1ec7b36ba257b389a44a362756b1272c6f5253e3d046f1"
Oct 30 13:28:14.120288 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:28:14.120270    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4e8057dbfc71e745c1ec7b36ba257b389a44a362756b1272c6f5253e3d046f1\": container with ID starting with b4e8057dbfc71e745c1ec7b36ba257b389a44a362756b1272c6f5253e3d046f1 not found: ID does not exist" containerID="b4e8057dbfc71e745c1ec7b36ba257b389a44a362756b1272c6f5253e3d046f1"
Oct 30 13:28:14.120330 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:14.120292    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4e8057dbfc71e745c1ec7b36ba257b389a44a362756b1272c6f5253e3d046f1"} err="failed to get container status \"b4e8057dbfc71e745c1ec7b36ba257b389a44a362756b1272c6f5253e3d046f1\": rpc error: code = NotFound desc = could not find container \"b4e8057dbfc71e745c1ec7b36ba257b389a44a362756b1272c6f5253e3d046f1\": container with ID starting with b4e8057dbfc71e745c1ec7b36ba257b389a44a362756b1272c6f5253e3d046f1 not found: ID does not exist"
Oct 30 13:28:14.196192 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:14.196135    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c47903-546d9c8468-kwdcc" podStartSLOduration=1.854309954 podStartE2EDuration="2.196116014s" totalImagesPullingTime="341.80606ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:28:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:28:14.194946364 +0000 UTC m=+2193.361311013" watchObservedRunningTime="2026-10-30 13:28:14.196116014 +0000 UTC m=+2193.362480662"
Oct 30 13:28:14.251949 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:14.251916    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebad4af8-38a2-4da7-a1ec-b56ef4c48a79-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebad4af8-38a2-4da7-a1ec-b56ef4c48a79-istiod-ca-cert\") pod \"ebad4af8-38a2-4da7-a1ec-b56ef4c48a79\" (UID: \"ebad4af8-38a2-4da7-a1ec-b56ef4c48a79\") "
Oct 30 13:28:14.252122 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:14.251973    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebad4af8-38a2-4da7-a1ec-b56ef4c48a79-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebad4af8-38a2-4da7-a1ec-b56ef4c48a79-credential-socket\") pod \"ebad4af8-38a2-4da7-a1ec-b56ef4c48a79\" (UID: \"ebad4af8-38a2-4da7-a1ec-b56ef4c48a79\") "
Oct 30 13:28:14.252122 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:14.251991    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebad4af8-38a2-4da7-a1ec-b56ef4c48a79-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebad4af8-38a2-4da7-a1ec-b56ef4c48a79-workload-certs\") pod \"ebad4af8-38a2-4da7-a1ec-b56ef4c48a79\" (UID: \"ebad4af8-38a2-4da7-a1ec-b56ef4c48a79\") "
Oct 30 13:28:14.252122 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:14.252009    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ebad4af8-38a2-4da7-a1ec-b56ef4c48a79-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebad4af8-38a2-4da7-a1ec-b56ef4c48a79-istio-podinfo\") pod \"ebad4af8-38a2-4da7-a1ec-b56ef4c48a79\" (UID: \"ebad4af8-38a2-4da7-a1ec-b56ef4c48a79\") "
Oct 30 13:28:14.252122 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:14.252038    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebad4af8-38a2-4da7-a1ec-b56ef4c48a79-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebad4af8-38a2-4da7-a1ec-b56ef4c48a79-istio-envoy\") pod \"ebad4af8-38a2-4da7-a1ec-b56ef4c48a79\" (UID: \"ebad4af8-38a2-4da7-a1ec-b56ef4c48a79\") "
Oct 30 13:28:14.252122 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:14.252097    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebad4af8-38a2-4da7-a1ec-b56ef4c48a79-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebad4af8-38a2-4da7-a1ec-b56ef4c48a79-istio-data\") pod \"ebad4af8-38a2-4da7-a1ec-b56ef4c48a79\" (UID: \"ebad4af8-38a2-4da7-a1ec-b56ef4c48a79\") "
Oct 30 13:28:14.252412 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:14.252153    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebad4af8-38a2-4da7-a1ec-b56ef4c48a79-kube-api-access-9cvgc\" (UniqueName: \"kubernetes.io/projected/ebad4af8-38a2-4da7-a1ec-b56ef4c48a79-kube-api-access-9cvgc\") pod \"ebad4af8-38a2-4da7-a1ec-b56ef4c48a79\" (UID: \"ebad4af8-38a2-4da7-a1ec-b56ef4c48a79\") "
Oct 30 13:28:14.252412 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:14.252307    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebad4af8-38a2-4da7-a1ec-b56ef4c48a79-istio-token\" (UniqueName: \"kubernetes.io/projected/ebad4af8-38a2-4da7-a1ec-b56ef4c48a79-istio-token\") pod \"ebad4af8-38a2-4da7-a1ec-b56ef4c48a79\" (UID: \"ebad4af8-38a2-4da7-a1ec-b56ef4c48a79\") "
Oct 30 13:28:14.252540 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:14.252403    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebad4af8-38a2-4da7-a1ec-b56ef4c48a79-workload-certs" pod "ebad4af8-38a2-4da7-a1ec-b56ef4c48a79" (UID: "ebad4af8-38a2-4da7-a1ec-b56ef4c48a79"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:28:14.252540 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:14.252428    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebad4af8-38a2-4da7-a1ec-b56ef4c48a79-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebad4af8-38a2-4da7-a1ec-b56ef4c48a79-workload-socket\") pod \"ebad4af8-38a2-4da7-a1ec-b56ef4c48a79\" (UID: \"ebad4af8-38a2-4da7-a1ec-b56ef4c48a79\") "
Oct 30 13:28:14.252540 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:14.252439    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebad4af8-38a2-4da7-a1ec-b56ef4c48a79-istiod-ca-cert" pod "ebad4af8-38a2-4da7-a1ec-b56ef4c48a79" (UID: "ebad4af8-38a2-4da7-a1ec-b56ef4c48a79"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:28:14.252761 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:14.252575    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebad4af8-38a2-4da7-a1ec-b56ef4c48a79-istio-data" pod "ebad4af8-38a2-4da7-a1ec-b56ef4c48a79" (UID: "ebad4af8-38a2-4da7-a1ec-b56ef4c48a79"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:28:14.252761 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:14.252738    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebad4af8-38a2-4da7-a1ec-b56ef4c48a79-credential-socket" pod "ebad4af8-38a2-4da7-a1ec-b56ef4c48a79" (UID: "ebad4af8-38a2-4da7-a1ec-b56ef4c48a79"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:28:14.252887 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:14.252754    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebad4af8-38a2-4da7-a1ec-b56ef4c48a79-workload-socket" pod "ebad4af8-38a2-4da7-a1ec-b56ef4c48a79" (UID: "ebad4af8-38a2-4da7-a1ec-b56ef4c48a79"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:28:14.253403 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:14.253377    4364 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebad4af8-38a2-4da7-a1ec-b56ef4c48a79-istiod-ca-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:28:14.253516 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:14.253407    4364 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebad4af8-38a2-4da7-a1ec-b56ef4c48a79-credential-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:28:14.253516 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:14.253422    4364 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebad4af8-38a2-4da7-a1ec-b56ef4c48a79-workload-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:28:14.253516 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:14.253436    4364 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebad4af8-38a2-4da7-a1ec-b56ef4c48a79-istio-data\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:28:14.253516 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:14.253452    4364 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebad4af8-38a2-4da7-a1ec-b56ef4c48a79-workload-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:28:14.254434 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:14.254410    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebad4af8-38a2-4da7-a1ec-b56ef4c48a79-istio-envoy" pod "ebad4af8-38a2-4da7-a1ec-b56ef4c48a79" (UID: "ebad4af8-38a2-4da7-a1ec-b56ef4c48a79"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:28:14.254579 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:14.254564    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ebad4af8-38a2-4da7-a1ec-b56ef4c48a79-istio-podinfo" pod "ebad4af8-38a2-4da7-a1ec-b56ef4c48a79" (UID: "ebad4af8-38a2-4da7-a1ec-b56ef4c48a79"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:28:14.254666 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:14.254575    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebad4af8-38a2-4da7-a1ec-b56ef4c48a79-kube-api-access-9cvgc" pod "ebad4af8-38a2-4da7-a1ec-b56ef4c48a79" (UID: "ebad4af8-38a2-4da7-a1ec-b56ef4c48a79"). InnerVolumeSpecName "kube-api-access-9cvgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:28:14.254666 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:14.254639    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebad4af8-38a2-4da7-a1ec-b56ef4c48a79-istio-token" pod "ebad4af8-38a2-4da7-a1ec-b56ef4c48a79" (UID: "ebad4af8-38a2-4da7-a1ec-b56ef4c48a79"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:28:14.361671 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:14.360532    4364 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebad4af8-38a2-4da7-a1ec-b56ef4c48a79-istio-token\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:28:14.361671 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:14.360567    4364 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebad4af8-38a2-4da7-a1ec-b56ef4c48a79-istio-podinfo\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:28:14.361671 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:14.360582    4364 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebad4af8-38a2-4da7-a1ec-b56ef4c48a79-istio-envoy\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:28:14.361671 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:14.360612    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cvgc\" (UniqueName: \"kubernetes.io/projected/ebad4af8-38a2-4da7-a1ec-b56ef4c48a79-kube-api-access-9cvgc\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:28:15.108425 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:15.108392    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn"
Oct 30 13:28:15.175864 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:15.175830    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn"]
Oct 30 13:28:15.185425 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:15.185396    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-8467ed-openshift-default-7bd8f54d77-bqfbn"]
Oct 30 13:28:16.972719 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:16.972677    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebad4af8-38a2-4da7-a1ec-b56ef4c48a79" path="/var/lib/kubelet/pods/ebad4af8-38a2-4da7-a1ec-b56ef4c48a79/volumes"
Oct 30 13:28:17.851820 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:17.851791    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-09e4c0-5454d677c9-bqbgj"
Oct 30 13:28:18.018276 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:18.018239    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79263302-817d-49f0-91a0-43281bff1ebd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79263302-817d-49f0-91a0-43281bff1ebd-plugins-config-volume\") pod \"79263302-817d-49f0-91a0-43281bff1ebd\" (UID: \"79263302-817d-49f0-91a0-43281bff1ebd\") "
Oct 30 13:28:18.018689 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:18.018396    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79263302-817d-49f0-91a0-43281bff1ebd-kube-api-access-npf2g\" (UniqueName: \"kubernetes.io/projected/79263302-817d-49f0-91a0-43281bff1ebd-kube-api-access-npf2g\") pod \"79263302-817d-49f0-91a0-43281bff1ebd\" (UID: \"79263302-817d-49f0-91a0-43281bff1ebd\") "
Oct 30 13:28:18.018689 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:18.018632    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79263302-817d-49f0-91a0-43281bff1ebd-plugins-config-volume" pod "79263302-817d-49f0-91a0-43281bff1ebd" (UID: "79263302-817d-49f0-91a0-43281bff1ebd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:28:18.019139 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:18.019114    4364 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79263302-817d-49f0-91a0-43281bff1ebd-plugins-config-volume\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:28:18.020608 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:18.020577    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79263302-817d-49f0-91a0-43281bff1ebd-kube-api-access-npf2g" pod "79263302-817d-49f0-91a0-43281bff1ebd" (UID: "79263302-817d-49f0-91a0-43281bff1ebd"). InnerVolumeSpecName "kube-api-access-npf2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:28:18.121201 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:18.121173    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npf2g\" (UniqueName: \"kubernetes.io/projected/79263302-817d-49f0-91a0-43281bff1ebd-kube-api-access-npf2g\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:28:18.134200 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:18.134166    4364 generic.go:356] "Generic (PLEG): container finished" podID="79263302-817d-49f0-91a0-43281bff1ebd" containerID="a984316c0e942feb36207be82bcfc69f877d2610eb7a7ae31d1dcbd19d381e16" exitCode=0
Oct 30 13:28:18.134379 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:18.134216    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-09e4c0-5454d677c9-bqbgj"
Oct 30 13:28:18.134379 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:18.134259    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-09e4c0-5454d677c9-bqbgj" event={"ID":"79263302-817d-49f0-91a0-43281bff1ebd","Type":"ContainerDied","Data":"a984316c0e942feb36207be82bcfc69f877d2610eb7a7ae31d1dcbd19d381e16"}
Oct 30 13:28:18.134379 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:18.134324    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-09e4c0-5454d677c9-bqbgj" event={"ID":"79263302-817d-49f0-91a0-43281bff1ebd","Type":"ContainerDied","Data":"ba63d80bc5b3b524d38e796b3aeaa1c95ab55e8ac4054268e405c9351336903b"}
Oct 30 13:28:18.134379 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:18.134354    4364 scope.go:122] "RemoveContainer" containerID="a984316c0e942feb36207be82bcfc69f877d2610eb7a7ae31d1dcbd19d381e16"
Oct 30 13:28:18.145971 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:18.145954    4364 scope.go:122] "RemoveContainer" containerID="a984316c0e942feb36207be82bcfc69f877d2610eb7a7ae31d1dcbd19d381e16"
Oct 30 13:28:18.146227 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:28:18.146209    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a984316c0e942feb36207be82bcfc69f877d2610eb7a7ae31d1dcbd19d381e16\": container with ID starting with a984316c0e942feb36207be82bcfc69f877d2610eb7a7ae31d1dcbd19d381e16 not found: ID does not exist" containerID="a984316c0e942feb36207be82bcfc69f877d2610eb7a7ae31d1dcbd19d381e16"
Oct 30 13:28:18.146299 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:18.146237    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a984316c0e942feb36207be82bcfc69f877d2610eb7a7ae31d1dcbd19d381e16"} err="failed to get container status \"a984316c0e942feb36207be82bcfc69f877d2610eb7a7ae31d1dcbd19d381e16\": rpc error: code = NotFound desc = could not find container \"a984316c0e942feb36207be82bcfc69f877d2610eb7a7ae31d1dcbd19d381e16\": container with ID starting with a984316c0e942feb36207be82bcfc69f877d2610eb7a7ae31d1dcbd19d381e16 not found: ID does not exist"
Oct 30 13:28:18.187887 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:18.187858    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-09e4c0-5454d677c9-bqbgj"]
Oct 30 13:28:18.192453 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:18.192428    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-09e4c0-5454d677c9-bqbgj"]
Oct 30 13:28:18.972826 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:18.972792    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79263302-817d-49f0-91a0-43281bff1ebd" path="/var/lib/kubelet/pods/79263302-817d-49f0-91a0-43281bff1ebd/volumes"
Oct 30 13:28:20.114840 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:20.114808    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c47903-546d9c8468-kwdcc"
Oct 30 13:28:20.224542 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:20.224469    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-4651e8/test-model-4651e8-kserve-5c7fcb7f7-8n9zr" podUID="ff5899b6-dd04-481f-bf5d-d4ce4e1f981d" containerName="main" containerID="cri-o://25c4b6cb4be2601706f0fd9eb0d6ee888769d027159f33f5fcf4e97628b6b4a3" gracePeriod=45
Oct 30 13:28:20.565736 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:20.565707    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4651e8/test-model-4651e8-kserve-5c7fcb7f7-8n9zr"
Oct 30 13:28:20.670443 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:20.670357    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff5899b6-dd04-481f-bf5d-d4ce4e1f981d-tls-certs\" (UniqueName: \"kubernetes.io/secret/ff5899b6-dd04-481f-bf5d-d4ce4e1f981d-tls-certs\") pod \"ff5899b6-dd04-481f-bf5d-d4ce4e1f981d\" (UID: \"ff5899b6-dd04-481f-bf5d-d4ce4e1f981d\") "
Oct 30 13:28:20.670443 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:20.670396    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff5899b6-dd04-481f-bf5d-d4ce4e1f981d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff5899b6-dd04-481f-bf5d-d4ce4e1f981d-tmp-dir\") pod \"ff5899b6-dd04-481f-bf5d-d4ce4e1f981d\" (UID: \"ff5899b6-dd04-481f-bf5d-d4ce4e1f981d\") "
Oct 30 13:28:20.670714 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:20.670528    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff5899b6-dd04-481f-bf5d-d4ce4e1f981d-home\" (UniqueName: \"kubernetes.io/empty-dir/ff5899b6-dd04-481f-bf5d-d4ce4e1f981d-home\") pod \"ff5899b6-dd04-481f-bf5d-d4ce4e1f981d\" (UID: \"ff5899b6-dd04-481f-bf5d-d4ce4e1f981d\") "
Oct 30 13:28:20.670714 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:20.670637    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff5899b6-dd04-481f-bf5d-d4ce4e1f981d-kube-api-access-gbfzj\" (UniqueName: \"kubernetes.io/projected/ff5899b6-dd04-481f-bf5d-d4ce4e1f981d-kube-api-access-gbfzj\") pod \"ff5899b6-dd04-481f-bf5d-d4ce4e1f981d\" (UID: \"ff5899b6-dd04-481f-bf5d-d4ce4e1f981d\") "
Oct 30 13:28:20.670714 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:20.670673    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff5899b6-dd04-481f-bf5d-d4ce4e1f981d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff5899b6-dd04-481f-bf5d-d4ce4e1f981d-model-cache\") pod \"ff5899b6-dd04-481f-bf5d-d4ce4e1f981d\" (UID: \"ff5899b6-dd04-481f-bf5d-d4ce4e1f981d\") "
Oct 30 13:28:20.670714 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:20.670698    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff5899b6-dd04-481f-bf5d-d4ce4e1f981d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff5899b6-dd04-481f-bf5d-d4ce4e1f981d-dshm\") pod \"ff5899b6-dd04-481f-bf5d-d4ce4e1f981d\" (UID: \"ff5899b6-dd04-481f-bf5d-d4ce4e1f981d\") "
Oct 30 13:28:20.670902 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:20.670741    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff5899b6-dd04-481f-bf5d-d4ce4e1f981d-tmp-dir" pod "ff5899b6-dd04-481f-bf5d-d4ce4e1f981d" (UID: "ff5899b6-dd04-481f-bf5d-d4ce4e1f981d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:28:20.670902 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:20.670812    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff5899b6-dd04-481f-bf5d-d4ce4e1f981d-home" pod "ff5899b6-dd04-481f-bf5d-d4ce4e1f981d" (UID: "ff5899b6-dd04-481f-bf5d-d4ce4e1f981d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:28:20.670971 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:20.670920    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff5899b6-dd04-481f-bf5d-d4ce4e1f981d-model-cache" pod "ff5899b6-dd04-481f-bf5d-d4ce4e1f981d" (UID: "ff5899b6-dd04-481f-bf5d-d4ce4e1f981d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:28:20.671223 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:20.671200    4364 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff5899b6-dd04-481f-bf5d-d4ce4e1f981d-tmp-dir\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:28:20.671223 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:20.671223    4364 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff5899b6-dd04-481f-bf5d-d4ce4e1f981d-home\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:28:20.671363 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:20.671232    4364 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff5899b6-dd04-481f-bf5d-d4ce4e1f981d-model-cache\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:28:20.672783 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:20.672757    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff5899b6-dd04-481f-bf5d-d4ce4e1f981d-tls-certs" pod "ff5899b6-dd04-481f-bf5d-d4ce4e1f981d" (UID: "ff5899b6-dd04-481f-bf5d-d4ce4e1f981d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:28:20.672888 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:20.672817    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff5899b6-dd04-481f-bf5d-d4ce4e1f981d-dshm" pod "ff5899b6-dd04-481f-bf5d-d4ce4e1f981d" (UID: "ff5899b6-dd04-481f-bf5d-d4ce4e1f981d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:28:20.672888 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:20.672831    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff5899b6-dd04-481f-bf5d-d4ce4e1f981d-kube-api-access-gbfzj" pod "ff5899b6-dd04-481f-bf5d-d4ce4e1f981d" (UID: "ff5899b6-dd04-481f-bf5d-d4ce4e1f981d"). InnerVolumeSpecName "kube-api-access-gbfzj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:28:20.772787 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:20.772755    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbfzj\" (UniqueName: \"kubernetes.io/projected/ff5899b6-dd04-481f-bf5d-d4ce4e1f981d-kube-api-access-gbfzj\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:28:20.772787 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:20.772781    4364 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff5899b6-dd04-481f-bf5d-d4ce4e1f981d-dshm\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:28:20.772787 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:20.772791    4364 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff5899b6-dd04-481f-bf5d-d4ce4e1f981d-tls-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:28:21.162271 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:21.162231    4364 generic.go:356] "Generic (PLEG): container finished" podID="ff5899b6-dd04-481f-bf5d-d4ce4e1f981d" containerID="25c4b6cb4be2601706f0fd9eb0d6ee888769d027159f33f5fcf4e97628b6b4a3" exitCode=0
Oct 30 13:28:21.162754 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:21.162285    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4651e8/test-model-4651e8-kserve-5c7fcb7f7-8n9zr"
Oct 30 13:28:21.162754 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:21.162310    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4651e8/test-model-4651e8-kserve-5c7fcb7f7-8n9zr" event={"ID":"ff5899b6-dd04-481f-bf5d-d4ce4e1f981d","Type":"ContainerDied","Data":"25c4b6cb4be2601706f0fd9eb0d6ee888769d027159f33f5fcf4e97628b6b4a3"}
Oct 30 13:28:21.162754 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:21.162354    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4651e8/test-model-4651e8-kserve-5c7fcb7f7-8n9zr" event={"ID":"ff5899b6-dd04-481f-bf5d-d4ce4e1f981d","Type":"ContainerDied","Data":"0bc5b674f267270180b595dcd39c0dc868879e4c4eb7296d66df703e44ee0230"}
Oct 30 13:28:21.162754 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:21.162372    4364 scope.go:122] "RemoveContainer" containerID="25c4b6cb4be2601706f0fd9eb0d6ee888769d027159f33f5fcf4e97628b6b4a3"
Oct 30 13:28:21.173531 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:21.173500    4364 scope.go:122] "RemoveContainer" containerID="25c4b6cb4be2601706f0fd9eb0d6ee888769d027159f33f5fcf4e97628b6b4a3"
Oct 30 13:28:21.173840 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:28:21.173820    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25c4b6cb4be2601706f0fd9eb0d6ee888769d027159f33f5fcf4e97628b6b4a3\": container with ID starting with 25c4b6cb4be2601706f0fd9eb0d6ee888769d027159f33f5fcf4e97628b6b4a3 not found: ID does not exist" containerID="25c4b6cb4be2601706f0fd9eb0d6ee888769d027159f33f5fcf4e97628b6b4a3"
Oct 30 13:28:21.173909 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:21.173848    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25c4b6cb4be2601706f0fd9eb0d6ee888769d027159f33f5fcf4e97628b6b4a3"} err="failed to get container status \"25c4b6cb4be2601706f0fd9eb0d6ee888769d027159f33f5fcf4e97628b6b4a3\": rpc error: code = NotFound desc = could not find container \"25c4b6cb4be2601706f0fd9eb0d6ee888769d027159f33f5fcf4e97628b6b4a3\": container with ID starting with 25c4b6cb4be2601706f0fd9eb0d6ee888769d027159f33f5fcf4e97628b6b4a3 not found: ID does not exist"
Oct 30 13:28:21.216836 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:21.216811    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-4651e8/test-model-4651e8-kserve-5c7fcb7f7-8n9zr"]
Oct 30 13:28:21.229182 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:21.229157    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-4651e8/test-model-4651e8-kserve-5c7fcb7f7-8n9zr"]
Oct 30 13:28:22.973126 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:22.973095    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff5899b6-dd04-481f-bf5d-d4ce4e1f981d" path="/var/lib/kubelet/pods/ff5899b6-dd04-481f-bf5d-d4ce4e1f981d/volumes"
Oct 30 13:28:28.117906 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:28.117824    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf"]
Oct 30 13:28:28.118650 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:28.118634    4364 cpu_manager.go:414] "containerMap: removing container" podUID="79263302-817d-49f0-91a0-43281bff1ebd" containerName="payload-pre-processing"
Oct 30 13:28:28.118708 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:28.118654    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79263302-817d-49f0-91a0-43281bff1ebd" containerName="payload-pre-processing"
Oct 30 13:28:28.118708 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:28.118676    4364 cpu_manager.go:414] "containerMap: removing container" podUID="ebad4af8-38a2-4da7-a1ec-b56ef4c48a79" containerName="istio-proxy"
Oct 30 13:28:28.118708 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:28.118682    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebad4af8-38a2-4da7-a1ec-b56ef4c48a79" containerName="istio-proxy"
Oct 30 13:28:28.118824 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:28.118716    4364 cpu_manager.go:414] "containerMap: removing container" podUID="ff5899b6-dd04-481f-bf5d-d4ce4e1f981d" containerName="main"
Oct 30 13:28:28.118824 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:28.118721    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff5899b6-dd04-481f-bf5d-d4ce4e1f981d" containerName="main"
Oct 30 13:28:28.118882 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:28.118840    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="79263302-817d-49f0-91a0-43281bff1ebd" containerName="payload-pre-processing"
Oct 30 13:28:28.118882 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:28.118851    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebad4af8-38a2-4da7-a1ec-b56ef4c48a79" containerName="istio-proxy"
Oct 30 13:28:28.118882 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:28.118864    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff5899b6-dd04-481f-bf5d-d4ce4e1f981d" containerName="main"
Oct 30 13:28:28.122954 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:28.122935    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf"
Oct 30 13:28:28.151681 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:28.151649    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf"]
Oct 30 13:28:28.152384 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:28.152353    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-cda0d9-openshift-default-dockercfg-jlswf\""
Oct 30 13:28:28.159470 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:28.159435    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6d93861-6630-4e50-95b8-d34b8e83e92e-istio-token\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf\" (UID: \"b6d93861-6630-4e50-95b8-d34b8e83e92e\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf"
Oct 30 13:28:28.159771 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:28.159744    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6d93861-6630-4e50-95b8-d34b8e83e92e-istiod-ca-cert\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf\" (UID: \"b6d93861-6630-4e50-95b8-d34b8e83e92e\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf"
Oct 30 13:28:28.159948 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:28.159929    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6d93861-6630-4e50-95b8-d34b8e83e92e-istio-envoy\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf\" (UID: \"b6d93861-6630-4e50-95b8-d34b8e83e92e\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf"
Oct 30 13:28:28.160111 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:28.160094    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6d93861-6630-4e50-95b8-d34b8e83e92e-istio-podinfo\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf\" (UID: \"b6d93861-6630-4e50-95b8-d34b8e83e92e\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf"
Oct 30 13:28:28.160307 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:28.160283    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6d93861-6630-4e50-95b8-d34b8e83e92e-workload-certs\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf\" (UID: \"b6d93861-6630-4e50-95b8-d34b8e83e92e\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf"
Oct 30 13:28:28.160471 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:28.160318    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6d93861-6630-4e50-95b8-d34b8e83e92e-istio-data\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf\" (UID: \"b6d93861-6630-4e50-95b8-d34b8e83e92e\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf"
Oct 30 13:28:28.160471 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:28.160374    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cvz5\" (UniqueName: \"kubernetes.io/projected/b6d93861-6630-4e50-95b8-d34b8e83e92e-kube-api-access-7cvz5\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf\" (UID: \"b6d93861-6630-4e50-95b8-d34b8e83e92e\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf"
Oct 30 13:28:28.160471 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:28.160416    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6d93861-6630-4e50-95b8-d34b8e83e92e-credential-socket\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf\" (UID: \"b6d93861-6630-4e50-95b8-d34b8e83e92e\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf"
Oct 30 13:28:28.160754 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:28.160501    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6d93861-6630-4e50-95b8-d34b8e83e92e-workload-socket\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf\" (UID: \"b6d93861-6630-4e50-95b8-d34b8e83e92e\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf"
Oct 30 13:28:28.261387 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:28.261351    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cvz5\" (UniqueName: \"kubernetes.io/projected/b6d93861-6630-4e50-95b8-d34b8e83e92e-kube-api-access-7cvz5\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf\" (UID: \"b6d93861-6630-4e50-95b8-d34b8e83e92e\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf"
Oct 30 13:28:28.261574 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:28.261400    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6d93861-6630-4e50-95b8-d34b8e83e92e-credential-socket\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf\" (UID: \"b6d93861-6630-4e50-95b8-d34b8e83e92e\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf"
Oct 30 13:28:28.261574 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:28.261482    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6d93861-6630-4e50-95b8-d34b8e83e92e-workload-socket\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf\" (UID: \"b6d93861-6630-4e50-95b8-d34b8e83e92e\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf"
Oct 30 13:28:28.261574 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:28.261535    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6d93861-6630-4e50-95b8-d34b8e83e92e-istio-token\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf\" (UID: \"b6d93861-6630-4e50-95b8-d34b8e83e92e\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf"
Oct 30 13:28:28.261733 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:28.261592    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6d93861-6630-4e50-95b8-d34b8e83e92e-istiod-ca-cert\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf\" (UID: \"b6d93861-6630-4e50-95b8-d34b8e83e92e\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf"
Oct 30 13:28:28.261789 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:28.261771    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6d93861-6630-4e50-95b8-d34b8e83e92e-istio-envoy\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf\" (UID: \"b6d93861-6630-4e50-95b8-d34b8e83e92e\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf"
Oct 30 13:28:28.261878 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:28.261862    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6d93861-6630-4e50-95b8-d34b8e83e92e-istio-podinfo\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf\" (UID: \"b6d93861-6630-4e50-95b8-d34b8e83e92e\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf"
Oct 30 13:28:28.261962 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:28.261939    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6d93861-6630-4e50-95b8-d34b8e83e92e-credential-socket\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf\" (UID: \"b6d93861-6630-4e50-95b8-d34b8e83e92e\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf"
Oct 30 13:28:28.262006 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:28.261972    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6d93861-6630-4e50-95b8-d34b8e83e92e-workload-socket\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf\" (UID: \"b6d93861-6630-4e50-95b8-d34b8e83e92e\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf"
Oct 30 13:28:28.262046 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:28.262011    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6d93861-6630-4e50-95b8-d34b8e83e92e-workload-certs\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf\" (UID: \"b6d93861-6630-4e50-95b8-d34b8e83e92e\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf"
Oct 30 13:28:28.262094 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:28.262041    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6d93861-6630-4e50-95b8-d34b8e83e92e-istio-data\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf\" (UID: \"b6d93861-6630-4e50-95b8-d34b8e83e92e\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf"
Oct 30 13:28:28.262259 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:28.262234    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6d93861-6630-4e50-95b8-d34b8e83e92e-workload-certs\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf\" (UID: \"b6d93861-6630-4e50-95b8-d34b8e83e92e\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf"
Oct 30 13:28:28.262398 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:28.262376    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6d93861-6630-4e50-95b8-d34b8e83e92e-istio-data\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf\" (UID: \"b6d93861-6630-4e50-95b8-d34b8e83e92e\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf"
Oct 30 13:28:28.262463 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:28.262398    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6d93861-6630-4e50-95b8-d34b8e83e92e-istiod-ca-cert\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf\" (UID: \"b6d93861-6630-4e50-95b8-d34b8e83e92e\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf"
Oct 30 13:28:28.264273 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:28.264245    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6d93861-6630-4e50-95b8-d34b8e83e92e-istio-envoy\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf\" (UID: \"b6d93861-6630-4e50-95b8-d34b8e83e92e\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf"
Oct 30 13:28:28.264701 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:28.264679    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6d93861-6630-4e50-95b8-d34b8e83e92e-istio-podinfo\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf\" (UID: \"b6d93861-6630-4e50-95b8-d34b8e83e92e\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf"
Oct 30 13:28:28.290300 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:28.290270    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cvz5\" (UniqueName: \"kubernetes.io/projected/b6d93861-6630-4e50-95b8-d34b8e83e92e-kube-api-access-7cvz5\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf\" (UID: \"b6d93861-6630-4e50-95b8-d34b8e83e92e\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf"
Oct 30 13:28:28.291295 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:28.291266    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6d93861-6630-4e50-95b8-d34b8e83e92e-istio-token\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf\" (UID: \"b6d93861-6630-4e50-95b8-d34b8e83e92e\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf"
Oct 30 13:28:28.439527 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:28.439440    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf"
Oct 30 13:28:28.769106 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:28.769015    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf"]
Oct 30 13:28:28.769811 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:28:28.769763    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6d93861_6630_4e50_95b8_d34b8e83e92e.slice/crio-7ee3eb1a4c9a5739de21b0c2b5a12614817ac8a52324110f8e829ab72a9c5ce6 WatchSource:0}: Error finding container 7ee3eb1a4c9a5739de21b0c2b5a12614817ac8a52324110f8e829ab72a9c5ce6: Status 404 returned error can't find the container with id 7ee3eb1a4c9a5739de21b0c2b5a12614817ac8a52324110f8e829ab72a9c5ce6
Oct 30 13:28:28.774225 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:28.774184    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:28:28.774355 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:28.774283    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:28:28.774355 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:28.774329    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:28:29.232365 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:29.232313    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf" event={"ID":"b6d93861-6630-4e50-95b8-d34b8e83e92e","Type":"ContainerStarted","Data":"474c7970fef0a0f1e8e43f9e5a6ca011a12260ceffa91f1b32b8d27648a6b363"}
Oct 30 13:28:29.232910 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:29.232383    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf" event={"ID":"b6d93861-6630-4e50-95b8-d34b8e83e92e","Type":"ContainerStarted","Data":"7ee3eb1a4c9a5739de21b0c2b5a12614817ac8a52324110f8e829ab72a9c5ce6"}
Oct 30 13:28:29.293377 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:29.293317    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf" podStartSLOduration=1.2933029120000001 podStartE2EDuration="1.293302912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:28:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:28:29.290784017 +0000 UTC m=+2208.457148665" watchObservedRunningTime="2026-10-30 13:28:29.293302912 +0000 UTC m=+2208.459667634"
Oct 30 13:28:29.440228 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:29.440194    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf"
Oct 30 13:28:29.442224 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:29.442199    4364 patch_prober.go:29] interesting pod/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body=
Oct 30 13:28:29.442351 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:29.442262    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf" podUID="b6d93861-6630-4e50-95b8-d34b8e83e92e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused"
Oct 30 13:28:30.440274 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:30.440240    4364 patch_prober.go:29] interesting pod/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body=
Oct 30 13:28:30.440724 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:30.440293    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf" podUID="b6d93861-6630-4e50-95b8-d34b8e83e92e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused"
Oct 30 13:28:31.439882 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:31.439842    4364 patch_prober.go:29] interesting pod/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body=
Oct 30 13:28:31.440082 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:31.439906    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf" podUID="b6d93861-6630-4e50-95b8-d34b8e83e92e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused"
Oct 30 13:28:32.442973 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:32.442937    4364 patch_prober.go:29] interesting pod/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body=
Oct 30 13:28:32.443398 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:32.442995    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf" podUID="b6d93861-6630-4e50-95b8-d34b8e83e92e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused"
Oct 30 13:28:32.974153 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:32.974044    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-47628d-668fb856c4-q5lp4"]
Oct 30 13:28:33.026865 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:33.009809    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-47628d-6c75669467-c66k4"]
Oct 30 13:28:33.440636 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:33.440170    4364 patch_prober.go:29] interesting pod/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body=
Oct 30 13:28:33.440636 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:33.440242    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf" podUID="b6d93861-6630-4e50-95b8-d34b8e83e92e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused"
Oct 30 13:28:34.440480 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:34.440448    4364 patch_prober.go:29] interesting pod/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body=
Oct 30 13:28:34.440886 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:34.440499    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf" podUID="b6d93861-6630-4e50-95b8-d34b8e83e92e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused"
Oct 30 13:28:35.440718 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:35.440685    4364 patch_prober.go:29] interesting pod/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body=
Oct 30 13:28:35.441118 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:35.440748    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf" podUID="b6d93861-6630-4e50-95b8-d34b8e83e92e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused"
Oct 30 13:28:36.445173 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:36.445145    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf"
Oct 30 13:28:36.445585 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:36.445453    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf"
Oct 30 13:28:36.446054 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:36.446035    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf"
Oct 30 13:28:37.974764 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:37.974720    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-47628d-668fb856c4-q5lp4" podUID="981394e3-c7de-433d-a190-0ca90b59c64b" containerName="payload-processing" containerID="cri-o://94b7100e50a0976a08db439bd7595920a8f47523d9407abec263109f5c967917" gracePeriod=25
Oct 30 13:28:37.995500 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:37.995467    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4651e8-67665669fd-qdsnz"]
Oct 30 13:28:38.010767 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:38.010727    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-47628d-6c75669467-c66k4" podUID="db25e66a-ea74-44b2-9d02-f1ece73cef2a" containerName="payload-pre-processing" containerID="cri-o://63a43f91ba1f819858c9e5f331ed8a9db3c3527a7476382379323b43102aa649" gracePeriod=25
Oct 30 13:28:38.040690 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:38.040663    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4651e8-8f64b59bc-2xng9"]
Oct 30 13:28:38.701247 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:38.701208    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-8b77b0/sub-test-model-8b77b0-kserve-5f5b478b5c-dcqbx"]
Oct 30 13:28:38.702862 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:38.702816    4364 cpu_manager.go:414] "containerMap: removing container" podUID="ebad4af8-38a2-4da7-a1ec-b56ef4c48a79" containerName="istio-proxy"
Oct 30 13:28:38.702862 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:38.702859    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebad4af8-38a2-4da7-a1ec-b56ef4c48a79" containerName="istio-proxy"
Oct 30 13:28:38.703081 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:38.703051    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebad4af8-38a2-4da7-a1ec-b56ef4c48a79" containerName="istio-proxy"
Oct 30 13:28:38.707723 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:38.707698    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-8b77b0/sub-test-model-8b77b0-kserve-5f5b478b5c-dcqbx"
Oct 30 13:28:38.729272 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:38.729243    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-8b77b0/sub-test-model-8b77b0-kserve-5f5b478b5c-dcqbx"]
Oct 30 13:28:38.730147 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:38.729988    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-8b77b0\"/\"kube-root-ca.crt\""
Oct 30 13:28:38.730290 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:38.730173    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-8b77b0\"/\"openshift-service-ca.crt\""
Oct 30 13:28:38.733927 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:38.733894    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-8b77b0\"/\"sub-test-model-8b77b0-kserve-self-signed-certs\""
Oct 30 13:28:38.733927 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:38.733897    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-8b77b0\"/\"default-dockercfg-z7nx4\""
Oct 30 13:28:38.875845 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:38.875813    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb255481-3535-4bbe-a2c5-b4c1ba925fe4-tls-certs\") pod \"sub-test-model-8b77b0-kserve-5f5b478b5c-dcqbx\" (UID: \"eb255481-3535-4bbe-a2c5-b4c1ba925fe4\") " pod="ai-tenant-e2e-shared-a-w2-8b77b0/sub-test-model-8b77b0-kserve-5f5b478b5c-dcqbx"
Oct 30 13:28:38.876032 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:38.875977    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb255481-3535-4bbe-a2c5-b4c1ba925fe4-home\") pod \"sub-test-model-8b77b0-kserve-5f5b478b5c-dcqbx\" (UID: \"eb255481-3535-4bbe-a2c5-b4c1ba925fe4\") " pod="ai-tenant-e2e-shared-a-w2-8b77b0/sub-test-model-8b77b0-kserve-5f5b478b5c-dcqbx"
Oct 30 13:28:38.876096 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:38.876075    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srskp\" (UniqueName: \"kubernetes.io/projected/eb255481-3535-4bbe-a2c5-b4c1ba925fe4-kube-api-access-srskp\") pod \"sub-test-model-8b77b0-kserve-5f5b478b5c-dcqbx\" (UID: \"eb255481-3535-4bbe-a2c5-b4c1ba925fe4\") " pod="ai-tenant-e2e-shared-a-w2-8b77b0/sub-test-model-8b77b0-kserve-5f5b478b5c-dcqbx"
Oct 30 13:28:38.876213 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:38.876191    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb255481-3535-4bbe-a2c5-b4c1ba925fe4-tmp-dir\") pod \"sub-test-model-8b77b0-kserve-5f5b478b5c-dcqbx\" (UID: \"eb255481-3535-4bbe-a2c5-b4c1ba925fe4\") " pod="ai-tenant-e2e-shared-a-w2-8b77b0/sub-test-model-8b77b0-kserve-5f5b478b5c-dcqbx"
Oct 30 13:28:38.876280 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:38.876228    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb255481-3535-4bbe-a2c5-b4c1ba925fe4-model-cache\") pod \"sub-test-model-8b77b0-kserve-5f5b478b5c-dcqbx\" (UID: \"eb255481-3535-4bbe-a2c5-b4c1ba925fe4\") " pod="ai-tenant-e2e-shared-a-w2-8b77b0/sub-test-model-8b77b0-kserve-5f5b478b5c-dcqbx"
Oct 30 13:28:38.876351 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:38.876329    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb255481-3535-4bbe-a2c5-b4c1ba925fe4-dshm\") pod \"sub-test-model-8b77b0-kserve-5f5b478b5c-dcqbx\" (UID: \"eb255481-3535-4bbe-a2c5-b4c1ba925fe4\") " pod="ai-tenant-e2e-shared-a-w2-8b77b0/sub-test-model-8b77b0-kserve-5f5b478b5c-dcqbx"
Oct 30 13:28:38.978095 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:38.978017    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb255481-3535-4bbe-a2c5-b4c1ba925fe4-tls-certs\") pod \"sub-test-model-8b77b0-kserve-5f5b478b5c-dcqbx\" (UID: \"eb255481-3535-4bbe-a2c5-b4c1ba925fe4\") " pod="ai-tenant-e2e-shared-a-w2-8b77b0/sub-test-model-8b77b0-kserve-5f5b478b5c-dcqbx"
Oct 30 13:28:38.978575 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:38.978130    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb255481-3535-4bbe-a2c5-b4c1ba925fe4-home\") pod \"sub-test-model-8b77b0-kserve-5f5b478b5c-dcqbx\" (UID: \"eb255481-3535-4bbe-a2c5-b4c1ba925fe4\") " pod="ai-tenant-e2e-shared-a-w2-8b77b0/sub-test-model-8b77b0-kserve-5f5b478b5c-dcqbx"
Oct 30 13:28:38.978575 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:38.978184    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srskp\" (UniqueName: \"kubernetes.io/projected/eb255481-3535-4bbe-a2c5-b4c1ba925fe4-kube-api-access-srskp\") pod \"sub-test-model-8b77b0-kserve-5f5b478b5c-dcqbx\" (UID: \"eb255481-3535-4bbe-a2c5-b4c1ba925fe4\") " pod="ai-tenant-e2e-shared-a-w2-8b77b0/sub-test-model-8b77b0-kserve-5f5b478b5c-dcqbx"
Oct 30 13:28:38.978575 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:38.978230    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb255481-3535-4bbe-a2c5-b4c1ba925fe4-tmp-dir\") pod \"sub-test-model-8b77b0-kserve-5f5b478b5c-dcqbx\" (UID: \"eb255481-3535-4bbe-a2c5-b4c1ba925fe4\") " pod="ai-tenant-e2e-shared-a-w2-8b77b0/sub-test-model-8b77b0-kserve-5f5b478b5c-dcqbx"
Oct 30 13:28:38.978575 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:38.978254    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb255481-3535-4bbe-a2c5-b4c1ba925fe4-model-cache\") pod \"sub-test-model-8b77b0-kserve-5f5b478b5c-dcqbx\" (UID: \"eb255481-3535-4bbe-a2c5-b4c1ba925fe4\") " pod="ai-tenant-e2e-shared-a-w2-8b77b0/sub-test-model-8b77b0-kserve-5f5b478b5c-dcqbx"
Oct 30 13:28:38.978575 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:38.978400    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb255481-3535-4bbe-a2c5-b4c1ba925fe4-dshm\") pod \"sub-test-model-8b77b0-kserve-5f5b478b5c-dcqbx\" (UID: \"eb255481-3535-4bbe-a2c5-b4c1ba925fe4\") " pod="ai-tenant-e2e-shared-a-w2-8b77b0/sub-test-model-8b77b0-kserve-5f5b478b5c-dcqbx"
Oct 30 13:28:38.978868 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:38.978643    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb255481-3535-4bbe-a2c5-b4c1ba925fe4-home\") pod \"sub-test-model-8b77b0-kserve-5f5b478b5c-dcqbx\" (UID: \"eb255481-3535-4bbe-a2c5-b4c1ba925fe4\") " pod="ai-tenant-e2e-shared-a-w2-8b77b0/sub-test-model-8b77b0-kserve-5f5b478b5c-dcqbx"
Oct 30 13:28:38.978868 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:38.978683    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb255481-3535-4bbe-a2c5-b4c1ba925fe4-tmp-dir\") pod \"sub-test-model-8b77b0-kserve-5f5b478b5c-dcqbx\" (UID: \"eb255481-3535-4bbe-a2c5-b4c1ba925fe4\") " pod="ai-tenant-e2e-shared-a-w2-8b77b0/sub-test-model-8b77b0-kserve-5f5b478b5c-dcqbx"
Oct 30 13:28:38.978868 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:38.978703    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb255481-3535-4bbe-a2c5-b4c1ba925fe4-model-cache\") pod \"sub-test-model-8b77b0-kserve-5f5b478b5c-dcqbx\" (UID: \"eb255481-3535-4bbe-a2c5-b4c1ba925fe4\") " pod="ai-tenant-e2e-shared-a-w2-8b77b0/sub-test-model-8b77b0-kserve-5f5b478b5c-dcqbx"
Oct 30 13:28:38.980462 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:38.980439    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb255481-3535-4bbe-a2c5-b4c1ba925fe4-dshm\") pod \"sub-test-model-8b77b0-kserve-5f5b478b5c-dcqbx\" (UID: \"eb255481-3535-4bbe-a2c5-b4c1ba925fe4\") " pod="ai-tenant-e2e-shared-a-w2-8b77b0/sub-test-model-8b77b0-kserve-5f5b478b5c-dcqbx"
Oct 30 13:28:38.980673 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:38.980655    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb255481-3535-4bbe-a2c5-b4c1ba925fe4-tls-certs\") pod \"sub-test-model-8b77b0-kserve-5f5b478b5c-dcqbx\" (UID: \"eb255481-3535-4bbe-a2c5-b4c1ba925fe4\") " pod="ai-tenant-e2e-shared-a-w2-8b77b0/sub-test-model-8b77b0-kserve-5f5b478b5c-dcqbx"
Oct 30 13:28:39.004157 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:39.004064    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srskp\" (UniqueName: \"kubernetes.io/projected/eb255481-3535-4bbe-a2c5-b4c1ba925fe4-kube-api-access-srskp\") pod \"sub-test-model-8b77b0-kserve-5f5b478b5c-dcqbx\" (UID: \"eb255481-3535-4bbe-a2c5-b4c1ba925fe4\") " pod="ai-tenant-e2e-shared-a-w2-8b77b0/sub-test-model-8b77b0-kserve-5f5b478b5c-dcqbx"
Oct 30 13:28:39.047566 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:39.046924    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-8b77b0/sub-test-model-8b77b0-kserve-5f5b478b5c-dcqbx"
Oct 30 13:28:39.567307 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:39.567278    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-8b77b0/sub-test-model-8b77b0-kserve-5f5b478b5c-dcqbx"]
Oct 30 13:28:39.567930 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:28:39.567902    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb255481_3535_4bbe_a2c5_b4c1ba925fe4.slice/crio-2e1a4b25958a238d3edc298c4cbb4fea39c072e673ba09b87b554bffbd3b0a9c WatchSource:0}: Error finding container 2e1a4b25958a238d3edc298c4cbb4fea39c072e673ba09b87b554bffbd3b0a9c: Status 404 returned error can't find the container with id 2e1a4b25958a238d3edc298c4cbb4fea39c072e673ba09b87b554bffbd3b0a9c
Oct 30 13:28:40.187075 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:40.187041    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:28:40.530361 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:40.530259    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-8b77b0/sub-test-model-8b77b0-kserve-5f5b478b5c-dcqbx" event={"ID":"eb255481-3535-4bbe-a2c5-b4c1ba925fe4","Type":"ContainerStarted","Data":"53486d32b0639e288121c74089dce280c3d588465574eceda30b3797bda049e9"}
Oct 30 13:28:40.530361 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:40.530313    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-8b77b0/sub-test-model-8b77b0-kserve-5f5b478b5c-dcqbx" event={"ID":"eb255481-3535-4bbe-a2c5-b4c1ba925fe4","Type":"ContainerStarted","Data":"2e1a4b25958a238d3edc298c4cbb4fea39c072e673ba09b87b554bffbd3b0a9c"}
Oct 30 13:28:40.600649 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:40.600586    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-8b77b0/sub-test-model-8b77b0-kserve-5f5b478b5c-dcqbx" podStartSLOduration=2.600571858 podStartE2EDuration="2.600571858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:28:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:28:40.596976506 +0000 UTC m=+2219.763341154" watchObservedRunningTime="2026-10-30 13:28:40.600571858 +0000 UTC m=+2219.766936506"
Oct 30 13:28:41.048554 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:41.048528    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-8b77b0/sub-test-model-8b77b0-kserve-5f5b478b5c-dcqbx"
Oct 30 13:28:41.544713 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:28:41.544677    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:28:42.999763 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:42.999423    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-4651e8-67665669fd-qdsnz" podUID="df0a589d-208c-4471-8626-07bc7b3ad709" containerName="payload-processing" containerID="cri-o://0f9f324765eca0c3b27095b8a1b4f694a5c229573ae25e9db526209020a39b04" gracePeriod=25
Oct 30 13:28:43.044634 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:43.041174    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4651e8-8f64b59bc-2xng9" podUID="25395e75-3362-4355-a31d-de00a81b32c8" containerName="payload-pre-processing" containerID="cri-o://ef0824db0e869daf55acff3879db70c5cd757db63fdb01ae114316bb80af7d6d" gracePeriod=25
Oct 30 13:28:43.513333 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:43.513272    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4651e8-67665669fd-qdsnz"
Oct 30 13:28:43.541161 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:43.541135    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df0a589d-208c-4471-8626-07bc7b3ad709-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df0a589d-208c-4471-8626-07bc7b3ad709-plugins-config-volume\") pod \"df0a589d-208c-4471-8626-07bc7b3ad709\" (UID: \"df0a589d-208c-4471-8626-07bc7b3ad709\") "
Oct 30 13:28:43.541314 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:43.541232    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df0a589d-208c-4471-8626-07bc7b3ad709-kube-api-access-ctcbw\" (UniqueName: \"kubernetes.io/projected/df0a589d-208c-4471-8626-07bc7b3ad709-kube-api-access-ctcbw\") pod \"df0a589d-208c-4471-8626-07bc7b3ad709\" (UID: \"df0a589d-208c-4471-8626-07bc7b3ad709\") "
Oct 30 13:28:43.541559 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:43.541532    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df0a589d-208c-4471-8626-07bc7b3ad709-plugins-config-volume" pod "df0a589d-208c-4471-8626-07bc7b3ad709" (UID: "df0a589d-208c-4471-8626-07bc7b3ad709"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:28:43.541815 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:43.541800    4364 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df0a589d-208c-4471-8626-07bc7b3ad709-plugins-config-volume\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:28:43.543384 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:43.543361    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df0a589d-208c-4471-8626-07bc7b3ad709-kube-api-access-ctcbw" pod "df0a589d-208c-4471-8626-07bc7b3ad709" (UID: "df0a589d-208c-4471-8626-07bc7b3ad709"). InnerVolumeSpecName "kube-api-access-ctcbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:28:43.604557 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:43.604517    4364 generic.go:356] "Generic (PLEG): container finished" podID="df0a589d-208c-4471-8626-07bc7b3ad709" containerID="0f9f324765eca0c3b27095b8a1b4f694a5c229573ae25e9db526209020a39b04" exitCode=0
Oct 30 13:28:43.604761 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:43.604643    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4651e8-67665669fd-qdsnz" event={"ID":"df0a589d-208c-4471-8626-07bc7b3ad709","Type":"ContainerDied","Data":"0f9f324765eca0c3b27095b8a1b4f694a5c229573ae25e9db526209020a39b04"}
Oct 30 13:28:43.604761 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:43.604696    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4651e8-67665669fd-qdsnz" event={"ID":"df0a589d-208c-4471-8626-07bc7b3ad709","Type":"ContainerDied","Data":"5728499dd201d7f3bbe523fc10470544764c73ce360832f1f5a0e85bd081a191"}
Oct 30 13:28:43.604761 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:43.604732    4364 scope.go:122] "RemoveContainer" containerID="0f9f324765eca0c3b27095b8a1b4f694a5c229573ae25e9db526209020a39b04"
Oct 30 13:28:43.604965 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:43.604907    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4651e8-67665669fd-qdsnz"
Oct 30 13:28:43.620358 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:43.620333    4364 scope.go:122] "RemoveContainer" containerID="0f9f324765eca0c3b27095b8a1b4f694a5c229573ae25e9db526209020a39b04"
Oct 30 13:28:43.620751 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:28:43.620721    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f9f324765eca0c3b27095b8a1b4f694a5c229573ae25e9db526209020a39b04\": container with ID starting with 0f9f324765eca0c3b27095b8a1b4f694a5c229573ae25e9db526209020a39b04 not found: ID does not exist" containerID="0f9f324765eca0c3b27095b8a1b4f694a5c229573ae25e9db526209020a39b04"
Oct 30 13:28:43.620827 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:43.620764    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f9f324765eca0c3b27095b8a1b4f694a5c229573ae25e9db526209020a39b04"} err="failed to get container status \"0f9f324765eca0c3b27095b8a1b4f694a5c229573ae25e9db526209020a39b04\": rpc error: code = NotFound desc = could not find container \"0f9f324765eca0c3b27095b8a1b4f694a5c229573ae25e9db526209020a39b04\": container with ID starting with 0f9f324765eca0c3b27095b8a1b4f694a5c229573ae25e9db526209020a39b04 not found: ID does not exist"
Oct 30 13:28:43.646232 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:43.643638    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctcbw\" (UniqueName: \"kubernetes.io/projected/df0a589d-208c-4471-8626-07bc7b3ad709-kube-api-access-ctcbw\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:28:43.660758 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:43.660734    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4651e8-67665669fd-qdsnz"]
Oct 30 13:28:43.665592 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:43.665568    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4651e8-67665669fd-qdsnz"]
Oct 30 13:28:43.922346 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:43.922305    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-47628d-668fb856c4-q5lp4" podUID="981394e3-c7de-433d-a190-0ca90b59c64b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:28:43.922620 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:43.922368    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-47628d-6c75669467-c66k4" podUID="db25e66a-ea74-44b2-9d02-f1ece73cef2a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:28:44.119266 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:44.119203    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4651e8-8f64b59bc-2xng9" podUID="25395e75-3362-4355-a31d-de00a81b32c8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:28:44.972528 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:44.972489    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df0a589d-208c-4471-8626-07bc7b3ad709" path="/var/lib/kubelet/pods/df0a589d-208c-4471-8626-07bc7b3ad709/volumes"
Oct 30 13:28:49.047694 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:49.047662    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-8b77b0/sub-test-model-8b77b0-kserve-5f5b478b5c-dcqbx"
Oct 30 13:28:49.061094 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:49.061067    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-8b77b0/sub-test-model-8b77b0-kserve-5f5b478b5c-dcqbx"
Oct 30 13:28:49.675651 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:49.675625    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-8b77b0/sub-test-model-8b77b0-kserve-5f5b478b5c-dcqbx"
Oct 30 13:28:53.922410 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:53.922367    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-47628d-6c75669467-c66k4" podUID="db25e66a-ea74-44b2-9d02-f1ece73cef2a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:28:53.922857 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:53.922371    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-47628d-668fb856c4-q5lp4" podUID="981394e3-c7de-433d-a190-0ca90b59c64b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:28:54.110157 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:54.110122    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4651e8-8f64b59bc-2xng9" podUID="25395e75-3362-4355-a31d-de00a81b32c8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:28:58.303073 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:58.303045    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-47628d-668fb856c4-q5lp4"
Oct 30 13:28:58.340883 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:58.340853    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/981394e3-c7de-433d-a190-0ca90b59c64b-kube-api-access-sfvnx\" (UniqueName: \"kubernetes.io/projected/981394e3-c7de-433d-a190-0ca90b59c64b-kube-api-access-sfvnx\") pod \"981394e3-c7de-433d-a190-0ca90b59c64b\" (UID: \"981394e3-c7de-433d-a190-0ca90b59c64b\") "
Oct 30 13:28:58.341019 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:58.340906    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/981394e3-c7de-433d-a190-0ca90b59c64b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/981394e3-c7de-433d-a190-0ca90b59c64b-plugins-config-volume\") pod \"981394e3-c7de-433d-a190-0ca90b59c64b\" (UID: \"981394e3-c7de-433d-a190-0ca90b59c64b\") "
Oct 30 13:28:58.341258 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:58.341238    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/981394e3-c7de-433d-a190-0ca90b59c64b-plugins-config-volume" pod "981394e3-c7de-433d-a190-0ca90b59c64b" (UID: "981394e3-c7de-433d-a190-0ca90b59c64b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:28:58.341456 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:58.341440    4364 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/981394e3-c7de-433d-a190-0ca90b59c64b-plugins-config-volume\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:28:58.343031 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:58.343002    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/981394e3-c7de-433d-a190-0ca90b59c64b-kube-api-access-sfvnx" pod "981394e3-c7de-433d-a190-0ca90b59c64b" (UID: "981394e3-c7de-433d-a190-0ca90b59c64b"). InnerVolumeSpecName "kube-api-access-sfvnx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:28:58.363516 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:58.363483    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-47628d-6c75669467-c66k4"
Oct 30 13:28:58.442516 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:58.442482    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db25e66a-ea74-44b2-9d02-f1ece73cef2a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db25e66a-ea74-44b2-9d02-f1ece73cef2a-plugins-config-volume\") pod \"db25e66a-ea74-44b2-9d02-f1ece73cef2a\" (UID: \"db25e66a-ea74-44b2-9d02-f1ece73cef2a\") "
Oct 30 13:28:58.442754 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:58.442635    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db25e66a-ea74-44b2-9d02-f1ece73cef2a-kube-api-access-jjnmd\" (UniqueName: \"kubernetes.io/projected/db25e66a-ea74-44b2-9d02-f1ece73cef2a-kube-api-access-jjnmd\") pod \"db25e66a-ea74-44b2-9d02-f1ece73cef2a\" (UID: \"db25e66a-ea74-44b2-9d02-f1ece73cef2a\") "
Oct 30 13:28:58.443011 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:58.442983    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db25e66a-ea74-44b2-9d02-f1ece73cef2a-plugins-config-volume" pod "db25e66a-ea74-44b2-9d02-f1ece73cef2a" (UID: "db25e66a-ea74-44b2-9d02-f1ece73cef2a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:28:58.443373 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:58.443352    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfvnx\" (UniqueName: \"kubernetes.io/projected/981394e3-c7de-433d-a190-0ca90b59c64b-kube-api-access-sfvnx\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:28:58.443450 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:58.443378    4364 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db25e66a-ea74-44b2-9d02-f1ece73cef2a-plugins-config-volume\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:28:58.444940 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:58.444919    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db25e66a-ea74-44b2-9d02-f1ece73cef2a-kube-api-access-jjnmd" pod "db25e66a-ea74-44b2-9d02-f1ece73cef2a" (UID: "db25e66a-ea74-44b2-9d02-f1ece73cef2a"). InnerVolumeSpecName "kube-api-access-jjnmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:28:58.544763 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:58.544679    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjnmd\" (UniqueName: \"kubernetes.io/projected/db25e66a-ea74-44b2-9d02-f1ece73cef2a-kube-api-access-jjnmd\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:28:58.728142 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:58.727832    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9"]
Oct 30 13:28:58.729382 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:58.729353    4364 cpu_manager.go:414] "containerMap: removing container" podUID="db25e66a-ea74-44b2-9d02-f1ece73cef2a" containerName="payload-pre-processing"
Oct 30 13:28:58.729382 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:58.729380    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db25e66a-ea74-44b2-9d02-f1ece73cef2a" containerName="payload-pre-processing"
Oct 30 13:28:58.729623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:58.729395    4364 cpu_manager.go:414] "containerMap: removing container" podUID="df0a589d-208c-4471-8626-07bc7b3ad709" containerName="payload-processing"
Oct 30 13:28:58.729623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:58.729403    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df0a589d-208c-4471-8626-07bc7b3ad709" containerName="payload-processing"
Oct 30 13:28:58.729623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:58.729444    4364 cpu_manager.go:414] "containerMap: removing container" podUID="981394e3-c7de-433d-a190-0ca90b59c64b" containerName="payload-processing"
Oct 30 13:28:58.729623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:58.729454    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="981394e3-c7de-433d-a190-0ca90b59c64b" containerName="payload-processing"
Oct 30 13:28:58.729848 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:58.729635    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="db25e66a-ea74-44b2-9d02-f1ece73cef2a" containerName="payload-pre-processing"
Oct 30 13:28:58.729848 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:58.729649    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="df0a589d-208c-4471-8626-07bc7b3ad709" containerName="payload-processing"
Oct 30 13:28:58.729848 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:58.729667    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="981394e3-c7de-433d-a190-0ca90b59c64b" containerName="payload-processing"
Oct 30 13:28:58.734228 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:58.734207    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9"
Oct 30 13:28:58.770194 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:58.770161    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9"]
Oct 30 13:28:58.774593 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:58.774562    4364 generic.go:356] "Generic (PLEG): container finished" podID="981394e3-c7de-433d-a190-0ca90b59c64b" containerID="94b7100e50a0976a08db439bd7595920a8f47523d9407abec263109f5c967917" exitCode=0
Oct 30 13:28:58.774800 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:58.774642    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-47628d-668fb856c4-q5lp4"
Oct 30 13:28:58.774800 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:58.774660    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-47628d-668fb856c4-q5lp4" event={"ID":"981394e3-c7de-433d-a190-0ca90b59c64b","Type":"ContainerDied","Data":"94b7100e50a0976a08db439bd7595920a8f47523d9407abec263109f5c967917"}
Oct 30 13:28:58.774800 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:58.774724    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-47628d-668fb856c4-q5lp4" event={"ID":"981394e3-c7de-433d-a190-0ca90b59c64b","Type":"ContainerDied","Data":"e105f8e66c8bb0cb635cdcc41c4c84a6289c96e7b795c8083547cf0c12906e3d"}
Oct 30 13:28:58.774800 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:58.774753    4364 scope.go:122] "RemoveContainer" containerID="94b7100e50a0976a08db439bd7595920a8f47523d9407abec263109f5c967917"
Oct 30 13:28:58.776824 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:58.776804    4364 generic.go:356] "Generic (PLEG): container finished" podID="db25e66a-ea74-44b2-9d02-f1ece73cef2a" containerID="63a43f91ba1f819858c9e5f331ed8a9db3c3527a7476382379323b43102aa649" exitCode=0
Oct 30 13:28:58.777050 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:58.777011    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-47628d-6c75669467-c66k4"
Oct 30 13:28:58.777158 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:58.777008    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-47628d-6c75669467-c66k4" event={"ID":"db25e66a-ea74-44b2-9d02-f1ece73cef2a","Type":"ContainerDied","Data":"63a43f91ba1f819858c9e5f331ed8a9db3c3527a7476382379323b43102aa649"}
Oct 30 13:28:58.777158 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:58.777130    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-47628d-6c75669467-c66k4" event={"ID":"db25e66a-ea74-44b2-9d02-f1ece73cef2a","Type":"ContainerDied","Data":"b136513c9b38371e3907daaf6de353aa1f08502b5483bd92797652091cd22f40"}
Oct 30 13:28:58.792212 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:58.792188    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-4b0ea9-openshift-default-dockercfg-wbmk9\""
Oct 30 13:28:58.797129 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:58.797094    4364 scope.go:122] "RemoveContainer" containerID="94b7100e50a0976a08db439bd7595920a8f47523d9407abec263109f5c967917"
Oct 30 13:28:58.797530 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:28:58.797500    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94b7100e50a0976a08db439bd7595920a8f47523d9407abec263109f5c967917\": container with ID starting with 94b7100e50a0976a08db439bd7595920a8f47523d9407abec263109f5c967917 not found: ID does not exist" containerID="94b7100e50a0976a08db439bd7595920a8f47523d9407abec263109f5c967917"
Oct 30 13:28:58.797663 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:58.797540    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94b7100e50a0976a08db439bd7595920a8f47523d9407abec263109f5c967917"} err="failed to get container status \"94b7100e50a0976a08db439bd7595920a8f47523d9407abec263109f5c967917\": rpc error: code = NotFound desc = could not find container \"94b7100e50a0976a08db439bd7595920a8f47523d9407abec263109f5c967917\": container with ID starting with 94b7100e50a0976a08db439bd7595920a8f47523d9407abec263109f5c967917 not found: ID does not exist"
Oct 30 13:28:58.797663 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:58.797584    4364 scope.go:122] "RemoveContainer" containerID="63a43f91ba1f819858c9e5f331ed8a9db3c3527a7476382379323b43102aa649"
Oct 30 13:28:58.814883 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:58.814856    4364 scope.go:122] "RemoveContainer" containerID="63a43f91ba1f819858c9e5f331ed8a9db3c3527a7476382379323b43102aa649"
Oct 30 13:28:58.815214 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:28:58.815189    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63a43f91ba1f819858c9e5f331ed8a9db3c3527a7476382379323b43102aa649\": container with ID starting with 63a43f91ba1f819858c9e5f331ed8a9db3c3527a7476382379323b43102aa649 not found: ID does not exist" containerID="63a43f91ba1f819858c9e5f331ed8a9db3c3527a7476382379323b43102aa649"
Oct 30 13:28:58.815303 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:58.815224    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63a43f91ba1f819858c9e5f331ed8a9db3c3527a7476382379323b43102aa649"} err="failed to get container status \"63a43f91ba1f819858c9e5f331ed8a9db3c3527a7476382379323b43102aa649\": rpc error: code = NotFound desc = could not find container \"63a43f91ba1f819858c9e5f331ed8a9db3c3527a7476382379323b43102aa649\": container with ID starting with 63a43f91ba1f819858c9e5f331ed8a9db3c3527a7476382379323b43102aa649 not found: ID does not exist"
Oct 30 13:28:58.869621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:58.852515    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a967f02-7c93-484e-9bd4-97e1a4f5fdf5-credential-socket\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9\" (UID: \"9a967f02-7c93-484e-9bd4-97e1a4f5fdf5\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9"
Oct 30 13:28:58.869621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:58.852581    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a967f02-7c93-484e-9bd4-97e1a4f5fdf5-istiod-ca-cert\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9\" (UID: \"9a967f02-7c93-484e-9bd4-97e1a4f5fdf5\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9"
Oct 30 13:28:58.869621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:58.852655    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a967f02-7c93-484e-9bd4-97e1a4f5fdf5-istio-data\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9\" (UID: \"9a967f02-7c93-484e-9bd4-97e1a4f5fdf5\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9"
Oct 30 13:28:58.869621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:58.852695    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a967f02-7c93-484e-9bd4-97e1a4f5fdf5-istio-token\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9\" (UID: \"9a967f02-7c93-484e-9bd4-97e1a4f5fdf5\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9"
Oct 30 13:28:58.869621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:58.852775    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a967f02-7c93-484e-9bd4-97e1a4f5fdf5-istio-podinfo\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9\" (UID: \"9a967f02-7c93-484e-9bd4-97e1a4f5fdf5\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9"
Oct 30 13:28:58.869621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:58.852823    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a967f02-7c93-484e-9bd4-97e1a4f5fdf5-istio-envoy\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9\" (UID: \"9a967f02-7c93-484e-9bd4-97e1a4f5fdf5\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9"
Oct 30 13:28:58.869621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:58.852858    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdkjw\" (UniqueName: \"kubernetes.io/projected/9a967f02-7c93-484e-9bd4-97e1a4f5fdf5-kube-api-access-xdkjw\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9\" (UID: \"9a967f02-7c93-484e-9bd4-97e1a4f5fdf5\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9"
Oct 30 13:28:58.869621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:58.852918    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a967f02-7c93-484e-9bd4-97e1a4f5fdf5-workload-socket\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9\" (UID: \"9a967f02-7c93-484e-9bd4-97e1a4f5fdf5\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9"
Oct 30 13:28:58.869621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:58.853025    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a967f02-7c93-484e-9bd4-97e1a4f5fdf5-workload-certs\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9\" (UID: \"9a967f02-7c93-484e-9bd4-97e1a4f5fdf5\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9"
Oct 30 13:28:59.036695 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:59.029945    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a967f02-7c93-484e-9bd4-97e1a4f5fdf5-credential-socket\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9\" (UID: \"9a967f02-7c93-484e-9bd4-97e1a4f5fdf5\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9"
Oct 30 13:28:59.036695 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:59.030009    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a967f02-7c93-484e-9bd4-97e1a4f5fdf5-istiod-ca-cert\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9\" (UID: \"9a967f02-7c93-484e-9bd4-97e1a4f5fdf5\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9"
Oct 30 13:28:59.036695 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:59.030057    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a967f02-7c93-484e-9bd4-97e1a4f5fdf5-istio-data\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9\" (UID: \"9a967f02-7c93-484e-9bd4-97e1a4f5fdf5\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9"
Oct 30 13:28:59.036695 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:59.030092    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a967f02-7c93-484e-9bd4-97e1a4f5fdf5-istio-token\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9\" (UID: \"9a967f02-7c93-484e-9bd4-97e1a4f5fdf5\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9"
Oct 30 13:28:59.036695 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:59.030168    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a967f02-7c93-484e-9bd4-97e1a4f5fdf5-istio-podinfo\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9\" (UID: \"9a967f02-7c93-484e-9bd4-97e1a4f5fdf5\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9"
Oct 30 13:28:59.036695 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:59.030216    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a967f02-7c93-484e-9bd4-97e1a4f5fdf5-istio-envoy\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9\" (UID: \"9a967f02-7c93-484e-9bd4-97e1a4f5fdf5\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9"
Oct 30 13:28:59.036695 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:59.030262    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdkjw\" (UniqueName: \"kubernetes.io/projected/9a967f02-7c93-484e-9bd4-97e1a4f5fdf5-kube-api-access-xdkjw\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9\" (UID: \"9a967f02-7c93-484e-9bd4-97e1a4f5fdf5\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9"
Oct 30 13:28:59.036695 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:59.030332    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a967f02-7c93-484e-9bd4-97e1a4f5fdf5-workload-socket\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9\" (UID: \"9a967f02-7c93-484e-9bd4-97e1a4f5fdf5\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9"
Oct 30 13:28:59.036695 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:59.030469    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a967f02-7c93-484e-9bd4-97e1a4f5fdf5-workload-certs\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9\" (UID: \"9a967f02-7c93-484e-9bd4-97e1a4f5fdf5\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9"
Oct 30 13:28:59.036695 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:59.031089    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a967f02-7c93-484e-9bd4-97e1a4f5fdf5-workload-certs\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9\" (UID: \"9a967f02-7c93-484e-9bd4-97e1a4f5fdf5\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9"
Oct 30 13:28:59.138232 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:59.138193    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a967f02-7c93-484e-9bd4-97e1a4f5fdf5-credential-socket\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9\" (UID: \"9a967f02-7c93-484e-9bd4-97e1a4f5fdf5\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9"
Oct 30 13:28:59.138914 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:59.138886    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a967f02-7c93-484e-9bd4-97e1a4f5fdf5-istiod-ca-cert\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9\" (UID: \"9a967f02-7c93-484e-9bd4-97e1a4f5fdf5\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9"
Oct 30 13:28:59.139164 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:59.139142    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a967f02-7c93-484e-9bd4-97e1a4f5fdf5-istio-data\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9\" (UID: \"9a967f02-7c93-484e-9bd4-97e1a4f5fdf5\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9"
Oct 30 13:28:59.142582 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:59.142554    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a967f02-7c93-484e-9bd4-97e1a4f5fdf5-workload-socket\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9\" (UID: \"9a967f02-7c93-484e-9bd4-97e1a4f5fdf5\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9"
Oct 30 13:28:59.146583 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:59.145525    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a967f02-7c93-484e-9bd4-97e1a4f5fdf5-istio-envoy\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9\" (UID: \"9a967f02-7c93-484e-9bd4-97e1a4f5fdf5\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9"
Oct 30 13:28:59.147507 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:59.147484    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a967f02-7c93-484e-9bd4-97e1a4f5fdf5-istio-podinfo\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9\" (UID: \"9a967f02-7c93-484e-9bd4-97e1a4f5fdf5\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9"
Oct 30 13:28:59.158664 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:59.157299    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-47628d-668fb856c4-q5lp4"]
Oct 30 13:28:59.158664 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:59.157344    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-47628d-668fb856c4-q5lp4"]
Oct 30 13:28:59.158664 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:59.157368    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-47628d-6c75669467-c66k4"]
Oct 30 13:28:59.168549 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:59.165678    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-47628d-6c75669467-c66k4"]
Oct 30 13:28:59.202360 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:59.202310    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdkjw\" (UniqueName: \"kubernetes.io/projected/9a967f02-7c93-484e-9bd4-97e1a4f5fdf5-kube-api-access-xdkjw\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9\" (UID: \"9a967f02-7c93-484e-9bd4-97e1a4f5fdf5\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9"
Oct 30 13:28:59.202984 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:59.202700    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a967f02-7c93-484e-9bd4-97e1a4f5fdf5-istio-token\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9\" (UID: \"9a967f02-7c93-484e-9bd4-97e1a4f5fdf5\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9"
Oct 30 13:28:59.371955 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:59.366102    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9"
Oct 30 13:28:59.709973 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:59.709939    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9"]
Oct 30 13:28:59.710377 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:28:59.710353    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a967f02_7c93_484e_9bd4_97e1a4f5fdf5.slice/crio-862e2b625017ff886ca44397a855b99bae1aa9e7d724de267e88049510e1082a WatchSource:0}: Error finding container 862e2b625017ff886ca44397a855b99bae1aa9e7d724de267e88049510e1082a: Status 404 returned error can't find the container with id 862e2b625017ff886ca44397a855b99bae1aa9e7d724de267e88049510e1082a
Oct 30 13:28:59.713321 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:59.713287    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:28:59.713407 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:59.713356    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:28:59.713407 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:59.713385    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:28:59.788225 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:28:59.788195    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9" event={"ID":"9a967f02-7c93-484e-9bd4-97e1a4f5fdf5","Type":"ContainerStarted","Data":"862e2b625017ff886ca44397a855b99bae1aa9e7d724de267e88049510e1082a"}
Oct 30 13:29:00.799785 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:00.799750    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9" event={"ID":"9a967f02-7c93-484e-9bd4-97e1a4f5fdf5","Type":"ContainerStarted","Data":"95dc1e29203e28a63d9a2c873fa4bd9873cccc99f836275aa57b1dedabca4994"}
Oct 30 13:29:00.853654 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:00.853586    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9" podStartSLOduration=2.853569084 podStartE2EDuration="2.853569084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:28:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:29:00.85081352 +0000 UTC m=+2240.017178169" watchObservedRunningTime="2026-10-30 13:29:00.853569084 +0000 UTC m=+2240.019933731"
Oct 30 13:29:00.991660 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:00.990718    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="981394e3-c7de-433d-a190-0ca90b59c64b" path="/var/lib/kubelet/pods/981394e3-c7de-433d-a190-0ca90b59c64b/volumes"
Oct 30 13:29:00.991660 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:00.991328    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db25e66a-ea74-44b2-9d02-f1ece73cef2a" path="/var/lib/kubelet/pods/db25e66a-ea74-44b2-9d02-f1ece73cef2a/volumes"
Oct 30 13:29:01.368826 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:01.368797    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9"
Oct 30 13:29:01.374099 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:01.374072    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9"
Oct 30 13:29:01.809257 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:01.809177    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9"
Oct 30 13:29:01.810116 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:01.810098    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9"
Oct 30 13:29:03.362924 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:03.362894    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4651e8-8f64b59bc-2xng9"
Oct 30 13:29:03.445252 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:03.445219    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25395e75-3362-4355-a31d-de00a81b32c8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25395e75-3362-4355-a31d-de00a81b32c8-plugins-config-volume\") pod \"25395e75-3362-4355-a31d-de00a81b32c8\" (UID: \"25395e75-3362-4355-a31d-de00a81b32c8\") "
Oct 30 13:29:03.445415 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:03.445309    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25395e75-3362-4355-a31d-de00a81b32c8-kube-api-access-rfntw\" (UniqueName: \"kubernetes.io/projected/25395e75-3362-4355-a31d-de00a81b32c8-kube-api-access-rfntw\") pod \"25395e75-3362-4355-a31d-de00a81b32c8\" (UID: \"25395e75-3362-4355-a31d-de00a81b32c8\") "
Oct 30 13:29:03.445611 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:03.445578    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25395e75-3362-4355-a31d-de00a81b32c8-plugins-config-volume" pod "25395e75-3362-4355-a31d-de00a81b32c8" (UID: "25395e75-3362-4355-a31d-de00a81b32c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:29:03.445969 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:03.445941    4364 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25395e75-3362-4355-a31d-de00a81b32c8-plugins-config-volume\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:29:03.447868 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:03.447839    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25395e75-3362-4355-a31d-de00a81b32c8-kube-api-access-rfntw" pod "25395e75-3362-4355-a31d-de00a81b32c8" (UID: "25395e75-3362-4355-a31d-de00a81b32c8"). InnerVolumeSpecName "kube-api-access-rfntw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:29:03.600677 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:03.600127    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfntw\" (UniqueName: \"kubernetes.io/projected/25395e75-3362-4355-a31d-de00a81b32c8-kube-api-access-rfntw\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:29:03.840117 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:03.840081    4364 generic.go:356] "Generic (PLEG): container finished" podID="25395e75-3362-4355-a31d-de00a81b32c8" containerID="ef0824db0e869daf55acff3879db70c5cd757db63fdb01ae114316bb80af7d6d" exitCode=0
Oct 30 13:29:03.840335 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:03.840128    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4651e8-8f64b59bc-2xng9"
Oct 30 13:29:03.840335 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:03.840171    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4651e8-8f64b59bc-2xng9" event={"ID":"25395e75-3362-4355-a31d-de00a81b32c8","Type":"ContainerDied","Data":"ef0824db0e869daf55acff3879db70c5cd757db63fdb01ae114316bb80af7d6d"}
Oct 30 13:29:03.840335 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:03.840231    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4651e8-8f64b59bc-2xng9" event={"ID":"25395e75-3362-4355-a31d-de00a81b32c8","Type":"ContainerDied","Data":"92b762214b637a42830807ea099eff878a92f380bbf945cffe4d9c52a0e299f0"}
Oct 30 13:29:03.840335 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:03.840261    4364 scope.go:122] "RemoveContainer" containerID="ef0824db0e869daf55acff3879db70c5cd757db63fdb01ae114316bb80af7d6d"
Oct 30 13:29:03.854469 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:03.854450    4364 scope.go:122] "RemoveContainer" containerID="ef0824db0e869daf55acff3879db70c5cd757db63fdb01ae114316bb80af7d6d"
Oct 30 13:29:03.854766 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:29:03.854742    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef0824db0e869daf55acff3879db70c5cd757db63fdb01ae114316bb80af7d6d\": container with ID starting with ef0824db0e869daf55acff3879db70c5cd757db63fdb01ae114316bb80af7d6d not found: ID does not exist" containerID="ef0824db0e869daf55acff3879db70c5cd757db63fdb01ae114316bb80af7d6d"
Oct 30 13:29:03.854842 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:03.854773    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef0824db0e869daf55acff3879db70c5cd757db63fdb01ae114316bb80af7d6d"} err="failed to get container status \"ef0824db0e869daf55acff3879db70c5cd757db63fdb01ae114316bb80af7d6d\": rpc error: code = NotFound desc = could not find container \"ef0824db0e869daf55acff3879db70c5cd757db63fdb01ae114316bb80af7d6d\": container with ID starting with ef0824db0e869daf55acff3879db70c5cd757db63fdb01ae114316bb80af7d6d not found: ID does not exist"
Oct 30 13:29:03.905413 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:03.905346    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4651e8-8f64b59bc-2xng9"]
Oct 30 13:29:03.989787 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:03.986843    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4651e8-8f64b59bc-2xng9"]
Oct 30 13:29:04.972850 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:04.972817    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25395e75-3362-4355-a31d-de00a81b32c8" path="/var/lib/kubelet/pods/25395e75-3362-4355-a31d-de00a81b32c8/volumes"
Oct 30 13:29:05.232235 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:05.232145    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b0ea9-5f686c7795-7527w"]
Oct 30 13:29:05.233069 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:05.233041    4364 cpu_manager.go:414] "containerMap: removing container" podUID="25395e75-3362-4355-a31d-de00a81b32c8" containerName="payload-pre-processing"
Oct 30 13:29:05.233069 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:05.233062    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25395e75-3362-4355-a31d-de00a81b32c8" containerName="payload-pre-processing"
Oct 30 13:29:05.233269 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:05.233257    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="25395e75-3362-4355-a31d-de00a81b32c8" containerName="payload-pre-processing"
Oct 30 13:29:05.238399 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:05.238379    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b0ea9-5f686c7795-7527w"
Oct 30 13:29:05.268077 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:05.267114    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-4b0ea9\""
Oct 30 13:29:05.268077 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:05.267305    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-4b0ea9-dockercfg-r9ddf\""
Oct 30 13:29:05.273731 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:05.272779    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b0ea9-5f686c7795-7527w"]
Oct 30 13:29:05.539681 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:05.538809    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2dwb\" (UniqueName: \"kubernetes.io/projected/c2597429-b2f7-4263-b715-19410183a9a7-kube-api-access-l2dwb\") pod \"payload-pre-processing-e2e-shared-b-w3-4b0ea9-5f686c7795-7527w\" (UID: \"c2597429-b2f7-4263-b715-19410183a9a7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b0ea9-5f686c7795-7527w"
Oct 30 13:29:05.561678 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:05.552000    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2597429-b2f7-4263-b715-19410183a9a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-4b0ea9-5f686c7795-7527w\" (UID: \"c2597429-b2f7-4263-b715-19410183a9a7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b0ea9-5f686c7795-7527w"
Oct 30 13:29:05.653349 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:05.653307    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2597429-b2f7-4263-b715-19410183a9a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-4b0ea9-5f686c7795-7527w\" (UID: \"c2597429-b2f7-4263-b715-19410183a9a7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b0ea9-5f686c7795-7527w"
Oct 30 13:29:05.653529 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:05.653426    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2dwb\" (UniqueName: \"kubernetes.io/projected/c2597429-b2f7-4263-b715-19410183a9a7-kube-api-access-l2dwb\") pod \"payload-pre-processing-e2e-shared-b-w3-4b0ea9-5f686c7795-7527w\" (UID: \"c2597429-b2f7-4263-b715-19410183a9a7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b0ea9-5f686c7795-7527w"
Oct 30 13:29:05.654084 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:05.654065    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2597429-b2f7-4263-b715-19410183a9a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-4b0ea9-5f686c7795-7527w\" (UID: \"c2597429-b2f7-4263-b715-19410183a9a7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b0ea9-5f686c7795-7527w"
Oct 30 13:29:05.678374 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:05.678345    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2dwb\" (UniqueName: \"kubernetes.io/projected/c2597429-b2f7-4263-b715-19410183a9a7-kube-api-access-l2dwb\") pod \"payload-pre-processing-e2e-shared-b-w3-4b0ea9-5f686c7795-7527w\" (UID: \"c2597429-b2f7-4263-b715-19410183a9a7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b0ea9-5f686c7795-7527w"
Oct 30 13:29:05.855868 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:05.855776    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b0ea9-5f686c7795-7527w"
Oct 30 13:29:06.163428 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:06.163394    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b0ea9-5f686c7795-7527w"]
Oct 30 13:29:06.163428 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:29:06.163412    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2597429_b2f7_4263_b715_19410183a9a7.slice/crio-353589a7a05af7db6d7cc40253ccc071f6db919e25a9c9709dd435db9d1befe3 WatchSource:0}: Error finding container 353589a7a05af7db6d7cc40253ccc071f6db919e25a9c9709dd435db9d1befe3: Status 404 returned error can't find the container with id 353589a7a05af7db6d7cc40253ccc071f6db919e25a9c9709dd435db9d1befe3
Oct 30 13:29:06.868714 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:06.868652    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b0ea9-5f686c7795-7527w" event={"ID":"c2597429-b2f7-4263-b715-19410183a9a7","Type":"ContainerStarted","Data":"f36d0c3f3e10c89bf508565ad24228081b8d4ed406030f13d9f89d110bbf498a"}
Oct 30 13:29:06.868926 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:06.868723    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b0ea9-5f686c7795-7527w" event={"ID":"c2597429-b2f7-4263-b715-19410183a9a7","Type":"ContainerStarted","Data":"353589a7a05af7db6d7cc40253ccc071f6db919e25a9c9709dd435db9d1befe3"}
Oct 30 13:29:06.868995 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:06.868924    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b0ea9-5f686c7795-7527w"
Oct 30 13:29:06.911828 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:06.911773    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b0ea9-5f686c7795-7527w" podStartSLOduration=1.911757964 podStartE2EDuration="1.911757964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:29:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:29:06.908570555 +0000 UTC m=+2246.074935202" watchObservedRunningTime="2026-10-30 13:29:06.911757964 +0000 UTC m=+2246.078122611"
Oct 30 13:29:08.188091 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:08.188055    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-5a6281/sub-test-model-5a6281-kserve-694f48b47-4j6cv"]
Oct 30 13:29:08.213871 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:08.203735    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-5a6281/sub-test-model-5a6281-kserve-694f48b47-4j6cv"
Oct 30 13:29:08.224767 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:08.222300    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-5a6281\"/\"default-dockercfg-hlf24\""
Oct 30 13:29:08.227610 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:08.227559    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-5a6281/sub-test-model-5a6281-kserve-694f48b47-4j6cv"]
Oct 30 13:29:08.238131 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:08.238101    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-5a6281\"/\"openshift-service-ca.crt\""
Oct 30 13:29:08.238131 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:08.238118    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-5a6281\"/\"kube-root-ca.crt\""
Oct 30 13:29:08.238348 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:08.238314    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-5a6281\"/\"sub-test-model-5a6281-kserve-self-signed-certs\""
Oct 30 13:29:08.313892 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:08.313858    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10cce01a-1759-4617-bf74-5433cb240a30-tmp-dir\") pod \"sub-test-model-5a6281-kserve-694f48b47-4j6cv\" (UID: \"10cce01a-1759-4617-bf74-5433cb240a30\") " pod="ai-tenant-e2e-shared-b-w2-5a6281/sub-test-model-5a6281-kserve-694f48b47-4j6cv"
Oct 30 13:29:08.313892 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:08.313904    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10cce01a-1759-4617-bf74-5433cb240a30-tls-certs\") pod \"sub-test-model-5a6281-kserve-694f48b47-4j6cv\" (UID: \"10cce01a-1759-4617-bf74-5433cb240a30\") " pod="ai-tenant-e2e-shared-b-w2-5a6281/sub-test-model-5a6281-kserve-694f48b47-4j6cv"
Oct 30 13:29:08.314167 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:08.314111    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10cce01a-1759-4617-bf74-5433cb240a30-home\") pod \"sub-test-model-5a6281-kserve-694f48b47-4j6cv\" (UID: \"10cce01a-1759-4617-bf74-5433cb240a30\") " pod="ai-tenant-e2e-shared-b-w2-5a6281/sub-test-model-5a6281-kserve-694f48b47-4j6cv"
Oct 30 13:29:08.314232 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:08.314200    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10cce01a-1759-4617-bf74-5433cb240a30-dshm\") pod \"sub-test-model-5a6281-kserve-694f48b47-4j6cv\" (UID: \"10cce01a-1759-4617-bf74-5433cb240a30\") " pod="ai-tenant-e2e-shared-b-w2-5a6281/sub-test-model-5a6281-kserve-694f48b47-4j6cv"
Oct 30 13:29:08.314293 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:08.314236    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10cce01a-1759-4617-bf74-5433cb240a30-model-cache\") pod \"sub-test-model-5a6281-kserve-694f48b47-4j6cv\" (UID: \"10cce01a-1759-4617-bf74-5433cb240a30\") " pod="ai-tenant-e2e-shared-b-w2-5a6281/sub-test-model-5a6281-kserve-694f48b47-4j6cv"
Oct 30 13:29:08.314345 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:08.314324    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccgvh\" (UniqueName: \"kubernetes.io/projected/10cce01a-1759-4617-bf74-5433cb240a30-kube-api-access-ccgvh\") pod \"sub-test-model-5a6281-kserve-694f48b47-4j6cv\" (UID: \"10cce01a-1759-4617-bf74-5433cb240a30\") " pod="ai-tenant-e2e-shared-b-w2-5a6281/sub-test-model-5a6281-kserve-694f48b47-4j6cv"
Oct 30 13:29:08.416357 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:08.416317    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10cce01a-1759-4617-bf74-5433cb240a30-tmp-dir\") pod \"sub-test-model-5a6281-kserve-694f48b47-4j6cv\" (UID: \"10cce01a-1759-4617-bf74-5433cb240a30\") " pod="ai-tenant-e2e-shared-b-w2-5a6281/sub-test-model-5a6281-kserve-694f48b47-4j6cv"
Oct 30 13:29:08.416538 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:08.416381    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10cce01a-1759-4617-bf74-5433cb240a30-tls-certs\") pod \"sub-test-model-5a6281-kserve-694f48b47-4j6cv\" (UID: \"10cce01a-1759-4617-bf74-5433cb240a30\") " pod="ai-tenant-e2e-shared-b-w2-5a6281/sub-test-model-5a6281-kserve-694f48b47-4j6cv"
Oct 30 13:29:08.416674 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:08.416572    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10cce01a-1759-4617-bf74-5433cb240a30-home\") pod \"sub-test-model-5a6281-kserve-694f48b47-4j6cv\" (UID: \"10cce01a-1759-4617-bf74-5433cb240a30\") " pod="ai-tenant-e2e-shared-b-w2-5a6281/sub-test-model-5a6281-kserve-694f48b47-4j6cv"
Oct 30 13:29:08.416674 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:08.416640    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10cce01a-1759-4617-bf74-5433cb240a30-dshm\") pod \"sub-test-model-5a6281-kserve-694f48b47-4j6cv\" (UID: \"10cce01a-1759-4617-bf74-5433cb240a30\") " pod="ai-tenant-e2e-shared-b-w2-5a6281/sub-test-model-5a6281-kserve-694f48b47-4j6cv"
Oct 30 13:29:08.416674 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:08.416666    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10cce01a-1759-4617-bf74-5433cb240a30-model-cache\") pod \"sub-test-model-5a6281-kserve-694f48b47-4j6cv\" (UID: \"10cce01a-1759-4617-bf74-5433cb240a30\") " pod="ai-tenant-e2e-shared-b-w2-5a6281/sub-test-model-5a6281-kserve-694f48b47-4j6cv"
Oct 30 13:29:08.416856 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:08.416717    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccgvh\" (UniqueName: \"kubernetes.io/projected/10cce01a-1759-4617-bf74-5433cb240a30-kube-api-access-ccgvh\") pod \"sub-test-model-5a6281-kserve-694f48b47-4j6cv\" (UID: \"10cce01a-1759-4617-bf74-5433cb240a30\") " pod="ai-tenant-e2e-shared-b-w2-5a6281/sub-test-model-5a6281-kserve-694f48b47-4j6cv"
Oct 30 13:29:08.416856 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:08.416758    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10cce01a-1759-4617-bf74-5433cb240a30-tmp-dir\") pod \"sub-test-model-5a6281-kserve-694f48b47-4j6cv\" (UID: \"10cce01a-1759-4617-bf74-5433cb240a30\") " pod="ai-tenant-e2e-shared-b-w2-5a6281/sub-test-model-5a6281-kserve-694f48b47-4j6cv"
Oct 30 13:29:08.417119 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:08.417088    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10cce01a-1759-4617-bf74-5433cb240a30-model-cache\") pod \"sub-test-model-5a6281-kserve-694f48b47-4j6cv\" (UID: \"10cce01a-1759-4617-bf74-5433cb240a30\") " pod="ai-tenant-e2e-shared-b-w2-5a6281/sub-test-model-5a6281-kserve-694f48b47-4j6cv"
Oct 30 13:29:08.417332 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:08.417309    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10cce01a-1759-4617-bf74-5433cb240a30-home\") pod \"sub-test-model-5a6281-kserve-694f48b47-4j6cv\" (UID: \"10cce01a-1759-4617-bf74-5433cb240a30\") " pod="ai-tenant-e2e-shared-b-w2-5a6281/sub-test-model-5a6281-kserve-694f48b47-4j6cv"
Oct 30 13:29:08.419114 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:08.419095    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10cce01a-1759-4617-bf74-5433cb240a30-tls-certs\") pod \"sub-test-model-5a6281-kserve-694f48b47-4j6cv\" (UID: \"10cce01a-1759-4617-bf74-5433cb240a30\") " pod="ai-tenant-e2e-shared-b-w2-5a6281/sub-test-model-5a6281-kserve-694f48b47-4j6cv"
Oct 30 13:29:08.419303 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:08.419282    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10cce01a-1759-4617-bf74-5433cb240a30-dshm\") pod \"sub-test-model-5a6281-kserve-694f48b47-4j6cv\" (UID: \"10cce01a-1759-4617-bf74-5433cb240a30\") " pod="ai-tenant-e2e-shared-b-w2-5a6281/sub-test-model-5a6281-kserve-694f48b47-4j6cv"
Oct 30 13:29:08.439692 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:08.439628    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccgvh\" (UniqueName: \"kubernetes.io/projected/10cce01a-1759-4617-bf74-5433cb240a30-kube-api-access-ccgvh\") pod \"sub-test-model-5a6281-kserve-694f48b47-4j6cv\" (UID: \"10cce01a-1759-4617-bf74-5433cb240a30\") " pod="ai-tenant-e2e-shared-b-w2-5a6281/sub-test-model-5a6281-kserve-694f48b47-4j6cv"
Oct 30 13:29:08.538360 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:08.537142    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-5a6281/sub-test-model-5a6281-kserve-694f48b47-4j6cv"
Oct 30 13:29:08.981511 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:29:08.981469    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod10cce01a_1759_4617_bf74_5433cb240a30.slice/crio-446d2a7739e732f9a16e887189be383f69b1655d759befe589e316f23faea915 WatchSource:0}: Error finding container 446d2a7739e732f9a16e887189be383f69b1655d759befe589e316f23faea915: Status 404 returned error can't find the container with id 446d2a7739e732f9a16e887189be383f69b1655d759befe589e316f23faea915
Oct 30 13:29:08.982186 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:08.982165    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-5a6281/sub-test-model-5a6281-kserve-694f48b47-4j6cv"]
Oct 30 13:29:09.898193 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:09.898154    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-5a6281/sub-test-model-5a6281-kserve-694f48b47-4j6cv" event={"ID":"10cce01a-1759-4617-bf74-5433cb240a30","Type":"ContainerStarted","Data":"1e6ef79452d544f8edd2bf46a2957db2ff671772df03b0cafed7739150aec696"}
Oct 30 13:29:09.898193 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:09.898203    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-5a6281/sub-test-model-5a6281-kserve-694f48b47-4j6cv" event={"ID":"10cce01a-1759-4617-bf74-5433cb240a30","Type":"ContainerStarted","Data":"446d2a7739e732f9a16e887189be383f69b1655d759befe589e316f23faea915"}
Oct 30 13:29:09.943104 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:09.943053    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-5a6281/sub-test-model-5a6281-kserve-694f48b47-4j6cv" podStartSLOduration=1.9430384699999999 podStartE2EDuration="1.94303847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:29:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:29:09.940314286 +0000 UTC m=+2249.106678935" watchObservedRunningTime="2026-10-30 13:29:09.94303847 +0000 UTC m=+2249.109403117"
Oct 30 13:29:10.538239 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:10.538105    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-5a6281/sub-test-model-5a6281-kserve-694f48b47-4j6cv"
Oct 30 13:29:11.512995 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:29:11.512956    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:29:17.881711 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:17.881674    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b0ea9-5f686c7795-7527w"
Oct 30 13:29:18.538587 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:18.538550    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-5a6281/sub-test-model-5a6281-kserve-694f48b47-4j6cv"
Oct 30 13:29:18.551261 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:18.551237    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-5a6281/sub-test-model-5a6281-kserve-694f48b47-4j6cv"
Oct 30 13:29:19.055607 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:19.055559    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-5a6281/sub-test-model-5a6281-kserve-694f48b47-4j6cv"
Oct 30 13:29:21.101720 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:21.101689    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks"]
Oct 30 13:29:21.102240 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:21.101953    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks" podUID="fb9e1069-efce-4243-b809-ad470e124bd1" containerName="istio-proxy" containerID="cri-o://8f51baee7dce5aa432e881fd4599e65672b53548272a2126d277054e6185aad7" gracePeriod=30
Oct 30 13:29:23.578516 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:23.578480    4364 patch_prober.go:29] interesting pod/e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:29:23.579026 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:23.578552    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks" podUID="fb9e1069-efce-4243-b809-ad470e124bd1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:29:26.412390 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:26.412361    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks"
Oct 30 13:29:26.518886 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:26.518856    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb9e1069-efce-4243-b809-ad470e124bd1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb9e1069-efce-4243-b809-ad470e124bd1-workload-certs\") pod \"fb9e1069-efce-4243-b809-ad470e124bd1\" (UID: \"fb9e1069-efce-4243-b809-ad470e124bd1\") "
Oct 30 13:29:26.519069 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:26.518940    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb9e1069-efce-4243-b809-ad470e124bd1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb9e1069-efce-4243-b809-ad470e124bd1-istiod-ca-cert\") pod \"fb9e1069-efce-4243-b809-ad470e124bd1\" (UID: \"fb9e1069-efce-4243-b809-ad470e124bd1\") "
Oct 30 13:29:26.519069 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:26.519031    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb9e1069-efce-4243-b809-ad470e124bd1-kube-api-access-hp8zx\" (UniqueName: \"kubernetes.io/projected/fb9e1069-efce-4243-b809-ad470e124bd1-kube-api-access-hp8zx\") pod \"fb9e1069-efce-4243-b809-ad470e124bd1\" (UID: \"fb9e1069-efce-4243-b809-ad470e124bd1\") "
Oct 30 13:29:26.519147 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:26.519078    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb9e1069-efce-4243-b809-ad470e124bd1-istio-token\" (UniqueName: \"kubernetes.io/projected/fb9e1069-efce-4243-b809-ad470e124bd1-istio-token\") pod \"fb9e1069-efce-4243-b809-ad470e124bd1\" (UID: \"fb9e1069-efce-4243-b809-ad470e124bd1\") "
Oct 30 13:29:26.519264 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:26.519174    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb9e1069-efce-4243-b809-ad470e124bd1-workload-certs" pod "fb9e1069-efce-4243-b809-ad470e124bd1" (UID: "fb9e1069-efce-4243-b809-ad470e124bd1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:29:26.519264 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:26.519214    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fb9e1069-efce-4243-b809-ad470e124bd1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb9e1069-efce-4243-b809-ad470e124bd1-istio-podinfo\") pod \"fb9e1069-efce-4243-b809-ad470e124bd1\" (UID: \"fb9e1069-efce-4243-b809-ad470e124bd1\") "
Oct 30 13:29:26.519375 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:26.519286    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb9e1069-efce-4243-b809-ad470e124bd1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb9e1069-efce-4243-b809-ad470e124bd1-istio-data\") pod \"fb9e1069-efce-4243-b809-ad470e124bd1\" (UID: \"fb9e1069-efce-4243-b809-ad470e124bd1\") "
Oct 30 13:29:26.519432 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:26.519399    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb9e1069-efce-4243-b809-ad470e124bd1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb9e1069-efce-4243-b809-ad470e124bd1-workload-socket\") pod \"fb9e1069-efce-4243-b809-ad470e124bd1\" (UID: \"fb9e1069-efce-4243-b809-ad470e124bd1\") "
Oct 30 13:29:26.519432 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:26.519414    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb9e1069-efce-4243-b809-ad470e124bd1-istiod-ca-cert" pod "fb9e1069-efce-4243-b809-ad470e124bd1" (UID: "fb9e1069-efce-4243-b809-ad470e124bd1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:29:26.519539 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:26.519513    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb9e1069-efce-4243-b809-ad470e124bd1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb9e1069-efce-4243-b809-ad470e124bd1-credential-socket\") pod \"fb9e1069-efce-4243-b809-ad470e124bd1\" (UID: \"fb9e1069-efce-4243-b809-ad470e124bd1\") "
Oct 30 13:29:26.519618 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:26.519582    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb9e1069-efce-4243-b809-ad470e124bd1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb9e1069-efce-4243-b809-ad470e124bd1-istio-envoy\") pod \"fb9e1069-efce-4243-b809-ad470e124bd1\" (UID: \"fb9e1069-efce-4243-b809-ad470e124bd1\") "
Oct 30 13:29:26.519689 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:26.519673    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb9e1069-efce-4243-b809-ad470e124bd1-istio-data" pod "fb9e1069-efce-4243-b809-ad470e124bd1" (UID: "fb9e1069-efce-4243-b809-ad470e124bd1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:29:26.519818 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:26.519773    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb9e1069-efce-4243-b809-ad470e124bd1-workload-socket" pod "fb9e1069-efce-4243-b809-ad470e124bd1" (UID: "fb9e1069-efce-4243-b809-ad470e124bd1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:29:26.519993 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:26.519932    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb9e1069-efce-4243-b809-ad470e124bd1-credential-socket" pod "fb9e1069-efce-4243-b809-ad470e124bd1" (UID: "fb9e1069-efce-4243-b809-ad470e124bd1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:29:26.520552 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:26.520524    4364 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb9e1069-efce-4243-b809-ad470e124bd1-workload-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:29:26.520743 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:26.520558    4364 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb9e1069-efce-4243-b809-ad470e124bd1-credential-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:29:26.520743 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:26.520573    4364 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb9e1069-efce-4243-b809-ad470e124bd1-workload-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:29:26.520743 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:26.520587    4364 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb9e1069-efce-4243-b809-ad470e124bd1-istiod-ca-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:29:26.520743 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:26.520637    4364 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb9e1069-efce-4243-b809-ad470e124bd1-istio-data\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:29:26.521779 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:26.521755    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb9e1069-efce-4243-b809-ad470e124bd1-istio-token" pod "fb9e1069-efce-4243-b809-ad470e124bd1" (UID: "fb9e1069-efce-4243-b809-ad470e124bd1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:29:26.521887 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:26.521850    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb9e1069-efce-4243-b809-ad470e124bd1-istio-podinfo" pod "fb9e1069-efce-4243-b809-ad470e124bd1" (UID: "fb9e1069-efce-4243-b809-ad470e124bd1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:29:26.522214 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:26.522195    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb9e1069-efce-4243-b809-ad470e124bd1-istio-envoy" pod "fb9e1069-efce-4243-b809-ad470e124bd1" (UID: "fb9e1069-efce-4243-b809-ad470e124bd1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:29:26.526050 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:26.526028    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb9e1069-efce-4243-b809-ad470e124bd1-kube-api-access-hp8zx" pod "fb9e1069-efce-4243-b809-ad470e124bd1" (UID: "fb9e1069-efce-4243-b809-ad470e124bd1"). InnerVolumeSpecName "kube-api-access-hp8zx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:29:26.621846 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:26.621807    4364 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb9e1069-efce-4243-b809-ad470e124bd1-istio-podinfo\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:29:26.621846 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:26.621837    4364 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb9e1069-efce-4243-b809-ad470e124bd1-istio-envoy\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:29:26.621846 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:26.621849    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hp8zx\" (UniqueName: \"kubernetes.io/projected/fb9e1069-efce-4243-b809-ad470e124bd1-kube-api-access-hp8zx\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:29:26.622085 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:26.621859    4364 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb9e1069-efce-4243-b809-ad470e124bd1-istio-token\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:29:27.126037 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:27.126004    4364 generic.go:356] "Generic (PLEG): container finished" podID="fb9e1069-efce-4243-b809-ad470e124bd1" containerID="8f51baee7dce5aa432e881fd4599e65672b53548272a2126d277054e6185aad7" exitCode=0
Oct 30 13:29:27.126279 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:27.126072    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks"
Oct 30 13:29:27.126279 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:27.126079    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks" event={"ID":"fb9e1069-efce-4243-b809-ad470e124bd1","Type":"ContainerDied","Data":"8f51baee7dce5aa432e881fd4599e65672b53548272a2126d277054e6185aad7"}
Oct 30 13:29:27.126279 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:27.126160    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks" event={"ID":"fb9e1069-efce-4243-b809-ad470e124bd1","Type":"ContainerDied","Data":"ce4d88a199fcf696ed42f84620a82c65d2884df1718dc8f9107deaf0cca0314e"}
Oct 30 13:29:27.126279 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:27.126184    4364 scope.go:122] "RemoveContainer" containerID="8f51baee7dce5aa432e881fd4599e65672b53548272a2126d277054e6185aad7"
Oct 30 13:29:27.137901 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:27.137884    4364 scope.go:122] "RemoveContainer" containerID="cae3c76fb0576f5591a338bc3d5958baa7cc4110ba759c4d7f4e2419ce57be51"
Oct 30 13:29:27.148234 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:27.148218    4364 scope.go:122] "RemoveContainer" containerID="8f51baee7dce5aa432e881fd4599e65672b53548272a2126d277054e6185aad7"
Oct 30 13:29:27.148526 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:29:27.148506    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f51baee7dce5aa432e881fd4599e65672b53548272a2126d277054e6185aad7\": container with ID starting with 8f51baee7dce5aa432e881fd4599e65672b53548272a2126d277054e6185aad7 not found: ID does not exist" containerID="8f51baee7dce5aa432e881fd4599e65672b53548272a2126d277054e6185aad7"
Oct 30 13:29:27.148573 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:27.148535    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f51baee7dce5aa432e881fd4599e65672b53548272a2126d277054e6185aad7"} err="failed to get container status \"8f51baee7dce5aa432e881fd4599e65672b53548272a2126d277054e6185aad7\": rpc error: code = NotFound desc = could not find container \"8f51baee7dce5aa432e881fd4599e65672b53548272a2126d277054e6185aad7\": container with ID starting with 8f51baee7dce5aa432e881fd4599e65672b53548272a2126d277054e6185aad7 not found: ID does not exist"
Oct 30 13:29:27.148573 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:27.148553    4364 scope.go:122] "RemoveContainer" containerID="cae3c76fb0576f5591a338bc3d5958baa7cc4110ba759c4d7f4e2419ce57be51"
Oct 30 13:29:27.148845 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:29:27.148829    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cae3c76fb0576f5591a338bc3d5958baa7cc4110ba759c4d7f4e2419ce57be51\": container with ID starting with cae3c76fb0576f5591a338bc3d5958baa7cc4110ba759c4d7f4e2419ce57be51 not found: ID does not exist" containerID="cae3c76fb0576f5591a338bc3d5958baa7cc4110ba759c4d7f4e2419ce57be51"
Oct 30 13:29:27.148902 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:27.148849    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cae3c76fb0576f5591a338bc3d5958baa7cc4110ba759c4d7f4e2419ce57be51"} err="failed to get container status \"cae3c76fb0576f5591a338bc3d5958baa7cc4110ba759c4d7f4e2419ce57be51\": rpc error: code = NotFound desc = could not find container \"cae3c76fb0576f5591a338bc3d5958baa7cc4110ba759c4d7f4e2419ce57be51\": container with ID starting with cae3c76fb0576f5591a338bc3d5958baa7cc4110ba759c4d7f4e2419ce57be51 not found: ID does not exist"
Oct 30 13:29:27.193943 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:27.193909    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks"]
Oct 30 13:29:27.205641 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:27.205612    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-4651e8-openshift-default-88ccf7749-cmrks"]
Oct 30 13:29:28.971905 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:28.971871    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb9e1069-efce-4243-b809-ad470e124bd1" path="/var/lib/kubelet/pods/fb9e1069-efce-4243-b809-ad470e124bd1/volumes"
Oct 30 13:29:33.429102 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:33.429069    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:29:41.507110 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:29:41.507066    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:29:51.908338 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:51.908297    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5c7ba3-574966bcc5-nvlwk"]
Oct 30 13:29:51.908944 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:51.908529    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5c7ba3-574966bcc5-nvlwk" podUID="b41649d1-4106-43bd-a16c-8c814f08de90" containerName="maas-api" containerID="cri-o://0fdc9161720893fd043167244786622614b7d6239f6d9c8affda0a22049ee55e" gracePeriod=30
Oct 30 13:29:52.323230 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:52.322819    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5c7ba3-574966bcc5-nvlwk"
Oct 30 13:29:52.443895 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:52.439622    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b41649d1-4106-43bd-a16c-8c814f08de90-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b41649d1-4106-43bd-a16c-8c814f08de90-maas-api-tls\") pod \"b41649d1-4106-43bd-a16c-8c814f08de90\" (UID: \"b41649d1-4106-43bd-a16c-8c814f08de90\") "
Oct 30 13:29:52.443895 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:52.439707    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b41649d1-4106-43bd-a16c-8c814f08de90-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b41649d1-4106-43bd-a16c-8c814f08de90-metrics-certs\") pod \"b41649d1-4106-43bd-a16c-8c814f08de90\" (UID: \"b41649d1-4106-43bd-a16c-8c814f08de90\") "
Oct 30 13:29:52.443895 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:52.439862    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b41649d1-4106-43bd-a16c-8c814f08de90-kube-api-access-wcj97\" (UniqueName: \"kubernetes.io/projected/b41649d1-4106-43bd-a16c-8c814f08de90-kube-api-access-wcj97\") pod \"b41649d1-4106-43bd-a16c-8c814f08de90\" (UID: \"b41649d1-4106-43bd-a16c-8c814f08de90\") "
Oct 30 13:29:52.443895 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:52.439927    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b41649d1-4106-43bd-a16c-8c814f08de90-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b41649d1-4106-43bd-a16c-8c814f08de90-openshift-service-ca\") pod \"b41649d1-4106-43bd-a16c-8c814f08de90\" (UID: \"b41649d1-4106-43bd-a16c-8c814f08de90\") "
Oct 30 13:29:52.446967 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:52.446925    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b41649d1-4106-43bd-a16c-8c814f08de90-openshift-service-ca" pod "b41649d1-4106-43bd-a16c-8c814f08de90" (UID: "b41649d1-4106-43bd-a16c-8c814f08de90"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:29:52.449540 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:52.449512    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b41649d1-4106-43bd-a16c-8c814f08de90-kube-api-access-wcj97" pod "b41649d1-4106-43bd-a16c-8c814f08de90" (UID: "b41649d1-4106-43bd-a16c-8c814f08de90"). InnerVolumeSpecName "kube-api-access-wcj97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:29:52.453851 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:52.453009    4364 generic.go:356] "Generic (PLEG): container finished" podID="b41649d1-4106-43bd-a16c-8c814f08de90" containerID="0fdc9161720893fd043167244786622614b7d6239f6d9c8affda0a22049ee55e" exitCode=0
Oct 30 13:29:52.453851 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:52.453107    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5c7ba3-574966bcc5-nvlwk" event={"ID":"b41649d1-4106-43bd-a16c-8c814f08de90","Type":"ContainerDied","Data":"0fdc9161720893fd043167244786622614b7d6239f6d9c8affda0a22049ee55e"}
Oct 30 13:29:52.453851 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:52.453157    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5c7ba3-574966bcc5-nvlwk" event={"ID":"b41649d1-4106-43bd-a16c-8c814f08de90","Type":"ContainerDied","Data":"15e928870996b7aa055a84851cf16716bd260a51a83fa6cc9beed5edbd838fe6"}
Oct 30 13:29:52.453851 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:52.453188    4364 scope.go:122] "RemoveContainer" containerID="0fdc9161720893fd043167244786622614b7d6239f6d9c8affda0a22049ee55e"
Oct 30 13:29:52.453851 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:52.453358    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5c7ba3-574966bcc5-nvlwk"
Oct 30 13:29:52.461580 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:52.461531    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b41649d1-4106-43bd-a16c-8c814f08de90-metrics-certs" pod "b41649d1-4106-43bd-a16c-8c814f08de90" (UID: "b41649d1-4106-43bd-a16c-8c814f08de90"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:29:52.461734 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:52.461662    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b41649d1-4106-43bd-a16c-8c814f08de90-maas-api-tls" pod "b41649d1-4106-43bd-a16c-8c814f08de90" (UID: "b41649d1-4106-43bd-a16c-8c814f08de90"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:29:52.486837 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:52.486806    4364 scope.go:122] "RemoveContainer" containerID="0fdc9161720893fd043167244786622614b7d6239f6d9c8affda0a22049ee55e"
Oct 30 13:29:52.488276 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:29:52.488243    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fdc9161720893fd043167244786622614b7d6239f6d9c8affda0a22049ee55e\": container with ID starting with 0fdc9161720893fd043167244786622614b7d6239f6d9c8affda0a22049ee55e not found: ID does not exist" containerID="0fdc9161720893fd043167244786622614b7d6239f6d9c8affda0a22049ee55e"
Oct 30 13:29:52.488506 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:52.488288    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fdc9161720893fd043167244786622614b7d6239f6d9c8affda0a22049ee55e"} err="failed to get container status \"0fdc9161720893fd043167244786622614b7d6239f6d9c8affda0a22049ee55e\": rpc error: code = NotFound desc = could not find container \"0fdc9161720893fd043167244786622614b7d6239f6d9c8affda0a22049ee55e\": container with ID starting with 0fdc9161720893fd043167244786622614b7d6239f6d9c8affda0a22049ee55e not found: ID does not exist"
Oct 30 13:29:52.619658 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:52.600886    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcj97\" (UniqueName: \"kubernetes.io/projected/b41649d1-4106-43bd-a16c-8c814f08de90-kube-api-access-wcj97\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:29:52.619658 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:52.600935    4364 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b41649d1-4106-43bd-a16c-8c814f08de90-openshift-service-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:29:52.619658 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:52.600949    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b41649d1-4106-43bd-a16c-8c814f08de90-maas-api-tls\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:29:52.619658 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:52.600967    4364 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b41649d1-4106-43bd-a16c-8c814f08de90-metrics-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:29:52.803530 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:52.803493    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5c7ba3-574966bcc5-nvlwk"]
Oct 30 13:29:52.814912 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:52.814885    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5c7ba3-574966bcc5-nvlwk"]
Oct 30 13:29:52.972143 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:29:52.972107    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b41649d1-4106-43bd-a16c-8c814f08de90" path="/var/lib/kubelet/pods/b41649d1-4106-43bd-a16c-8c814f08de90/volumes"
Oct 30 13:30:00.148431 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.148341    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-cda0d9-29889450-bnh6t"]
Oct 30 13:30:00.149345 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.149323    4364 cpu_manager.go:414] "containerMap: removing container" podUID="fb9e1069-efce-4243-b809-ad470e124bd1" containerName="istio-proxy"
Oct 30 13:30:00.149345 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.149346    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb9e1069-efce-4243-b809-ad470e124bd1" containerName="istio-proxy"
Oct 30 13:30:00.149531 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.149390    4364 cpu_manager.go:414] "containerMap: removing container" podUID="b41649d1-4106-43bd-a16c-8c814f08de90" containerName="maas-api"
Oct 30 13:30:00.149531 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.149399    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b41649d1-4106-43bd-a16c-8c814f08de90" containerName="maas-api"
Oct 30 13:30:00.149699 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.149650    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb9e1069-efce-4243-b809-ad470e124bd1" containerName="istio-proxy"
Oct 30 13:30:00.149699 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.149684    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="b41649d1-4106-43bd-a16c-8c814f08de90" containerName="maas-api"
Oct 30 13:30:00.149699 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.149696    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb9e1069-efce-4243-b809-ad470e124bd1" containerName="istio-proxy"
Oct 30 13:30:00.156577 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.156552    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8d82d5-29889450-g5qzp"]
Oct 30 13:30:00.156740 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.156701    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-cda0d9-29889450-bnh6t"
Oct 30 13:30:00.157499 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.157474    4364 cpu_manager.go:414] "containerMap: removing container" podUID="fb9e1069-efce-4243-b809-ad470e124bd1" containerName="istio-proxy"
Oct 30 13:30:00.157499 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.157497    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb9e1069-efce-4243-b809-ad470e124bd1" containerName="istio-proxy"
Oct 30 13:30:00.161042 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.161023    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8d82d5-29889450-g5qzp"
Oct 30 13:30:00.183191 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.183161    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-cda0d9-29889450-bnh6t"]
Oct 30 13:30:00.183894 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.183870    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\""
Oct 30 13:30:00.207582 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.207543    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d833bd-29889450-q6468"]
Oct 30 13:30:00.223051 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.220791    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8d82d5-29889450-g5qzp"]
Oct 30 13:30:00.223051 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.220951    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d833bd-29889450-q6468"
Oct 30 13:30:00.223051 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.222155    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b29df4-29889450-v8kpn"]
Oct 30 13:30:00.228312 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.228286    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b29df4-29889450-v8kpn"
Oct 30 13:30:00.228980 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.228961    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d833bd-29889450-q6468"]
Oct 30 13:30:00.244934 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.243338    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b29df4-29889450-v8kpn"]
Oct 30 13:30:00.250645 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.250588    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889450-4glnd"]
Oct 30 13:30:00.256395 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.256375    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889450-4glnd"
Oct 30 13:30:00.279835 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.266621    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889450-4glnd"]
Oct 30 13:30:00.303415 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.303381    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ace885ec-090f-469d-ba45-23bd11e84d9e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-8d82d5-29889450-g5qzp\" (UID: \"ace885ec-090f-469d-ba45-23bd11e84d9e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8d82d5-29889450-g5qzp"
Oct 30 13:30:00.303714 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.303687    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wck5\" (UniqueName: \"kubernetes.io/projected/d10daaa5-7cac-4205-8638-75ede670c28f-kube-api-access-6wck5\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-cda0d9-29889450-bnh6t\" (UID: \"d10daaa5-7cac-4205-8638-75ede670c28f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-cda0d9-29889450-bnh6t"
Oct 30 13:30:00.303912 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.303896    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d10daaa5-7cac-4205-8638-75ede670c28f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-cda0d9-29889450-bnh6t\" (UID: \"d10daaa5-7cac-4205-8638-75ede670c28f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-cda0d9-29889450-bnh6t"
Oct 30 13:30:00.304313 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.304292    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qf4w\" (UniqueName: \"kubernetes.io/projected/ace885ec-090f-469d-ba45-23bd11e84d9e-kube-api-access-2qf4w\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-8d82d5-29889450-g5qzp\" (UID: \"ace885ec-090f-469d-ba45-23bd11e84d9e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8d82d5-29889450-g5qzp"
Oct 30 13:30:00.309716 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.309689    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4b0ea9-29889450-dc8t7"]
Oct 30 13:30:00.321612 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.317063    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4b0ea9-29889450-dc8t7"
Oct 30 13:30:00.338697 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.338654    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9c6033-29889450-589x5"]
Oct 30 13:30:00.348354 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.348323    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9c6033-29889450-589x5"
Oct 30 13:30:00.356429 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.356397    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4b0ea9-29889450-dc8t7"]
Oct 30 13:30:00.366614 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.366571    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9c6033-29889450-589x5"]
Oct 30 13:30:00.411170 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.411077    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qf4w\" (UniqueName: \"kubernetes.io/projected/ace885ec-090f-469d-ba45-23bd11e84d9e-kube-api-access-2qf4w\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-8d82d5-29889450-g5qzp\" (UID: \"ace885ec-090f-469d-ba45-23bd11e84d9e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8d82d5-29889450-g5qzp"
Oct 30 13:30:00.411170 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.411158    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f08ebb94-561c-42b6-9430-3c1d78c5fc8a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-b29df4-29889450-v8kpn\" (UID: \"f08ebb94-561c-42b6-9430-3c1d78c5fc8a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b29df4-29889450-v8kpn"
Oct 30 13:30:00.411410 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.411185    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0239aaa-e1bf-4ad8-bf54-42920cc8009c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889450-4glnd\" (UID: \"a0239aaa-e1bf-4ad8-bf54-42920cc8009c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889450-4glnd"
Oct 30 13:30:00.411540 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.411524    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ace885ec-090f-469d-ba45-23bd11e84d9e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-8d82d5-29889450-g5qzp\" (UID: \"ace885ec-090f-469d-ba45-23bd11e84d9e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8d82d5-29889450-g5qzp"
Oct 30 13:30:00.411652 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.411585    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff661c30-4a5a-4a98-b640-3b2c3e4143e1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-d833bd-29889450-q6468\" (UID: \"ff661c30-4a5a-4a98-b640-3b2c3e4143e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d833bd-29889450-q6468"
Oct 30 13:30:00.411780 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.411758    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wck5\" (UniqueName: \"kubernetes.io/projected/d10daaa5-7cac-4205-8638-75ede670c28f-kube-api-access-6wck5\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-cda0d9-29889450-bnh6t\" (UID: \"d10daaa5-7cac-4205-8638-75ede670c28f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-cda0d9-29889450-bnh6t"
Oct 30 13:30:00.411898 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.411868    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d10daaa5-7cac-4205-8638-75ede670c28f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-cda0d9-29889450-bnh6t\" (UID: \"d10daaa5-7cac-4205-8638-75ede670c28f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-cda0d9-29889450-bnh6t"
Oct 30 13:30:00.412036 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.411922    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7ktw\" (UniqueName: \"kubernetes.io/projected/f08ebb94-561c-42b6-9430-3c1d78c5fc8a-kube-api-access-j7ktw\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-b29df4-29889450-v8kpn\" (UID: \"f08ebb94-561c-42b6-9430-3c1d78c5fc8a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b29df4-29889450-v8kpn"
Oct 30 13:30:00.412036 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.412001    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djtmf\" (UniqueName: \"kubernetes.io/projected/a0239aaa-e1bf-4ad8-bf54-42920cc8009c-kube-api-access-djtmf\") pod \"maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889450-4glnd\" (UID: \"a0239aaa-e1bf-4ad8-bf54-42920cc8009c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889450-4glnd"
Oct 30 13:30:00.412171 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.412046    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tq87\" (UniqueName: \"kubernetes.io/projected/ff661c30-4a5a-4a98-b640-3b2c3e4143e1-kube-api-access-6tq87\") pod \"maas-api-key-cleanup-e2e-worker-w3-d833bd-29889450-q6468\" (UID: \"ff661c30-4a5a-4a98-b640-3b2c3e4143e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d833bd-29889450-q6468"
Oct 30 13:30:00.412380 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.412359    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ace885ec-090f-469d-ba45-23bd11e84d9e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-8d82d5-29889450-g5qzp\" (UID: \"ace885ec-090f-469d-ba45-23bd11e84d9e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8d82d5-29889450-g5qzp"
Oct 30 13:30:00.412651 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.412588    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d10daaa5-7cac-4205-8638-75ede670c28f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-cda0d9-29889450-bnh6t\" (UID: \"d10daaa5-7cac-4205-8638-75ede670c28f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-cda0d9-29889450-bnh6t"
Oct 30 13:30:00.460707 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.460666    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wck5\" (UniqueName: \"kubernetes.io/projected/d10daaa5-7cac-4205-8638-75ede670c28f-kube-api-access-6wck5\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-cda0d9-29889450-bnh6t\" (UID: \"d10daaa5-7cac-4205-8638-75ede670c28f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-cda0d9-29889450-bnh6t"
Oct 30 13:30:00.461737 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.461714    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qf4w\" (UniqueName: \"kubernetes.io/projected/ace885ec-090f-469d-ba45-23bd11e84d9e-kube-api-access-2qf4w\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-8d82d5-29889450-g5qzp\" (UID: \"ace885ec-090f-469d-ba45-23bd11e84d9e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8d82d5-29889450-g5qzp"
Oct 30 13:30:00.507406 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.507371    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-cda0d9-29889450-bnh6t"
Oct 30 13:30:00.512876 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.512849    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8d82d5-29889450-g5qzp"
Oct 30 13:30:00.513509 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.513478    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gzs8\" (UniqueName: \"kubernetes.io/projected/17dc6af9-2a39-48fe-a3dc-405f9f9f1ea0-kube-api-access-9gzs8\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-9c6033-29889450-589x5\" (UID: \"17dc6af9-2a39-48fe-a3dc-405f9f9f1ea0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9c6033-29889450-589x5"
Oct 30 13:30:00.513671 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.513572    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f08ebb94-561c-42b6-9430-3c1d78c5fc8a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-b29df4-29889450-v8kpn\" (UID: \"f08ebb94-561c-42b6-9430-3c1d78c5fc8a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b29df4-29889450-v8kpn"
Oct 30 13:30:00.513671 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.513628    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0239aaa-e1bf-4ad8-bf54-42920cc8009c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889450-4glnd\" (UID: \"a0239aaa-e1bf-4ad8-bf54-42920cc8009c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889450-4glnd"
Oct 30 13:30:00.513815 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.513712    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff661c30-4a5a-4a98-b640-3b2c3e4143e1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-d833bd-29889450-q6468\" (UID: \"ff661c30-4a5a-4a98-b640-3b2c3e4143e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d833bd-29889450-q6468"
Oct 30 13:30:00.513815 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.513748    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fa046ef-c365-47a1-976f-18f12312ea4e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-4b0ea9-29889450-dc8t7\" (UID: \"0fa046ef-c365-47a1-976f-18f12312ea4e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4b0ea9-29889450-dc8t7"
Oct 30 13:30:00.513924 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.513815    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17dc6af9-2a39-48fe-a3dc-405f9f9f1ea0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-9c6033-29889450-589x5\" (UID: \"17dc6af9-2a39-48fe-a3dc-405f9f9f1ea0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9c6033-29889450-589x5"
Oct 30 13:30:00.513924 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.513844    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bj4f\" (UniqueName: \"kubernetes.io/projected/0fa046ef-c365-47a1-976f-18f12312ea4e-kube-api-access-4bj4f\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-4b0ea9-29889450-dc8t7\" (UID: \"0fa046ef-c365-47a1-976f-18f12312ea4e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4b0ea9-29889450-dc8t7"
Oct 30 13:30:00.514035 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.513986    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7ktw\" (UniqueName: \"kubernetes.io/projected/f08ebb94-561c-42b6-9430-3c1d78c5fc8a-kube-api-access-j7ktw\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-b29df4-29889450-v8kpn\" (UID: \"f08ebb94-561c-42b6-9430-3c1d78c5fc8a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b29df4-29889450-v8kpn"
Oct 30 13:30:00.514151 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.514132    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djtmf\" (UniqueName: \"kubernetes.io/projected/a0239aaa-e1bf-4ad8-bf54-42920cc8009c-kube-api-access-djtmf\") pod \"maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889450-4glnd\" (UID: \"a0239aaa-e1bf-4ad8-bf54-42920cc8009c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889450-4glnd"
Oct 30 13:30:00.514230 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.514212    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tq87\" (UniqueName: \"kubernetes.io/projected/ff661c30-4a5a-4a98-b640-3b2c3e4143e1-kube-api-access-6tq87\") pod \"maas-api-key-cleanup-e2e-worker-w3-d833bd-29889450-q6468\" (UID: \"ff661c30-4a5a-4a98-b640-3b2c3e4143e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d833bd-29889450-q6468"
Oct 30 13:30:00.514425 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.514402    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f08ebb94-561c-42b6-9430-3c1d78c5fc8a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-b29df4-29889450-v8kpn\" (UID: \"f08ebb94-561c-42b6-9430-3c1d78c5fc8a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b29df4-29889450-v8kpn"
Oct 30 13:30:00.514634 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.514584    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff661c30-4a5a-4a98-b640-3b2c3e4143e1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-d833bd-29889450-q6468\" (UID: \"ff661c30-4a5a-4a98-b640-3b2c3e4143e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d833bd-29889450-q6468"
Oct 30 13:30:00.515088 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.515066    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0239aaa-e1bf-4ad8-bf54-42920cc8009c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889450-4glnd\" (UID: \"a0239aaa-e1bf-4ad8-bf54-42920cc8009c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889450-4glnd"
Oct 30 13:30:00.518970 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.518949    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc"]
Oct 30 13:30:00.526930 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.526906    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc"
Oct 30 13:30:00.540738 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.540707    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7ktw\" (UniqueName: \"kubernetes.io/projected/f08ebb94-561c-42b6-9430-3c1d78c5fc8a-kube-api-access-j7ktw\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-b29df4-29889450-v8kpn\" (UID: \"f08ebb94-561c-42b6-9430-3c1d78c5fc8a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b29df4-29889450-v8kpn"
Oct 30 13:30:00.541071 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.541011    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tq87\" (UniqueName: \"kubernetes.io/projected/ff661c30-4a5a-4a98-b640-3b2c3e4143e1-kube-api-access-6tq87\") pod \"maas-api-key-cleanup-e2e-worker-w3-d833bd-29889450-q6468\" (UID: \"ff661c30-4a5a-4a98-b640-3b2c3e4143e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d833bd-29889450-q6468"
Oct 30 13:30:00.560622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.558264    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djtmf\" (UniqueName: \"kubernetes.io/projected/a0239aaa-e1bf-4ad8-bf54-42920cc8009c-kube-api-access-djtmf\") pod \"maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889450-4glnd\" (UID: \"a0239aaa-e1bf-4ad8-bf54-42920cc8009c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889450-4glnd"
Oct 30 13:30:00.569512 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.569275    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc"]
Oct 30 13:30:00.570896 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.570868    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d833bd-29889450-q6468"
Oct 30 13:30:00.611640 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.611592    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-82796d-openshift-default-dockercfg-rtjbw\""
Oct 30 13:30:00.613108 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.613084    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b29df4-29889450-v8kpn"
Oct 30 13:30:00.615428 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.615400    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d675cfa9-7df1-446a-984d-a4e1b0dda352-istio-token\") pod \"e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc\" (UID: \"d675cfa9-7df1-446a-984d-a4e1b0dda352\") " pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc"
Oct 30 13:30:00.615539 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.615497    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d675cfa9-7df1-446a-984d-a4e1b0dda352-istio-data\") pod \"e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc\" (UID: \"d675cfa9-7df1-446a-984d-a4e1b0dda352\") " pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc"
Oct 30 13:30:00.615539 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.615534    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d675cfa9-7df1-446a-984d-a4e1b0dda352-istio-podinfo\") pod \"e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc\" (UID: \"d675cfa9-7df1-446a-984d-a4e1b0dda352\") " pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc"
Oct 30 13:30:00.615696 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.615562    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d675cfa9-7df1-446a-984d-a4e1b0dda352-istiod-ca-cert\") pod \"e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc\" (UID: \"d675cfa9-7df1-446a-984d-a4e1b0dda352\") " pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc"
Oct 30 13:30:00.615696 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.615640    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d675cfa9-7df1-446a-984d-a4e1b0dda352-workload-socket\") pod \"e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc\" (UID: \"d675cfa9-7df1-446a-984d-a4e1b0dda352\") " pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc"
Oct 30 13:30:00.618982 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.618957    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889450-4glnd"
Oct 30 13:30:00.620662 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.615670    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gzs8\" (UniqueName: \"kubernetes.io/projected/17dc6af9-2a39-48fe-a3dc-405f9f9f1ea0-kube-api-access-9gzs8\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-9c6033-29889450-589x5\" (UID: \"17dc6af9-2a39-48fe-a3dc-405f9f9f1ea0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9c6033-29889450-589x5"
Oct 30 13:30:00.620782 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.620759    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d675cfa9-7df1-446a-984d-a4e1b0dda352-workload-certs\") pod \"e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc\" (UID: \"d675cfa9-7df1-446a-984d-a4e1b0dda352\") " pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc"
Oct 30 13:30:00.709628 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.705951    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d675cfa9-7df1-446a-984d-a4e1b0dda352-istio-envoy\") pod \"e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc\" (UID: \"d675cfa9-7df1-446a-984d-a4e1b0dda352\") " pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc"
Oct 30 13:30:00.709628 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.706043    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8v6l\" (UniqueName: \"kubernetes.io/projected/d675cfa9-7df1-446a-984d-a4e1b0dda352-kube-api-access-z8v6l\") pod \"e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc\" (UID: \"d675cfa9-7df1-446a-984d-a4e1b0dda352\") " pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc"
Oct 30 13:30:00.709628 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.706132    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fa046ef-c365-47a1-976f-18f12312ea4e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-4b0ea9-29889450-dc8t7\" (UID: \"0fa046ef-c365-47a1-976f-18f12312ea4e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4b0ea9-29889450-dc8t7"
Oct 30 13:30:00.709628 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.706323    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17dc6af9-2a39-48fe-a3dc-405f9f9f1ea0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-9c6033-29889450-589x5\" (UID: \"17dc6af9-2a39-48fe-a3dc-405f9f9f1ea0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9c6033-29889450-589x5"
Oct 30 13:30:00.709628 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.706363    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bj4f\" (UniqueName: \"kubernetes.io/projected/0fa046ef-c365-47a1-976f-18f12312ea4e-kube-api-access-4bj4f\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-4b0ea9-29889450-dc8t7\" (UID: \"0fa046ef-c365-47a1-976f-18f12312ea4e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4b0ea9-29889450-dc8t7"
Oct 30 13:30:00.709628 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.706425    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d675cfa9-7df1-446a-984d-a4e1b0dda352-credential-socket\") pod \"e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc\" (UID: \"d675cfa9-7df1-446a-984d-a4e1b0dda352\") " pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc"
Oct 30 13:30:00.747624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.715469    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fa046ef-c365-47a1-976f-18f12312ea4e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-4b0ea9-29889450-dc8t7\" (UID: \"0fa046ef-c365-47a1-976f-18f12312ea4e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4b0ea9-29889450-dc8t7"
Oct 30 13:30:00.747624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.717210    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gzs8\" (UniqueName: \"kubernetes.io/projected/17dc6af9-2a39-48fe-a3dc-405f9f9f1ea0-kube-api-access-9gzs8\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-9c6033-29889450-589x5\" (UID: \"17dc6af9-2a39-48fe-a3dc-405f9f9f1ea0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9c6033-29889450-589x5"
Oct 30 13:30:00.876630 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.862779    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d675cfa9-7df1-446a-984d-a4e1b0dda352-istio-data\") pod \"e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc\" (UID: \"d675cfa9-7df1-446a-984d-a4e1b0dda352\") " pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc"
Oct 30 13:30:00.876630 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.862847    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d675cfa9-7df1-446a-984d-a4e1b0dda352-istio-podinfo\") pod \"e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc\" (UID: \"d675cfa9-7df1-446a-984d-a4e1b0dda352\") " pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc"
Oct 30 13:30:00.876630 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.862887    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d675cfa9-7df1-446a-984d-a4e1b0dda352-istiod-ca-cert\") pod \"e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc\" (UID: \"d675cfa9-7df1-446a-984d-a4e1b0dda352\") " pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc"
Oct 30 13:30:00.876630 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.862997    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d675cfa9-7df1-446a-984d-a4e1b0dda352-workload-socket\") pod \"e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc\" (UID: \"d675cfa9-7df1-446a-984d-a4e1b0dda352\") " pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc"
Oct 30 13:30:00.876630 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.863133    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d675cfa9-7df1-446a-984d-a4e1b0dda352-workload-certs\") pod \"e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc\" (UID: \"d675cfa9-7df1-446a-984d-a4e1b0dda352\") " pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc"
Oct 30 13:30:00.876630 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.863177    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d675cfa9-7df1-446a-984d-a4e1b0dda352-istio-envoy\") pod \"e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc\" (UID: \"d675cfa9-7df1-446a-984d-a4e1b0dda352\") " pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc"
Oct 30 13:30:00.876630 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.863212    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8v6l\" (UniqueName: \"kubernetes.io/projected/d675cfa9-7df1-446a-984d-a4e1b0dda352-kube-api-access-z8v6l\") pod \"e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc\" (UID: \"d675cfa9-7df1-446a-984d-a4e1b0dda352\") " pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc"
Oct 30 13:30:00.876630 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.863356    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d675cfa9-7df1-446a-984d-a4e1b0dda352-credential-socket\") pod \"e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc\" (UID: \"d675cfa9-7df1-446a-984d-a4e1b0dda352\") " pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc"
Oct 30 13:30:00.876630 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.863469    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d675cfa9-7df1-446a-984d-a4e1b0dda352-istio-token\") pod \"e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc\" (UID: \"d675cfa9-7df1-446a-984d-a4e1b0dda352\") " pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc"
Oct 30 13:30:00.876630 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.876129    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d675cfa9-7df1-446a-984d-a4e1b0dda352-workload-socket\") pod \"e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc\" (UID: \"d675cfa9-7df1-446a-984d-a4e1b0dda352\") " pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc"
Oct 30 13:30:00.916717 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.893429    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d675cfa9-7df1-446a-984d-a4e1b0dda352-istio-data\") pod \"e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc\" (UID: \"d675cfa9-7df1-446a-984d-a4e1b0dda352\") " pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc"
Oct 30 13:30:00.916717 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.903438    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d675cfa9-7df1-446a-984d-a4e1b0dda352-istiod-ca-cert\") pod \"e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc\" (UID: \"d675cfa9-7df1-446a-984d-a4e1b0dda352\") " pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc"
Oct 30 13:30:00.916717 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.907254    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d675cfa9-7df1-446a-984d-a4e1b0dda352-workload-certs\") pod \"e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc\" (UID: \"d675cfa9-7df1-446a-984d-a4e1b0dda352\") " pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc"
Oct 30 13:30:00.916717 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.909003    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d675cfa9-7df1-446a-984d-a4e1b0dda352-credential-socket\") pod \"e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc\" (UID: \"d675cfa9-7df1-446a-984d-a4e1b0dda352\") " pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc"
Oct 30 13:30:00.976743 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.950143    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17dc6af9-2a39-48fe-a3dc-405f9f9f1ea0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-9c6033-29889450-589x5\" (UID: \"17dc6af9-2a39-48fe-a3dc-405f9f9f1ea0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9c6033-29889450-589x5"
Oct 30 13:30:00.996651 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.978042    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9c6033-29889450-589x5"
Oct 30 13:30:00.996651 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.991075    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d675cfa9-7df1-446a-984d-a4e1b0dda352-istio-envoy\") pod \"e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc\" (UID: \"d675cfa9-7df1-446a-984d-a4e1b0dda352\") " pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc"
Oct 30 13:30:00.996651 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.991477    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8v6l\" (UniqueName: \"kubernetes.io/projected/d675cfa9-7df1-446a-984d-a4e1b0dda352-kube-api-access-z8v6l\") pod \"e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc\" (UID: \"d675cfa9-7df1-446a-984d-a4e1b0dda352\") " pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc"
Oct 30 13:30:00.999480 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:00.999418    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d675cfa9-7df1-446a-984d-a4e1b0dda352-istio-podinfo\") pod \"e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc\" (UID: \"d675cfa9-7df1-446a-984d-a4e1b0dda352\") " pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc"
Oct 30 13:30:01.034928 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:01.034900    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d675cfa9-7df1-446a-984d-a4e1b0dda352-istio-token\") pod \"e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc\" (UID: \"d675cfa9-7df1-446a-984d-a4e1b0dda352\") " pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc"
Oct 30 13:30:01.036103 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:01.036072    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bj4f\" (UniqueName: \"kubernetes.io/projected/0fa046ef-c365-47a1-976f-18f12312ea4e-kube-api-access-4bj4f\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-4b0ea9-29889450-dc8t7\" (UID: \"0fa046ef-c365-47a1-976f-18f12312ea4e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4b0ea9-29889450-dc8t7"
Oct 30 13:30:01.198895 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:01.198862    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc"
Oct 30 13:30:01.240461 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:01.238427    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-cda0d9-29889450-bnh6t"]
Oct 30 13:30:01.244282 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:01.243764    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4b0ea9-29889450-dc8t7"
Oct 30 13:30:01.244282 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:30:01.243960    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd10daaa5_7cac_4205_8638_75ede670c28f.slice/crio-f5e7aa9c6240024332ea38fb51184a838e590eb301cca694c6def3523b9323f4 WatchSource:0}: Error finding container f5e7aa9c6240024332ea38fb51184a838e590eb301cca694c6def3523b9323f4: Status 404 returned error can't find the container with id f5e7aa9c6240024332ea38fb51184a838e590eb301cca694c6def3523b9323f4
Oct 30 13:30:01.257624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:01.257580    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8d82d5-29889450-g5qzp"]
Oct 30 13:30:01.259723 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:30:01.259683    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podace885ec_090f_469d_ba45_23bd11e84d9e.slice/crio-088961120cc2b696b2aedf712b6a423d003c1eff8fd3465428c216eeea352226 WatchSource:0}: Error finding container 088961120cc2b696b2aedf712b6a423d003c1eff8fd3465428c216eeea352226: Status 404 returned error can't find the container with id 088961120cc2b696b2aedf712b6a423d003c1eff8fd3465428c216eeea352226
Oct 30 13:30:01.271644 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:01.271362    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d833bd-29889450-q6468"]
Oct 30 13:30:01.276092 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:30:01.276044    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff661c30_4a5a_4a98_b640_3b2c3e4143e1.slice/crio-d2b6bd9ddc2c1ca900f19b77e606d47800c2d9d39d01dedaee2a57bd4fc52d48 WatchSource:0}: Error finding container d2b6bd9ddc2c1ca900f19b77e606d47800c2d9d39d01dedaee2a57bd4fc52d48: Status 404 returned error can't find the container with id d2b6bd9ddc2c1ca900f19b77e606d47800c2d9d39d01dedaee2a57bd4fc52d48
Oct 30 13:30:01.400640 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:01.394612    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw"]
Oct 30 13:30:01.400640 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:01.394941    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw" podUID="617447eb-2414-439d-9923-c47b809de841" containerName="istio-proxy" containerID="cri-o://1925ed105ee025e7459d40dead0c738808cda8573e6e0cf8bd847f8615448e50" gracePeriod=30
Oct 30 13:30:01.432925 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:01.432890    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b29df4-29889450-v8kpn"]
Oct 30 13:30:01.437927 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:01.437887    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9c6033-29889450-589x5"]
Oct 30 13:30:01.445422 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:01.445398    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889450-4glnd"]
Oct 30 13:30:01.783611 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:01.783450    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc"]
Oct 30 13:30:01.789279 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:01.788058    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8d82d5-29889450-g5qzp" event={"ID":"ace885ec-090f-469d-ba45-23bd11e84d9e","Type":"ContainerStarted","Data":"088961120cc2b696b2aedf712b6a423d003c1eff8fd3465428c216eeea352226"}
Oct 30 13:30:01.792028 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:01.791998    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d833bd-29889450-q6468" event={"ID":"ff661c30-4a5a-4a98-b640-3b2c3e4143e1","Type":"ContainerStarted","Data":"d2b6bd9ddc2c1ca900f19b77e606d47800c2d9d39d01dedaee2a57bd4fc52d48"}
Oct 30 13:30:01.793675 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:01.793652    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4b0ea9-29889450-dc8t7"]
Oct 30 13:30:01.794311 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:01.794286    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-cda0d9-29889450-bnh6t" event={"ID":"d10daaa5-7cac-4205-8638-75ede670c28f","Type":"ContainerStarted","Data":"f5e7aa9c6240024332ea38fb51184a838e590eb301cca694c6def3523b9323f4"}
Oct 30 13:30:01.799845 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:01.796289    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b29df4-29889450-v8kpn" event={"ID":"f08ebb94-561c-42b6-9430-3c1d78c5fc8a","Type":"ContainerStarted","Data":"95476c452530334f30c8bad1c67a85c1f5aafbb727eef631d779d8ac5094a87b"}
Oct 30 13:30:01.817355 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:01.817302    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:30:01.817622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:01.817577    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:30:01.817789 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:01.817770    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:30:02.927109 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:02.927068    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889450-4glnd" event={"ID":"a0239aaa-e1bf-4ad8-bf54-42920cc8009c","Type":"ContainerStarted","Data":"6476e2c17a17ce24426f052257fbf2c9e90df79f5c6e74f1ab9a1060c2297ee9"}
Oct 30 13:30:02.931715 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:02.931678    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889450-4glnd" event={"ID":"a0239aaa-e1bf-4ad8-bf54-42920cc8009c","Type":"ContainerStarted","Data":"9e12efbcf3591ac6dadce7e9d567d036f7bfb1346acb608abc9700c9500eb5a6"}
Oct 30 13:30:03.064627 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:03.064321    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889450-4glnd" podStartSLOduration=3.06429931 podStartE2EDuration="3.06429931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:03.041672553 +0000 UTC m=+2302.208037201" watchObservedRunningTime="2026-10-30 13:30:03.06429931 +0000 UTC m=+2302.230663959"
Oct 30 13:30:03.144669 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:03.139041    4364 generic.go:356] "Generic (PLEG): container finished" podID="ace885ec-090f-469d-ba45-23bd11e84d9e" containerID="ca59361330a46e8926b7e8c846ce60259194e290f700f523767ab9ecf579d36c" exitCode=60
Oct 30 13:30:03.144669 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:03.139796    4364 scope.go:122] "RemoveContainer" containerID="ca59361330a46e8926b7e8c846ce60259194e290f700f523767ab9ecf579d36c"
Oct 30 13:30:03.144669 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:03.143159    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc" event={"ID":"d675cfa9-7df1-446a-984d-a4e1b0dda352","Type":"ContainerStarted","Data":"42c7c544365e5d4fb53bee70de1e7283d4ce4c0a13e7c83c2f88f1c1ce4c966f"}
Oct 30 13:30:03.144669 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:03.143385    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc" event={"ID":"d675cfa9-7df1-446a-984d-a4e1b0dda352","Type":"ContainerStarted","Data":"a83f574dc10025974ea934d10547ed491ee0da27a2e3ce8875054659f7184099"}
Oct 30 13:30:03.144669 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:03.143454    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8d82d5-29889450-g5qzp" event={"ID":"ace885ec-090f-469d-ba45-23bd11e84d9e","Type":"ContainerDied","Data":"ca59361330a46e8926b7e8c846ce60259194e290f700f523767ab9ecf579d36c"}
Oct 30 13:30:03.144669 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:03.143496    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4b0ea9-29889450-dc8t7" event={"ID":"0fa046ef-c365-47a1-976f-18f12312ea4e","Type":"ContainerStarted","Data":"beb3ea7db97be209f890a13a51d0dbe0dc4c0a8e4555ba982b7bb362fe6f5663"}
Oct 30 13:30:03.144669 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:03.143543    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4b0ea9-29889450-dc8t7" event={"ID":"0fa046ef-c365-47a1-976f-18f12312ea4e","Type":"ContainerStarted","Data":"1c7ae317a6f8aabda635e606d0d7b1a70b7bbabb64698e81a0aefc280dc5eb2c"}
Oct 30 13:30:03.146503 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:03.146472    4364 generic.go:356] "Generic (PLEG): container finished" podID="ff661c30-4a5a-4a98-b640-3b2c3e4143e1" containerID="f30160b4f117edfd1245897941677cddaa862a705c9c68d0ef719c7ed833a2b5" exitCode=60
Oct 30 13:30:03.147011 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:03.146776    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d833bd-29889450-q6468" event={"ID":"ff661c30-4a5a-4a98-b640-3b2c3e4143e1","Type":"ContainerDied","Data":"f30160b4f117edfd1245897941677cddaa862a705c9c68d0ef719c7ed833a2b5"}
Oct 30 13:30:03.147169 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:03.147141    4364 scope.go:122] "RemoveContainer" containerID="f30160b4f117edfd1245897941677cddaa862a705c9c68d0ef719c7ed833a2b5"
Oct 30 13:30:03.155713 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:03.155393    4364 generic.go:356] "Generic (PLEG): container finished" podID="d10daaa5-7cac-4205-8638-75ede670c28f" containerID="e87fd66ec6587a5914cb93cdd77291802e5146025a7f8b4a080aea72e91712f8" exitCode=60
Oct 30 13:30:03.155713 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:03.155519    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-cda0d9-29889450-bnh6t" event={"ID":"d10daaa5-7cac-4205-8638-75ede670c28f","Type":"ContainerDied","Data":"e87fd66ec6587a5914cb93cdd77291802e5146025a7f8b4a080aea72e91712f8"}
Oct 30 13:30:03.156635 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:03.156277    4364 scope.go:122] "RemoveContainer" containerID="e87fd66ec6587a5914cb93cdd77291802e5146025a7f8b4a080aea72e91712f8"
Oct 30 13:30:03.186700 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:03.179183    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc" podStartSLOduration=3.179157902 podStartE2EDuration="3.179157902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:03.157845609 +0000 UTC m=+2302.324210258" watchObservedRunningTime="2026-10-30 13:30:03.179157902 +0000 UTC m=+2302.345522551"
Oct 30 13:30:03.199791 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:03.199755    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc"
Oct 30 13:30:03.207492 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:03.207401    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc"
Oct 30 13:30:03.215037 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:03.215008    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b29df4-29889450-v8kpn" event={"ID":"f08ebb94-561c-42b6-9430-3c1d78c5fc8a","Type":"ContainerStarted","Data":"ddd29077e8a2b248295d268a81c5d80b48d8526f406c224c594fedf5ebc980c3"}
Oct 30 13:30:03.215703 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:03.215673    4364 scope.go:122] "RemoveContainer" containerID="ddd29077e8a2b248295d268a81c5d80b48d8526f406c224c594fedf5ebc980c3"
Oct 30 13:30:03.218341 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:03.218316    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9c6033-29889450-589x5" event={"ID":"17dc6af9-2a39-48fe-a3dc-405f9f9f1ea0","Type":"ContainerStarted","Data":"a1b0f6615da9d7bb22eff35b560f07d3973ca09b54bc66ff535561f4e8c1c5dd"}
Oct 30 13:30:03.218508 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:03.218488    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9c6033-29889450-589x5" event={"ID":"17dc6af9-2a39-48fe-a3dc-405f9f9f1ea0","Type":"ContainerStarted","Data":"975104ee45a5c5f0698fa4752a14361748e13305a35a4f846061a046d608f50f"}
Oct 30 13:30:03.218689 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:03.218648    4364 scope.go:122] "RemoveContainer" containerID="a1b0f6615da9d7bb22eff35b560f07d3973ca09b54bc66ff535561f4e8c1c5dd"
Oct 30 13:30:03.237116 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:03.237059    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4b0ea9-29889450-dc8t7" podStartSLOduration=3.237038981 podStartE2EDuration="3.237038981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:03.235921485 +0000 UTC m=+2302.402286134" watchObservedRunningTime="2026-10-30 13:30:03.237038981 +0000 UTC m=+2302.403403631"
Oct 30 13:30:04.230335 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:04.230236    4364 generic.go:356] "Generic (PLEG): container finished" podID="0fa046ef-c365-47a1-976f-18f12312ea4e" containerID="beb3ea7db97be209f890a13a51d0dbe0dc4c0a8e4555ba982b7bb362fe6f5663" exitCode=60
Oct 30 13:30:04.230783 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:04.230324    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4b0ea9-29889450-dc8t7" event={"ID":"0fa046ef-c365-47a1-976f-18f12312ea4e","Type":"ContainerDied","Data":"beb3ea7db97be209f890a13a51d0dbe0dc4c0a8e4555ba982b7bb362fe6f5663"}
Oct 30 13:30:04.230783 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:04.230680    4364 scope.go:122] "RemoveContainer" containerID="beb3ea7db97be209f890a13a51d0dbe0dc4c0a8e4555ba982b7bb362fe6f5663"
Oct 30 13:30:04.232706 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:04.232554    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d833bd-29889450-q6468" event={"ID":"ff661c30-4a5a-4a98-b640-3b2c3e4143e1","Type":"ContainerStarted","Data":"dc73527c53e98c7dbbe11472b0bb89863e309682c54992bbbe17416f6e2a54cb"}
Oct 30 13:30:04.234665 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:04.234643    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-cda0d9-29889450-bnh6t" event={"ID":"d10daaa5-7cac-4205-8638-75ede670c28f","Type":"ContainerStarted","Data":"5ea480a392a7f2a1514b8a2c58f256407b7b8febac3682a0015337d0161f4fca"}
Oct 30 13:30:04.236770 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:04.236747    4364 generic.go:356] "Generic (PLEG): container finished" podID="f08ebb94-561c-42b6-9430-3c1d78c5fc8a" containerID="ddd29077e8a2b248295d268a81c5d80b48d8526f406c224c594fedf5ebc980c3" exitCode=60
Oct 30 13:30:04.236873 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:04.236831    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b29df4-29889450-v8kpn" event={"ID":"f08ebb94-561c-42b6-9430-3c1d78c5fc8a","Type":"ContainerDied","Data":"ddd29077e8a2b248295d268a81c5d80b48d8526f406c224c594fedf5ebc980c3"}
Oct 30 13:30:04.236951 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:04.236881    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b29df4-29889450-v8kpn" event={"ID":"f08ebb94-561c-42b6-9430-3c1d78c5fc8a","Type":"ContainerStarted","Data":"2c5af3f868c7a0399ffec13701c692393a6bccd56c36564e8a5643010a029fa5"}
Oct 30 13:30:04.238995 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:04.238973    4364 generic.go:356] "Generic (PLEG): container finished" podID="17dc6af9-2a39-48fe-a3dc-405f9f9f1ea0" containerID="a1b0f6615da9d7bb22eff35b560f07d3973ca09b54bc66ff535561f4e8c1c5dd" exitCode=60
Oct 30 13:30:04.239106 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:04.239023    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9c6033-29889450-589x5" event={"ID":"17dc6af9-2a39-48fe-a3dc-405f9f9f1ea0","Type":"ContainerDied","Data":"a1b0f6615da9d7bb22eff35b560f07d3973ca09b54bc66ff535561f4e8c1c5dd"}
Oct 30 13:30:04.239106 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:04.239091    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9c6033-29889450-589x5" event={"ID":"17dc6af9-2a39-48fe-a3dc-405f9f9f1ea0","Type":"ContainerStarted","Data":"bc5575d7b544564d427cbbab12a6c6ce56a64b7b5c0c135b67da0ff70f93860e"}
Oct 30 13:30:04.240789 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:04.240770    4364 generic.go:356] "Generic (PLEG): container finished" podID="a0239aaa-e1bf-4ad8-bf54-42920cc8009c" containerID="6476e2c17a17ce24426f052257fbf2c9e90df79f5c6e74f1ab9a1060c2297ee9" exitCode=60
Oct 30 13:30:04.240884 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:04.240834    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889450-4glnd" event={"ID":"a0239aaa-e1bf-4ad8-bf54-42920cc8009c","Type":"ContainerDied","Data":"6476e2c17a17ce24426f052257fbf2c9e90df79f5c6e74f1ab9a1060c2297ee9"}
Oct 30 13:30:04.241039 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:04.241023    4364 scope.go:122] "RemoveContainer" containerID="6476e2c17a17ce24426f052257fbf2c9e90df79f5c6e74f1ab9a1060c2297ee9"
Oct 30 13:30:04.242924 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:04.242899    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8d82d5-29889450-g5qzp" event={"ID":"ace885ec-090f-469d-ba45-23bd11e84d9e","Type":"ContainerStarted","Data":"6a7d7928484eae9643ceeffd1733851a316acb27aacab15d173dc7250c9b35d1"}
Oct 30 13:30:04.243084 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:04.243069    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc"
Oct 30 13:30:04.244363 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:04.244348    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc"
Oct 30 13:30:04.322168 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:04.322115    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8d82d5-29889450-g5qzp" podStartSLOduration=4.322099079 podStartE2EDuration="4.322099079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:04.318942719 +0000 UTC m=+2303.485307366" watchObservedRunningTime="2026-10-30 13:30:04.322099079 +0000 UTC m=+2303.488463726"
Oct 30 13:30:04.423998 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:04.423941    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-cda0d9-29889450-bnh6t" podStartSLOduration=4.423920049 podStartE2EDuration="4.423920049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:04.409100885 +0000 UTC m=+2303.575465567" watchObservedRunningTime="2026-10-30 13:30:04.423920049 +0000 UTC m=+2303.590284698"
Oct 30 13:30:04.527877 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:04.527823    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d833bd-29889450-q6468" podStartSLOduration=4.527807944 podStartE2EDuration="4.527807944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:04.523897847 +0000 UTC m=+2303.690262494" watchObservedRunningTime="2026-10-30 13:30:04.527807944 +0000 UTC m=+2303.694172592"
Oct 30 13:30:04.570651 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:04.570574    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b29df4-29889450-v8kpn" podStartSLOduration=4.570556903 podStartE2EDuration="4.570556903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:04.567542968 +0000 UTC m=+2303.733907616" watchObservedRunningTime="2026-10-30 13:30:04.570556903 +0000 UTC m=+2303.736921552"
Oct 30 13:30:04.612926 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:04.612866    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9c6033-29889450-589x5" podStartSLOduration=4.612847585 podStartE2EDuration="4.612847585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:04.607897273 +0000 UTC m=+2303.774261921" watchObservedRunningTime="2026-10-30 13:30:04.612847585 +0000 UTC m=+2303.779212233"
Oct 30 13:30:05.254319 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:05.254284    4364 generic.go:356] "Generic (PLEG): container finished" podID="d10daaa5-7cac-4205-8638-75ede670c28f" containerID="5ea480a392a7f2a1514b8a2c58f256407b7b8febac3682a0015337d0161f4fca" exitCode=60
Oct 30 13:30:05.254857 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:05.254363    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-cda0d9-29889450-bnh6t" event={"ID":"d10daaa5-7cac-4205-8638-75ede670c28f","Type":"ContainerDied","Data":"5ea480a392a7f2a1514b8a2c58f256407b7b8febac3682a0015337d0161f4fca"}
Oct 30 13:30:05.254857 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:05.254430    4364 scope.go:122] "RemoveContainer" containerID="e87fd66ec6587a5914cb93cdd77291802e5146025a7f8b4a080aea72e91712f8"
Oct 30 13:30:05.254857 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:05.254813    4364 scope.go:122] "RemoveContainer" containerID="5ea480a392a7f2a1514b8a2c58f256407b7b8febac3682a0015337d0161f4fca"
Oct 30 13:30:05.255239 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:30:05.255211    4364 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-cda0d9-29889450-bnh6t_odh-ai-gateway-infra(d10daaa5-7cac-4205-8638-75ede670c28f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-cda0d9-29889450-bnh6t" podUID="d10daaa5-7cac-4205-8638-75ede670c28f"
Oct 30 13:30:05.256884 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:05.256863    4364 generic.go:356] "Generic (PLEG): container finished" podID="f08ebb94-561c-42b6-9430-3c1d78c5fc8a" containerID="2c5af3f868c7a0399ffec13701c692393a6bccd56c36564e8a5643010a029fa5" exitCode=60
Oct 30 13:30:05.256987 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:05.256952    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b29df4-29889450-v8kpn" event={"ID":"f08ebb94-561c-42b6-9430-3c1d78c5fc8a","Type":"ContainerDied","Data":"2c5af3f868c7a0399ffec13701c692393a6bccd56c36564e8a5643010a029fa5"}
Oct 30 13:30:05.257257 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:05.257240    4364 scope.go:122] "RemoveContainer" containerID="2c5af3f868c7a0399ffec13701c692393a6bccd56c36564e8a5643010a029fa5"
Oct 30 13:30:05.257666 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:30:05.257643    4364 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-b29df4-29889450-v8kpn_odh-ai-gateway-infra(f08ebb94-561c-42b6-9430-3c1d78c5fc8a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b29df4-29889450-v8kpn" podUID="f08ebb94-561c-42b6-9430-3c1d78c5fc8a"
Oct 30 13:30:05.258998 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:05.258977    4364 generic.go:356] "Generic (PLEG): container finished" podID="17dc6af9-2a39-48fe-a3dc-405f9f9f1ea0" containerID="bc5575d7b544564d427cbbab12a6c6ce56a64b7b5c0c135b67da0ff70f93860e" exitCode=60
Oct 30 13:30:05.259098 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:05.259057    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9c6033-29889450-589x5" event={"ID":"17dc6af9-2a39-48fe-a3dc-405f9f9f1ea0","Type":"ContainerDied","Data":"bc5575d7b544564d427cbbab12a6c6ce56a64b7b5c0c135b67da0ff70f93860e"}
Oct 30 13:30:05.259676 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:05.259657    4364 scope.go:122] "RemoveContainer" containerID="bc5575d7b544564d427cbbab12a6c6ce56a64b7b5c0c135b67da0ff70f93860e"
Oct 30 13:30:05.260011 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:30:05.259992    4364 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-9c6033-29889450-589x5_odh-ai-gateway-infra(17dc6af9-2a39-48fe-a3dc-405f9f9f1ea0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9c6033-29889450-589x5" podUID="17dc6af9-2a39-48fe-a3dc-405f9f9f1ea0"
Oct 30 13:30:05.261138 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:05.261114    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889450-4glnd" event={"ID":"a0239aaa-e1bf-4ad8-bf54-42920cc8009c","Type":"ContainerStarted","Data":"99323dabcbc22c4a496ea6053bdb9274cd33134030599970cfca7731a3a31282"}
Oct 30 13:30:05.263119 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:05.263096    4364 generic.go:356] "Generic (PLEG): container finished" podID="ace885ec-090f-469d-ba45-23bd11e84d9e" containerID="6a7d7928484eae9643ceeffd1733851a316acb27aacab15d173dc7250c9b35d1" exitCode=60
Oct 30 13:30:05.263340 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:05.263314    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8d82d5-29889450-g5qzp" event={"ID":"ace885ec-090f-469d-ba45-23bd11e84d9e","Type":"ContainerDied","Data":"6a7d7928484eae9643ceeffd1733851a316acb27aacab15d173dc7250c9b35d1"}
Oct 30 13:30:05.263461 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:05.263445    4364 scope.go:122] "RemoveContainer" containerID="6a7d7928484eae9643ceeffd1733851a316acb27aacab15d173dc7250c9b35d1"
Oct 30 13:30:05.263817 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:30:05.263794    4364 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-8d82d5-29889450-g5qzp_odh-ai-gateway-infra(ace885ec-090f-469d-ba45-23bd11e84d9e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8d82d5-29889450-g5qzp" podUID="ace885ec-090f-469d-ba45-23bd11e84d9e"
Oct 30 13:30:05.265292 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:05.265264    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4b0ea9-29889450-dc8t7" event={"ID":"0fa046ef-c365-47a1-976f-18f12312ea4e","Type":"ContainerStarted","Data":"8ab1de124a77ef88afaafd81cb0d8170abeab194aa2451ea2a20c698de823a6e"}
Oct 30 13:30:05.267159 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:05.267135    4364 generic.go:356] "Generic (PLEG): container finished" podID="ff661c30-4a5a-4a98-b640-3b2c3e4143e1" containerID="dc73527c53e98c7dbbe11472b0bb89863e309682c54992bbbe17416f6e2a54cb" exitCode=60
Oct 30 13:30:05.267245 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:05.267164    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d833bd-29889450-q6468" event={"ID":"ff661c30-4a5a-4a98-b640-3b2c3e4143e1","Type":"ContainerDied","Data":"dc73527c53e98c7dbbe11472b0bb89863e309682c54992bbbe17416f6e2a54cb"}
Oct 30 13:30:05.267460 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:05.267444    4364 scope.go:122] "RemoveContainer" containerID="dc73527c53e98c7dbbe11472b0bb89863e309682c54992bbbe17416f6e2a54cb"
Oct 30 13:30:05.267708 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:30:05.267691    4364 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-d833bd-29889450-q6468_odh-ai-gateway-infra(ff661c30-4a5a-4a98-b640-3b2c3e4143e1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d833bd-29889450-q6468" podUID="ff661c30-4a5a-4a98-b640-3b2c3e4143e1"
Oct 30 13:30:05.271214 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:05.271198    4364 scope.go:122] "RemoveContainer" containerID="ddd29077e8a2b248295d268a81c5d80b48d8526f406c224c594fedf5ebc980c3"
Oct 30 13:30:05.285671 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:05.285642    4364 scope.go:122] "RemoveContainer" containerID="a1b0f6615da9d7bb22eff35b560f07d3973ca09b54bc66ff535561f4e8c1c5dd"
Oct 30 13:30:05.310806 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:05.310062    4364 scope.go:122] "RemoveContainer" containerID="ca59361330a46e8926b7e8c846ce60259194e290f700f523767ab9ecf579d36c"
Oct 30 13:30:05.335876 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:05.335843    4364 scope.go:122] "RemoveContainer" containerID="f30160b4f117edfd1245897941677cddaa862a705c9c68d0ef719c7ed833a2b5"
Oct 30 13:30:06.278186 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:06.278159    4364 scope.go:122] "RemoveContainer" containerID="dc73527c53e98c7dbbe11472b0bb89863e309682c54992bbbe17416f6e2a54cb"
Oct 30 13:30:06.278810 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:30:06.278708    4364 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-d833bd-29889450-q6468_odh-ai-gateway-infra(ff661c30-4a5a-4a98-b640-3b2c3e4143e1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d833bd-29889450-q6468" podUID="ff661c30-4a5a-4a98-b640-3b2c3e4143e1"
Oct 30 13:30:06.280084 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:06.280059    4364 scope.go:122] "RemoveContainer" containerID="5ea480a392a7f2a1514b8a2c58f256407b7b8febac3682a0015337d0161f4fca"
Oct 30 13:30:06.280433 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:30:06.280407    4364 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-cda0d9-29889450-bnh6t_odh-ai-gateway-infra(d10daaa5-7cac-4205-8638-75ede670c28f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-cda0d9-29889450-bnh6t" podUID="d10daaa5-7cac-4205-8638-75ede670c28f"
Oct 30 13:30:06.281847 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:06.281815    4364 scope.go:122] "RemoveContainer" containerID="2c5af3f868c7a0399ffec13701c692393a6bccd56c36564e8a5643010a029fa5"
Oct 30 13:30:06.282174 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:30:06.282142    4364 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-b29df4-29889450-v8kpn_odh-ai-gateway-infra(f08ebb94-561c-42b6-9430-3c1d78c5fc8a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b29df4-29889450-v8kpn" podUID="f08ebb94-561c-42b6-9430-3c1d78c5fc8a"
Oct 30 13:30:06.283475 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:06.283457    4364 scope.go:122] "RemoveContainer" containerID="bc5575d7b544564d427cbbab12a6c6ce56a64b7b5c0c135b67da0ff70f93860e"
Oct 30 13:30:06.283805 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:30:06.283783    4364 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-9c6033-29889450-589x5_odh-ai-gateway-infra(17dc6af9-2a39-48fe-a3dc-405f9f9f1ea0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9c6033-29889450-589x5" podUID="17dc6af9-2a39-48fe-a3dc-405f9f9f1ea0"
Oct 30 13:30:06.285157 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:06.285139    4364 generic.go:356] "Generic (PLEG): container finished" podID="a0239aaa-e1bf-4ad8-bf54-42920cc8009c" containerID="99323dabcbc22c4a496ea6053bdb9274cd33134030599970cfca7731a3a31282" exitCode=60
Oct 30 13:30:06.285260 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:06.285220    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889450-4glnd" event={"ID":"a0239aaa-e1bf-4ad8-bf54-42920cc8009c","Type":"ContainerDied","Data":"99323dabcbc22c4a496ea6053bdb9274cd33134030599970cfca7731a3a31282"}
Oct 30 13:30:06.285318 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:06.285284    4364 scope.go:122] "RemoveContainer" containerID="6476e2c17a17ce24426f052257fbf2c9e90df79f5c6e74f1ab9a1060c2297ee9"
Oct 30 13:30:06.285457 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:06.285439    4364 scope.go:122] "RemoveContainer" containerID="99323dabcbc22c4a496ea6053bdb9274cd33134030599970cfca7731a3a31282"
Oct 30 13:30:06.285812 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:30:06.285787    4364 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889450-4glnd_odh-ai-gateway-infra(a0239aaa-e1bf-4ad8-bf54-42920cc8009c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889450-4glnd" podUID="a0239aaa-e1bf-4ad8-bf54-42920cc8009c"
Oct 30 13:30:06.287333 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:06.287312    4364 scope.go:122] "RemoveContainer" containerID="6a7d7928484eae9643ceeffd1733851a316acb27aacab15d173dc7250c9b35d1"
Oct 30 13:30:06.287705 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:30:06.287661    4364 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-8d82d5-29889450-g5qzp_odh-ai-gateway-infra(ace885ec-090f-469d-ba45-23bd11e84d9e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8d82d5-29889450-g5qzp" podUID="ace885ec-090f-469d-ba45-23bd11e84d9e"
Oct 30 13:30:06.289264 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:06.289198    4364 generic.go:356] "Generic (PLEG): container finished" podID="0fa046ef-c365-47a1-976f-18f12312ea4e" containerID="8ab1de124a77ef88afaafd81cb0d8170abeab194aa2451ea2a20c698de823a6e" exitCode=60
Oct 30 13:30:06.289264 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:06.289234    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4b0ea9-29889450-dc8t7" event={"ID":"0fa046ef-c365-47a1-976f-18f12312ea4e","Type":"ContainerDied","Data":"8ab1de124a77ef88afaafd81cb0d8170abeab194aa2451ea2a20c698de823a6e"}
Oct 30 13:30:06.289548 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:06.289526    4364 scope.go:122] "RemoveContainer" containerID="8ab1de124a77ef88afaafd81cb0d8170abeab194aa2451ea2a20c698de823a6e"
Oct 30 13:30:06.289837 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:30:06.289807    4364 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-4b0ea9-29889450-dc8t7_odh-ai-gateway-infra(0fa046ef-c365-47a1-976f-18f12312ea4e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4b0ea9-29889450-dc8t7" podUID="0fa046ef-c365-47a1-976f-18f12312ea4e"
Oct 30 13:30:06.301576 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:06.301555    4364 scope.go:122] "RemoveContainer" containerID="beb3ea7db97be209f890a13a51d0dbe0dc4c0a8e4555ba982b7bb362fe6f5663"
Oct 30 13:30:06.776453 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:06.776426    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw"
Oct 30 13:30:06.919160 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:06.919115    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/617447eb-2414-439d-9923-c47b809de841-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/617447eb-2414-439d-9923-c47b809de841-istio-podinfo\") pod \"617447eb-2414-439d-9923-c47b809de841\" (UID: \"617447eb-2414-439d-9923-c47b809de841\") "
Oct 30 13:30:06.919160 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:06.919176    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/617447eb-2414-439d-9923-c47b809de841-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/617447eb-2414-439d-9923-c47b809de841-istiod-ca-cert\") pod \"617447eb-2414-439d-9923-c47b809de841\" (UID: \"617447eb-2414-439d-9923-c47b809de841\") "
Oct 30 13:30:06.919446 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:06.919201    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/617447eb-2414-439d-9923-c47b809de841-kube-api-access-dlbp5\" (UniqueName: \"kubernetes.io/projected/617447eb-2414-439d-9923-c47b809de841-kube-api-access-dlbp5\") pod \"617447eb-2414-439d-9923-c47b809de841\" (UID: \"617447eb-2414-439d-9923-c47b809de841\") "
Oct 30 13:30:06.919446 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:06.919232    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/617447eb-2414-439d-9923-c47b809de841-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/617447eb-2414-439d-9923-c47b809de841-credential-socket\") pod \"617447eb-2414-439d-9923-c47b809de841\" (UID: \"617447eb-2414-439d-9923-c47b809de841\") "
Oct 30 13:30:06.919446 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:06.919257    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/617447eb-2414-439d-9923-c47b809de841-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/617447eb-2414-439d-9923-c47b809de841-istio-envoy\") pod \"617447eb-2414-439d-9923-c47b809de841\" (UID: \"617447eb-2414-439d-9923-c47b809de841\") "
Oct 30 13:30:06.919640 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:06.919493    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/617447eb-2414-439d-9923-c47b809de841-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/617447eb-2414-439d-9923-c47b809de841-istio-data\") pod \"617447eb-2414-439d-9923-c47b809de841\" (UID: \"617447eb-2414-439d-9923-c47b809de841\") "
Oct 30 13:30:06.919640 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:06.919592    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/617447eb-2414-439d-9923-c47b809de841-istio-token\" (UniqueName: \"kubernetes.io/projected/617447eb-2414-439d-9923-c47b809de841-istio-token\") pod \"617447eb-2414-439d-9923-c47b809de841\" (UID: \"617447eb-2414-439d-9923-c47b809de841\") "
Oct 30 13:30:06.919751 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:06.919591    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/617447eb-2414-439d-9923-c47b809de841-credential-socket" pod "617447eb-2414-439d-9923-c47b809de841" (UID: "617447eb-2414-439d-9923-c47b809de841"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:30:06.919751 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:06.919711    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/617447eb-2414-439d-9923-c47b809de841-istiod-ca-cert" pod "617447eb-2414-439d-9923-c47b809de841" (UID: "617447eb-2414-439d-9923-c47b809de841"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:30:06.919751 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:06.919742    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/617447eb-2414-439d-9923-c47b809de841-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/617447eb-2414-439d-9923-c47b809de841-workload-certs\") pod \"617447eb-2414-439d-9923-c47b809de841\" (UID: \"617447eb-2414-439d-9923-c47b809de841\") "
Oct 30 13:30:06.919917 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:06.919787    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/617447eb-2414-439d-9923-c47b809de841-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/617447eb-2414-439d-9923-c47b809de841-workload-socket\") pod \"617447eb-2414-439d-9923-c47b809de841\" (UID: \"617447eb-2414-439d-9923-c47b809de841\") "
Oct 30 13:30:06.919917 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:06.919799    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/617447eb-2414-439d-9923-c47b809de841-istio-data" pod "617447eb-2414-439d-9923-c47b809de841" (UID: "617447eb-2414-439d-9923-c47b809de841"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:30:06.920096 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:06.920016    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/617447eb-2414-439d-9923-c47b809de841-workload-certs" pod "617447eb-2414-439d-9923-c47b809de841" (UID: "617447eb-2414-439d-9923-c47b809de841"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:30:06.920190 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:06.920075    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/617447eb-2414-439d-9923-c47b809de841-workload-socket" pod "617447eb-2414-439d-9923-c47b809de841" (UID: "617447eb-2414-439d-9923-c47b809de841"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:30:06.920490 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:06.920474    4364 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/617447eb-2414-439d-9923-c47b809de841-istio-data\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:30:06.920538 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:06.920492    4364 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/617447eb-2414-439d-9923-c47b809de841-workload-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:30:06.920538 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:06.920503    4364 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/617447eb-2414-439d-9923-c47b809de841-workload-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:30:06.920538 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:06.920511    4364 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/617447eb-2414-439d-9923-c47b809de841-istiod-ca-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:30:06.920538 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:06.920519    4364 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/617447eb-2414-439d-9923-c47b809de841-credential-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:30:06.922066 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:06.922034    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/617447eb-2414-439d-9923-c47b809de841-istio-podinfo" pod "617447eb-2414-439d-9923-c47b809de841" (UID: "617447eb-2414-439d-9923-c47b809de841"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:30:06.922165 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:06.922066    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/617447eb-2414-439d-9923-c47b809de841-istio-envoy" pod "617447eb-2414-439d-9923-c47b809de841" (UID: "617447eb-2414-439d-9923-c47b809de841"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:30:06.922165 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:06.922085    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/617447eb-2414-439d-9923-c47b809de841-kube-api-access-dlbp5" pod "617447eb-2414-439d-9923-c47b809de841" (UID: "617447eb-2414-439d-9923-c47b809de841"). InnerVolumeSpecName "kube-api-access-dlbp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:30:06.922165 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:06.922120    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/617447eb-2414-439d-9923-c47b809de841-istio-token" pod "617447eb-2414-439d-9923-c47b809de841" (UID: "617447eb-2414-439d-9923-c47b809de841"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:30:07.021585 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:07.021554    4364 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/617447eb-2414-439d-9923-c47b809de841-istio-token\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:30:07.021585 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:07.021582    4364 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/617447eb-2414-439d-9923-c47b809de841-istio-podinfo\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:30:07.021790 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:07.021592    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlbp5\" (UniqueName: \"kubernetes.io/projected/617447eb-2414-439d-9923-c47b809de841-kube-api-access-dlbp5\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:30:07.021790 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:07.021614    4364 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/617447eb-2414-439d-9923-c47b809de841-istio-envoy\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:30:07.299573 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:07.299485    4364 generic.go:356] "Generic (PLEG): container finished" podID="617447eb-2414-439d-9923-c47b809de841" containerID="1925ed105ee025e7459d40dead0c738808cda8573e6e0cf8bd847f8615448e50" exitCode=0
Oct 30 13:30:07.299573 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:07.299531    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw"
Oct 30 13:30:07.299573 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:07.299558    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw" event={"ID":"617447eb-2414-439d-9923-c47b809de841","Type":"ContainerDied","Data":"1925ed105ee025e7459d40dead0c738808cda8573e6e0cf8bd847f8615448e50"}
Oct 30 13:30:07.300206 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:07.299662    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw" event={"ID":"617447eb-2414-439d-9923-c47b809de841","Type":"ContainerDied","Data":"7577b4f5a0c12afd187af7443dc0ec355a2f7b35073ef4b611d43920561c714e"}
Oct 30 13:30:07.300206 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:07.299699    4364 scope.go:122] "RemoveContainer" containerID="1925ed105ee025e7459d40dead0c738808cda8573e6e0cf8bd847f8615448e50"
Oct 30 13:30:07.302151 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:07.302128    4364 scope.go:122] "RemoveContainer" containerID="99323dabcbc22c4a496ea6053bdb9274cd33134030599970cfca7731a3a31282"
Oct 30 13:30:07.302557 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:30:07.302528    4364 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889450-4glnd_odh-ai-gateway-infra(a0239aaa-e1bf-4ad8-bf54-42920cc8009c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889450-4glnd" podUID="a0239aaa-e1bf-4ad8-bf54-42920cc8009c"
Oct 30 13:30:07.304288 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:07.304272    4364 scope.go:122] "RemoveContainer" containerID="8ab1de124a77ef88afaafd81cb0d8170abeab194aa2451ea2a20c698de823a6e"
Oct 30 13:30:07.304524 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:30:07.304506    4364 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-4b0ea9-29889450-dc8t7_odh-ai-gateway-infra(0fa046ef-c365-47a1-976f-18f12312ea4e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4b0ea9-29889450-dc8t7" podUID="0fa046ef-c365-47a1-976f-18f12312ea4e"
Oct 30 13:30:07.313786 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:07.313767    4364 scope.go:122] "RemoveContainer" containerID="1925ed105ee025e7459d40dead0c738808cda8573e6e0cf8bd847f8615448e50"
Oct 30 13:30:07.314070 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:30:07.314047    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1925ed105ee025e7459d40dead0c738808cda8573e6e0cf8bd847f8615448e50\": container with ID starting with 1925ed105ee025e7459d40dead0c738808cda8573e6e0cf8bd847f8615448e50 not found: ID does not exist" containerID="1925ed105ee025e7459d40dead0c738808cda8573e6e0cf8bd847f8615448e50"
Oct 30 13:30:07.314123 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:07.314081    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1925ed105ee025e7459d40dead0c738808cda8573e6e0cf8bd847f8615448e50"} err="failed to get container status \"1925ed105ee025e7459d40dead0c738808cda8573e6e0cf8bd847f8615448e50\": rpc error: code = NotFound desc = could not find container \"1925ed105ee025e7459d40dead0c738808cda8573e6e0cf8bd847f8615448e50\": container with ID starting with 1925ed105ee025e7459d40dead0c738808cda8573e6e0cf8bd847f8615448e50 not found: ID does not exist"
Oct 30 13:30:07.425397 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:07.425363    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw"]
Oct 30 13:30:07.437048 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:07.437015    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-5c7ba3-openshift-default-7794f6786c-7dwnw"]
Oct 30 13:30:08.972152 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:08.972108    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="617447eb-2414-439d-9923-c47b809de841" path="/var/lib/kubelet/pods/617447eb-2414-439d-9923-c47b809de841/volumes"
Oct 30 13:30:10.018228 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:10.018194    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-fdd4d4-78689f8d4-5hq2s"]
Oct 30 13:30:10.019083 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:10.019064    4364 cpu_manager.go:414] "containerMap: removing container" podUID="617447eb-2414-439d-9923-c47b809de841" containerName="istio-proxy"
Oct 30 13:30:10.019157 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:10.019085    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="617447eb-2414-439d-9923-c47b809de841" containerName="istio-proxy"
Oct 30 13:30:10.019206 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:10.019197    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="617447eb-2414-439d-9923-c47b809de841" containerName="istio-proxy"
Oct 30 13:30:10.023152 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:10.023132    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-fdd4d4-78689f8d4-5hq2s"
Oct 30 13:30:10.071621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:10.069619    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-fdd4d4-dockercfg-6xzcw\""
Oct 30 13:30:10.106627 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:10.065586    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-fdd4d4\""
Oct 30 13:30:10.148624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:10.130206    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2qxt\" (UniqueName: \"kubernetes.io/projected/43de3339-76dc-44c5-9ce6-9e592b0fceba-kube-api-access-v2qxt\") pod \"payload-processing-e2e-shared-a-w0-fdd4d4-78689f8d4-5hq2s\" (UID: \"43de3339-76dc-44c5-9ce6-9e592b0fceba\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-fdd4d4-78689f8d4-5hq2s"
Oct 30 13:30:10.148624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:10.130304    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43de3339-76dc-44c5-9ce6-9e592b0fceba-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-fdd4d4-78689f8d4-5hq2s\" (UID: \"43de3339-76dc-44c5-9ce6-9e592b0fceba\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-fdd4d4-78689f8d4-5hq2s"
Oct 30 13:30:10.176471 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:10.149181    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-fdd4d4-78689f8d4-5hq2s"]
Oct 30 13:30:10.256374 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:10.256326    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2qxt\" (UniqueName: \"kubernetes.io/projected/43de3339-76dc-44c5-9ce6-9e592b0fceba-kube-api-access-v2qxt\") pod \"payload-processing-e2e-shared-a-w0-fdd4d4-78689f8d4-5hq2s\" (UID: \"43de3339-76dc-44c5-9ce6-9e592b0fceba\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-fdd4d4-78689f8d4-5hq2s"
Oct 30 13:30:10.261801 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:10.261763    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43de3339-76dc-44c5-9ce6-9e592b0fceba-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-fdd4d4-78689f8d4-5hq2s\" (UID: \"43de3339-76dc-44c5-9ce6-9e592b0fceba\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-fdd4d4-78689f8d4-5hq2s"
Oct 30 13:30:10.263439 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:10.263409    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43de3339-76dc-44c5-9ce6-9e592b0fceba-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-fdd4d4-78689f8d4-5hq2s\" (UID: \"43de3339-76dc-44c5-9ce6-9e592b0fceba\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-fdd4d4-78689f8d4-5hq2s"
Oct 30 13:30:10.340015 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:10.339937    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2qxt\" (UniqueName: \"kubernetes.io/projected/43de3339-76dc-44c5-9ce6-9e592b0fceba-kube-api-access-v2qxt\") pod \"payload-processing-e2e-shared-a-w0-fdd4d4-78689f8d4-5hq2s\" (UID: \"43de3339-76dc-44c5-9ce6-9e592b0fceba\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-fdd4d4-78689f8d4-5hq2s"
Oct 30 13:30:10.637941 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:10.637908    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-fdd4d4-78689f8d4-5hq2s"
Oct 30 13:30:10.955372 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:10.955342    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-fdd4d4-78689f8d4-5hq2s"]
Oct 30 13:30:10.955988 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:30:10.955959    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43de3339_76dc_44c5_9ce6_9e592b0fceba.slice/crio-b4ad05e820f748ec4613574a358326c37fbc3edee4c8b88d995ce7e7db0fe443 WatchSource:0}: Error finding container b4ad05e820f748ec4613574a358326c37fbc3edee4c8b88d995ce7e7db0fe443: Status 404 returned error can't find the container with id b4ad05e820f748ec4613574a358326c37fbc3edee4c8b88d995ce7e7db0fe443
Oct 30 13:30:11.463378 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:11.463340    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-fdd4d4-78689f8d4-5hq2s" event={"ID":"43de3339-76dc-44c5-9ce6-9e592b0fceba","Type":"ContainerStarted","Data":"9e6c937eb3fec06045d356b312c08dcf5e7bbed23903774927ef705de445f87d"}
Oct 30 13:30:11.463807 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:11.463405    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-fdd4d4-78689f8d4-5hq2s" event={"ID":"43de3339-76dc-44c5-9ce6-9e592b0fceba","Type":"ContainerStarted","Data":"b4ad05e820f748ec4613574a358326c37fbc3edee4c8b88d995ce7e7db0fe443"}
Oct 30 13:30:11.463807 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:11.463439    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-fdd4d4-78689f8d4-5hq2s"
Oct 30 13:30:11.510395 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:30:11.510361    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:30:11.511680 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:11.511626    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-fdd4d4-78689f8d4-5hq2s" podStartSLOduration=2.511592129 podStartE2EDuration="2.511592129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:11.504338263 +0000 UTC m=+2310.670702905" watchObservedRunningTime="2026-10-30 13:30:11.511592129 +0000 UTC m=+2310.677956776"
Oct 30 13:30:12.329506 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:12.327854    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6"]
Oct 30 13:30:12.329506 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:12.328182    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6" podUID="f9846347-72dc-472f-8bb1-57d39f1be48b" containerName="istio-proxy" containerID="cri-o://94816d09f019e13ca2216dfbd2b861a9fd1015f8deccabe1aaff68663dfb0341" gracePeriod=30
Oct 30 13:30:15.963289 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:15.963255    4364 scope.go:122] "RemoveContainer" containerID="6a7d7928484eae9643ceeffd1733851a316acb27aacab15d173dc7250c9b35d1"
Oct 30 13:30:16.595271 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:16.595236    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8d82d5-29889450-g5qzp" event={"ID":"ace885ec-090f-469d-ba45-23bd11e84d9e","Type":"ContainerStarted","Data":"dc84ee5f7011e9b11c52546b908670b5b37bf807df85052a6b0fb96c5c9d5064"}
Oct 30 13:30:17.611870 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:17.611792    4364 generic.go:356] "Generic (PLEG): container finished" podID="ace885ec-090f-469d-ba45-23bd11e84d9e" containerID="dc84ee5f7011e9b11c52546b908670b5b37bf807df85052a6b0fb96c5c9d5064" exitCode=60
Oct 30 13:30:17.612270 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:17.611863    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8d82d5-29889450-g5qzp" event={"ID":"ace885ec-090f-469d-ba45-23bd11e84d9e","Type":"ContainerDied","Data":"dc84ee5f7011e9b11c52546b908670b5b37bf807df85052a6b0fb96c5c9d5064"}
Oct 30 13:30:17.612270 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:17.611927    4364 scope.go:122] "RemoveContainer" containerID="6a7d7928484eae9643ceeffd1733851a316acb27aacab15d173dc7250c9b35d1"
Oct 30 13:30:17.612416 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:17.612395    4364 scope.go:122] "RemoveContainer" containerID="dc84ee5f7011e9b11c52546b908670b5b37bf807df85052a6b0fb96c5c9d5064"
Oct 30 13:30:17.612896 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:30:17.612875    4364 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-8d82d5-29889450-g5qzp_odh-ai-gateway-infra(ace885ec-090f-469d-ba45-23bd11e84d9e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8d82d5-29889450-g5qzp" podUID="ace885ec-090f-469d-ba45-23bd11e84d9e"
Oct 30 13:30:17.614964 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:17.614944    4364 generic.go:356] "Generic (PLEG): container finished" podID="f9846347-72dc-472f-8bb1-57d39f1be48b" containerID="94816d09f019e13ca2216dfbd2b861a9fd1015f8deccabe1aaff68663dfb0341" exitCode=0
Oct 30 13:30:17.615050 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:17.615018    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6" event={"ID":"f9846347-72dc-472f-8bb1-57d39f1be48b","Type":"ContainerDied","Data":"94816d09f019e13ca2216dfbd2b861a9fd1015f8deccabe1aaff68663dfb0341"}
Oct 30 13:30:17.615093 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:17.615072    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6" event={"ID":"f9846347-72dc-472f-8bb1-57d39f1be48b","Type":"ContainerDied","Data":"0b91d24eed723e32d9a4fb478237dbad36b93feb0cfc3742b1267e48833ede83"}
Oct 30 13:30:17.615134 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:17.615096    4364 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b91d24eed723e32d9a4fb478237dbad36b93feb0cfc3742b1267e48833ede83"
Oct 30 13:30:17.626489 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:17.626470    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6"
Oct 30 13:30:17.628750 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:17.628730    4364 scope.go:122] "RemoveContainer" containerID="ed645de8c0778c120d3496bc0a7411e7104faa949fcc3b120a03bf060d83e434"
Oct 30 13:30:17.649029 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:17.649004    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8d82d5-29889450-g5qzp"]
Oct 30 13:30:17.736218 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:17.736184    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9846347-72dc-472f-8bb1-57d39f1be48b-kube-api-access-f6d4l\" (UniqueName: \"kubernetes.io/projected/f9846347-72dc-472f-8bb1-57d39f1be48b-kube-api-access-f6d4l\") pod \"f9846347-72dc-472f-8bb1-57d39f1be48b\" (UID: \"f9846347-72dc-472f-8bb1-57d39f1be48b\") "
Oct 30 13:30:17.736410 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:17.736226    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9846347-72dc-472f-8bb1-57d39f1be48b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9846347-72dc-472f-8bb1-57d39f1be48b-istio-data\") pod \"f9846347-72dc-472f-8bb1-57d39f1be48b\" (UID: \"f9846347-72dc-472f-8bb1-57d39f1be48b\") "
Oct 30 13:30:17.736410 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:17.736247    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9846347-72dc-472f-8bb1-57d39f1be48b-istio-token\" (UniqueName: \"kubernetes.io/projected/f9846347-72dc-472f-8bb1-57d39f1be48b-istio-token\") pod \"f9846347-72dc-472f-8bb1-57d39f1be48b\" (UID: \"f9846347-72dc-472f-8bb1-57d39f1be48b\") "
Oct 30 13:30:17.736410 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:17.736280    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9846347-72dc-472f-8bb1-57d39f1be48b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9846347-72dc-472f-8bb1-57d39f1be48b-istio-envoy\") pod \"f9846347-72dc-472f-8bb1-57d39f1be48b\" (UID: \"f9846347-72dc-472f-8bb1-57d39f1be48b\") "
Oct 30 13:30:17.736410 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:17.736361    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9846347-72dc-472f-8bb1-57d39f1be48b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9846347-72dc-472f-8bb1-57d39f1be48b-istiod-ca-cert\") pod \"f9846347-72dc-472f-8bb1-57d39f1be48b\" (UID: \"f9846347-72dc-472f-8bb1-57d39f1be48b\") "
Oct 30 13:30:17.736410 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:17.736409    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f9846347-72dc-472f-8bb1-57d39f1be48b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9846347-72dc-472f-8bb1-57d39f1be48b-istio-podinfo\") pod \"f9846347-72dc-472f-8bb1-57d39f1be48b\" (UID: \"f9846347-72dc-472f-8bb1-57d39f1be48b\") "
Oct 30 13:30:17.736728 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:17.736497    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9846347-72dc-472f-8bb1-57d39f1be48b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9846347-72dc-472f-8bb1-57d39f1be48b-workload-certs\") pod \"f9846347-72dc-472f-8bb1-57d39f1be48b\" (UID: \"f9846347-72dc-472f-8bb1-57d39f1be48b\") "
Oct 30 13:30:17.736728 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:17.736504    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9846347-72dc-472f-8bb1-57d39f1be48b-istio-data" pod "f9846347-72dc-472f-8bb1-57d39f1be48b" (UID: "f9846347-72dc-472f-8bb1-57d39f1be48b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:30:17.736728 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:17.736581    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9846347-72dc-472f-8bb1-57d39f1be48b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9846347-72dc-472f-8bb1-57d39f1be48b-workload-socket\") pod \"f9846347-72dc-472f-8bb1-57d39f1be48b\" (UID: \"f9846347-72dc-472f-8bb1-57d39f1be48b\") "
Oct 30 13:30:17.736728 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:17.736630    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9846347-72dc-472f-8bb1-57d39f1be48b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9846347-72dc-472f-8bb1-57d39f1be48b-credential-socket\") pod \"f9846347-72dc-472f-8bb1-57d39f1be48b\" (UID: \"f9846347-72dc-472f-8bb1-57d39f1be48b\") "
Oct 30 13:30:17.736963 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:17.736904    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9846347-72dc-472f-8bb1-57d39f1be48b-workload-certs" pod "f9846347-72dc-472f-8bb1-57d39f1be48b" (UID: "f9846347-72dc-472f-8bb1-57d39f1be48b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:30:17.736963 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:17.736832    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9846347-72dc-472f-8bb1-57d39f1be48b-istiod-ca-cert" pod "f9846347-72dc-472f-8bb1-57d39f1be48b" (UID: "f9846347-72dc-472f-8bb1-57d39f1be48b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:30:17.737108 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:17.737061    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9846347-72dc-472f-8bb1-57d39f1be48b-workload-socket" pod "f9846347-72dc-472f-8bb1-57d39f1be48b" (UID: "f9846347-72dc-472f-8bb1-57d39f1be48b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:30:17.737227 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:17.737203    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9846347-72dc-472f-8bb1-57d39f1be48b-credential-socket" pod "f9846347-72dc-472f-8bb1-57d39f1be48b" (UID: "f9846347-72dc-472f-8bb1-57d39f1be48b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:30:17.737543 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:17.737519    4364 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9846347-72dc-472f-8bb1-57d39f1be48b-workload-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:30:17.737651 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:17.737550    4364 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9846347-72dc-472f-8bb1-57d39f1be48b-workload-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:30:17.737651 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:17.737567    4364 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9846347-72dc-472f-8bb1-57d39f1be48b-credential-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:30:17.737651 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:17.737582    4364 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9846347-72dc-472f-8bb1-57d39f1be48b-istio-data\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:30:17.737651 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:17.737613    4364 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9846347-72dc-472f-8bb1-57d39f1be48b-istiod-ca-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:30:17.738682 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:17.738655    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9846347-72dc-472f-8bb1-57d39f1be48b-kube-api-access-f6d4l" pod "f9846347-72dc-472f-8bb1-57d39f1be48b" (UID: "f9846347-72dc-472f-8bb1-57d39f1be48b"). InnerVolumeSpecName "kube-api-access-f6d4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:30:17.738782 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:17.738759    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9846347-72dc-472f-8bb1-57d39f1be48b-istio-podinfo" pod "f9846347-72dc-472f-8bb1-57d39f1be48b" (UID: "f9846347-72dc-472f-8bb1-57d39f1be48b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:30:17.738840 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:17.738823    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9846347-72dc-472f-8bb1-57d39f1be48b-istio-token" pod "f9846347-72dc-472f-8bb1-57d39f1be48b" (UID: "f9846347-72dc-472f-8bb1-57d39f1be48b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:30:17.738981 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:17.738957    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9846347-72dc-472f-8bb1-57d39f1be48b-istio-envoy" pod "f9846347-72dc-472f-8bb1-57d39f1be48b" (UID: "f9846347-72dc-472f-8bb1-57d39f1be48b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:30:17.838879 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:17.838839    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6d4l\" (UniqueName: \"kubernetes.io/projected/f9846347-72dc-472f-8bb1-57d39f1be48b-kube-api-access-f6d4l\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:30:17.838879 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:17.838868    4364 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9846347-72dc-472f-8bb1-57d39f1be48b-istio-token\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:30:17.838879 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:17.838877    4364 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9846347-72dc-472f-8bb1-57d39f1be48b-istio-envoy\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:30:17.838879 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:17.838885    4364 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9846347-72dc-472f-8bb1-57d39f1be48b-istio-podinfo\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:30:17.963199 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:17.963169    4364 scope.go:122] "RemoveContainer" containerID="dc73527c53e98c7dbbe11472b0bb89863e309682c54992bbbe17416f6e2a54cb"
Oct 30 13:30:17.963397 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:17.963279    4364 scope.go:122] "RemoveContainer" containerID="5ea480a392a7f2a1514b8a2c58f256407b7b8febac3682a0015337d0161f4fca"
Oct 30 13:30:18.629486 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:18.629457    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d833bd-29889450-q6468" event={"ID":"ff661c30-4a5a-4a98-b640-3b2c3e4143e1","Type":"ContainerStarted","Data":"6cf68a8fec1e1a6888254a0ebc8595b31f7a0478f117df9790d9ceb9a40a971d"}
Oct 30 13:30:18.631234 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:18.631210    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6"
Oct 30 13:30:18.633440 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:18.633417    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-cda0d9-29889450-bnh6t" event={"ID":"d10daaa5-7cac-4205-8638-75ede670c28f","Type":"ContainerStarted","Data":"7529f5ab95c88508b806b4cdf3755790380e22f86e008e13391b7569893dbc90"}
Oct 30 13:30:18.841939 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:18.841283    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8d82d5-29889450-g5qzp"
Oct 30 13:30:18.847032 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:18.846999    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6"]
Oct 30 13:30:18.851610 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:18.851571    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-bflg6"]
Oct 30 13:30:18.858390 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:18.858363    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ace885ec-090f-469d-ba45-23bd11e84d9e-kube-api-access-2qf4w\" (UniqueName: \"kubernetes.io/projected/ace885ec-090f-469d-ba45-23bd11e84d9e-kube-api-access-2qf4w\") pod \"ace885ec-090f-469d-ba45-23bd11e84d9e\" (UID: \"ace885ec-090f-469d-ba45-23bd11e84d9e\") "
Oct 30 13:30:18.858505 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:18.858487    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ace885ec-090f-469d-ba45-23bd11e84d9e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ace885ec-090f-469d-ba45-23bd11e84d9e-maas-api-service-ca\") pod \"ace885ec-090f-469d-ba45-23bd11e84d9e\" (UID: \"ace885ec-090f-469d-ba45-23bd11e84d9e\") "
Oct 30 13:30:18.858847 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:18.858825    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ace885ec-090f-469d-ba45-23bd11e84d9e-maas-api-service-ca" pod "ace885ec-090f-469d-ba45-23bd11e84d9e" (UID: "ace885ec-090f-469d-ba45-23bd11e84d9e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:30:18.859209 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:18.859188    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ace885ec-090f-469d-ba45-23bd11e84d9e-maas-api-service-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:30:18.860485 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:18.860461    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ace885ec-090f-469d-ba45-23bd11e84d9e-kube-api-access-2qf4w" pod "ace885ec-090f-469d-ba45-23bd11e84d9e" (UID: "ace885ec-090f-469d-ba45-23bd11e84d9e"). InnerVolumeSpecName "kube-api-access-2qf4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:30:18.961150 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:18.961108    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qf4w\" (UniqueName: \"kubernetes.io/projected/ace885ec-090f-469d-ba45-23bd11e84d9e-kube-api-access-2qf4w\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:30:18.963037 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:18.963010    4364 scope.go:122] "RemoveContainer" containerID="99323dabcbc22c4a496ea6053bdb9274cd33134030599970cfca7731a3a31282"
Oct 30 13:30:18.963212 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:18.963154    4364 scope.go:122] "RemoveContainer" containerID="8ab1de124a77ef88afaafd81cb0d8170abeab194aa2451ea2a20c698de823a6e"
Oct 30 13:30:18.963273 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:18.963224    4364 scope.go:122] "RemoveContainer" containerID="bc5575d7b544564d427cbbab12a6c6ce56a64b7b5c0c135b67da0ff70f93860e"
Oct 30 13:30:18.963381 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:18.963363    4364 scope.go:122] "RemoveContainer" containerID="2c5af3f868c7a0399ffec13701c692393a6bccd56c36564e8a5643010a029fa5"
Oct 30 13:30:18.977645 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:18.976151    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9846347-72dc-472f-8bb1-57d39f1be48b" path="/var/lib/kubelet/pods/f9846347-72dc-472f-8bb1-57d39f1be48b/volumes"
Oct 30 13:30:19.644874 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:19.644841    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9c6033-29889450-589x5" event={"ID":"17dc6af9-2a39-48fe-a3dc-405f9f9f1ea0","Type":"ContainerStarted","Data":"33f67978b800d01866e87db7b11c07d84e7a5f4dda9274ae754dc5e4e61c4baf"}
Oct 30 13:30:19.647194 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:19.647165    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889450-4glnd" event={"ID":"a0239aaa-e1bf-4ad8-bf54-42920cc8009c","Type":"ContainerStarted","Data":"a45166a070b767066963dd4e4fd26f881badaf5fc4e517b3a20e5a65b4588596"}
Oct 30 13:30:19.648752 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:19.648729    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8d82d5-29889450-g5qzp"
Oct 30 13:30:19.648851 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:19.648749    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8d82d5-29889450-g5qzp" event={"ID":"ace885ec-090f-469d-ba45-23bd11e84d9e","Type":"ContainerDied","Data":"088961120cc2b696b2aedf712b6a423d003c1eff8fd3465428c216eeea352226"}
Oct 30 13:30:19.648851 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:19.648803    4364 scope.go:122] "RemoveContainer" containerID="dc84ee5f7011e9b11c52546b908670b5b37bf807df85052a6b0fb96c5c9d5064"
Oct 30 13:30:19.651404 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:19.651378    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4b0ea9-29889450-dc8t7" event={"ID":"0fa046ef-c365-47a1-976f-18f12312ea4e","Type":"ContainerStarted","Data":"54c50e58bffbf90ffa7166605eda5fb7b3a15cc00e5cb22d4944e1280dbc0e3d"}
Oct 30 13:30:19.653541 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:19.653508    4364 generic.go:356] "Generic (PLEG): container finished" podID="ff661c30-4a5a-4a98-b640-3b2c3e4143e1" containerID="6cf68a8fec1e1a6888254a0ebc8595b31f7a0478f117df9790d9ceb9a40a971d" exitCode=60
Oct 30 13:30:19.653686 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:19.653552    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d833bd-29889450-q6468" event={"ID":"ff661c30-4a5a-4a98-b640-3b2c3e4143e1","Type":"ContainerDied","Data":"6cf68a8fec1e1a6888254a0ebc8595b31f7a0478f117df9790d9ceb9a40a971d"}
Oct 30 13:30:19.653970 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:19.653948    4364 scope.go:122] "RemoveContainer" containerID="6cf68a8fec1e1a6888254a0ebc8595b31f7a0478f117df9790d9ceb9a40a971d"
Oct 30 13:30:19.654383 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:30:19.654358    4364 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-d833bd-29889450-q6468_odh-ai-gateway-infra(ff661c30-4a5a-4a98-b640-3b2c3e4143e1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d833bd-29889450-q6468" podUID="ff661c30-4a5a-4a98-b640-3b2c3e4143e1"
Oct 30 13:30:19.655683 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:19.655662    4364 generic.go:356] "Generic (PLEG): container finished" podID="d10daaa5-7cac-4205-8638-75ede670c28f" containerID="7529f5ab95c88508b806b4cdf3755790380e22f86e008e13391b7569893dbc90" exitCode=60
Oct 30 13:30:19.655755 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:19.655722    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-cda0d9-29889450-bnh6t" event={"ID":"d10daaa5-7cac-4205-8638-75ede670c28f","Type":"ContainerDied","Data":"7529f5ab95c88508b806b4cdf3755790380e22f86e008e13391b7569893dbc90"}
Oct 30 13:30:19.656049 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:19.656037    4364 scope.go:122] "RemoveContainer" containerID="7529f5ab95c88508b806b4cdf3755790380e22f86e008e13391b7569893dbc90"
Oct 30 13:30:19.656311 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:30:19.656294    4364 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-cda0d9-29889450-bnh6t_odh-ai-gateway-infra(d10daaa5-7cac-4205-8638-75ede670c28f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-cda0d9-29889450-bnh6t" podUID="d10daaa5-7cac-4205-8638-75ede670c28f"
Oct 30 13:30:19.660939 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:19.660921    4364 scope.go:122] "RemoveContainer" containerID="dc73527c53e98c7dbbe11472b0bb89863e309682c54992bbbe17416f6e2a54cb"
Oct 30 13:30:19.676948 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:19.676926    4364 scope.go:122] "RemoveContainer" containerID="5ea480a392a7f2a1514b8a2c58f256407b7b8febac3682a0015337d0161f4fca"
Oct 30 13:30:19.730795 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:19.730765    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8d82d5-29889450-g5qzp"]
Oct 30 13:30:19.740294 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:19.740227    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d833bd-29889450-q6468"]
Oct 30 13:30:19.744896 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:19.744875    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-8d82d5-29889450-g5qzp"]
Oct 30 13:30:19.778535 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:19.778507    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-cda0d9-29889450-bnh6t"]
Oct 30 13:30:20.666055 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:20.666025    4364 generic.go:356] "Generic (PLEG): container finished" podID="f08ebb94-561c-42b6-9430-3c1d78c5fc8a" containerID="5d037801fcc45043ebec811b39dc761e5f0b97e233e65fd2e7ac229481deaf09" exitCode=60
Oct 30 13:30:20.666538 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:20.666109    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b29df4-29889450-v8kpn" event={"ID":"f08ebb94-561c-42b6-9430-3c1d78c5fc8a","Type":"ContainerDied","Data":"5d037801fcc45043ebec811b39dc761e5f0b97e233e65fd2e7ac229481deaf09"}
Oct 30 13:30:20.666538 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:20.666171    4364 scope.go:122] "RemoveContainer" containerID="2c5af3f868c7a0399ffec13701c692393a6bccd56c36564e8a5643010a029fa5"
Oct 30 13:30:20.666706 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:20.666674    4364 scope.go:122] "RemoveContainer" containerID="5d037801fcc45043ebec811b39dc761e5f0b97e233e65fd2e7ac229481deaf09"
Oct 30 13:30:20.667221 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:30:20.667112    4364 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-b29df4-29889450-v8kpn_odh-ai-gateway-infra(f08ebb94-561c-42b6-9430-3c1d78c5fc8a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b29df4-29889450-v8kpn" podUID="f08ebb94-561c-42b6-9430-3c1d78c5fc8a"
Oct 30 13:30:20.668430 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:20.668410    4364 generic.go:356] "Generic (PLEG): container finished" podID="17dc6af9-2a39-48fe-a3dc-405f9f9f1ea0" containerID="33f67978b800d01866e87db7b11c07d84e7a5f4dda9274ae754dc5e4e61c4baf" exitCode=60
Oct 30 13:30:20.668529 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:20.668452    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9c6033-29889450-589x5" event={"ID":"17dc6af9-2a39-48fe-a3dc-405f9f9f1ea0","Type":"ContainerDied","Data":"33f67978b800d01866e87db7b11c07d84e7a5f4dda9274ae754dc5e4e61c4baf"}
Oct 30 13:30:20.668917 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:20.668897    4364 scope.go:122] "RemoveContainer" containerID="33f67978b800d01866e87db7b11c07d84e7a5f4dda9274ae754dc5e4e61c4baf"
Oct 30 13:30:20.669258 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:30:20.669239    4364 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-9c6033-29889450-589x5_odh-ai-gateway-infra(17dc6af9-2a39-48fe-a3dc-405f9f9f1ea0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9c6033-29889450-589x5" podUID="17dc6af9-2a39-48fe-a3dc-405f9f9f1ea0"
Oct 30 13:30:20.670707 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:20.670687    4364 generic.go:356] "Generic (PLEG): container finished" podID="a0239aaa-e1bf-4ad8-bf54-42920cc8009c" containerID="a45166a070b767066963dd4e4fd26f881badaf5fc4e517b3a20e5a65b4588596" exitCode=60
Oct 30 13:30:20.670805 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:20.670756    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889450-4glnd" event={"ID":"a0239aaa-e1bf-4ad8-bf54-42920cc8009c","Type":"ContainerDied","Data":"a45166a070b767066963dd4e4fd26f881badaf5fc4e517b3a20e5a65b4588596"}
Oct 30 13:30:20.671099 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:20.671081    4364 scope.go:122] "RemoveContainer" containerID="a45166a070b767066963dd4e4fd26f881badaf5fc4e517b3a20e5a65b4588596"
Oct 30 13:30:20.671358 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:30:20.671336    4364 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889450-4glnd_odh-ai-gateway-infra(a0239aaa-e1bf-4ad8-bf54-42920cc8009c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889450-4glnd" podUID="a0239aaa-e1bf-4ad8-bf54-42920cc8009c"
Oct 30 13:30:20.674232 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:20.674211    4364 generic.go:356] "Generic (PLEG): container finished" podID="0fa046ef-c365-47a1-976f-18f12312ea4e" containerID="54c50e58bffbf90ffa7166605eda5fb7b3a15cc00e5cb22d4944e1280dbc0e3d" exitCode=60
Oct 30 13:30:20.674331 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:20.674280    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4b0ea9-29889450-dc8t7" event={"ID":"0fa046ef-c365-47a1-976f-18f12312ea4e","Type":"ContainerDied","Data":"54c50e58bffbf90ffa7166605eda5fb7b3a15cc00e5cb22d4944e1280dbc0e3d"}
Oct 30 13:30:20.674633 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:20.674589    4364 scope.go:122] "RemoveContainer" containerID="54c50e58bffbf90ffa7166605eda5fb7b3a15cc00e5cb22d4944e1280dbc0e3d"
Oct 30 13:30:20.674906 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:30:20.674885    4364 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-4b0ea9-29889450-dc8t7_odh-ai-gateway-infra(0fa046ef-c365-47a1-976f-18f12312ea4e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4b0ea9-29889450-dc8t7" podUID="0fa046ef-c365-47a1-976f-18f12312ea4e"
Oct 30 13:30:20.683463 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:20.683441    4364 scope.go:122] "RemoveContainer" containerID="bc5575d7b544564d427cbbab12a6c6ce56a64b7b5c0c135b67da0ff70f93860e"
Oct 30 13:30:20.713834 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:20.713797    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9c6033-29889450-589x5"]
Oct 30 13:30:20.725387 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:20.725344    4364 scope.go:122] "RemoveContainer" containerID="99323dabcbc22c4a496ea6053bdb9274cd33134030599970cfca7731a3a31282"
Oct 30 13:30:20.747617 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:20.747380    4364 scope.go:122] "RemoveContainer" containerID="8ab1de124a77ef88afaafd81cb0d8170abeab194aa2451ea2a20c698de823a6e"
Oct 30 13:30:20.810503 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:20.810469    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4b0ea9-29889450-dc8t7"]
Oct 30 13:30:20.870570 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:20.870536    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889450-4glnd"]
Oct 30 13:30:20.891673 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:20.891645    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d833bd-29889450-q6468"
Oct 30 13:30:20.913141 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:20.913113    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-cda0d9-29889450-bnh6t"
Oct 30 13:30:20.980027 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:20.979948    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ace885ec-090f-469d-ba45-23bd11e84d9e" path="/var/lib/kubelet/pods/ace885ec-090f-469d-ba45-23bd11e84d9e/volumes"
Oct 30 13:30:20.995986 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:20.995949    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff661c30-4a5a-4a98-b640-3b2c3e4143e1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff661c30-4a5a-4a98-b640-3b2c3e4143e1-maas-api-service-ca\") pod \"ff661c30-4a5a-4a98-b640-3b2c3e4143e1\" (UID: \"ff661c30-4a5a-4a98-b640-3b2c3e4143e1\") "
Oct 30 13:30:20.996183 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:20.996095    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff661c30-4a5a-4a98-b640-3b2c3e4143e1-kube-api-access-6tq87\" (UniqueName: \"kubernetes.io/projected/ff661c30-4a5a-4a98-b640-3b2c3e4143e1-kube-api-access-6tq87\") pod \"ff661c30-4a5a-4a98-b640-3b2c3e4143e1\" (UID: \"ff661c30-4a5a-4a98-b640-3b2c3e4143e1\") "
Oct 30 13:30:20.996264 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:20.996211    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d10daaa5-7cac-4205-8638-75ede670c28f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d10daaa5-7cac-4205-8638-75ede670c28f-maas-api-service-ca\") pod \"d10daaa5-7cac-4205-8638-75ede670c28f\" (UID: \"d10daaa5-7cac-4205-8638-75ede670c28f\") "
Oct 30 13:30:20.996264 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:20.996239    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d10daaa5-7cac-4205-8638-75ede670c28f-kube-api-access-6wck5\" (UniqueName: \"kubernetes.io/projected/d10daaa5-7cac-4205-8638-75ede670c28f-kube-api-access-6wck5\") pod \"d10daaa5-7cac-4205-8638-75ede670c28f\" (UID: \"d10daaa5-7cac-4205-8638-75ede670c28f\") "
Oct 30 13:30:20.997472 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:20.997441    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff661c30-4a5a-4a98-b640-3b2c3e4143e1-maas-api-service-ca" pod "ff661c30-4a5a-4a98-b640-3b2c3e4143e1" (UID: "ff661c30-4a5a-4a98-b640-3b2c3e4143e1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:30:20.998808 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:20.998776    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d10daaa5-7cac-4205-8638-75ede670c28f-maas-api-service-ca" pod "d10daaa5-7cac-4205-8638-75ede670c28f" (UID: "d10daaa5-7cac-4205-8638-75ede670c28f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:30:21.000113 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:20.999981    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d10daaa5-7cac-4205-8638-75ede670c28f-kube-api-access-6wck5" pod "d10daaa5-7cac-4205-8638-75ede670c28f" (UID: "d10daaa5-7cac-4205-8638-75ede670c28f"). InnerVolumeSpecName "kube-api-access-6wck5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:30:21.000113 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:21.000074    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff661c30-4a5a-4a98-b640-3b2c3e4143e1-kube-api-access-6tq87" pod "ff661c30-4a5a-4a98-b640-3b2c3e4143e1" (UID: "ff661c30-4a5a-4a98-b640-3b2c3e4143e1"). InnerVolumeSpecName "kube-api-access-6tq87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:30:21.098688 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:21.098620    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d10daaa5-7cac-4205-8638-75ede670c28f-maas-api-service-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:30:21.098688 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:21.098653    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wck5\" (UniqueName: \"kubernetes.io/projected/d10daaa5-7cac-4205-8638-75ede670c28f-kube-api-access-6wck5\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:30:21.098688 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:21.098667    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff661c30-4a5a-4a98-b640-3b2c3e4143e1-maas-api-service-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:30:21.098688 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:21.098681    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tq87\" (UniqueName: \"kubernetes.io/projected/ff661c30-4a5a-4a98-b640-3b2c3e4143e1-kube-api-access-6tq87\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:30:21.689891 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:21.689866    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d833bd-29889450-q6468"
Oct 30 13:30:21.690312 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:21.689870    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d833bd-29889450-q6468" event={"ID":"ff661c30-4a5a-4a98-b640-3b2c3e4143e1","Type":"ContainerDied","Data":"d2b6bd9ddc2c1ca900f19b77e606d47800c2d9d39d01dedaee2a57bd4fc52d48"}
Oct 30 13:30:21.690312 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:21.690002    4364 scope.go:122] "RemoveContainer" containerID="6cf68a8fec1e1a6888254a0ebc8595b31f7a0478f117df9790d9ceb9a40a971d"
Oct 30 13:30:21.691551 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:21.691526    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-cda0d9-29889450-bnh6t" event={"ID":"d10daaa5-7cac-4205-8638-75ede670c28f","Type":"ContainerDied","Data":"f5e7aa9c6240024332ea38fb51184a838e590eb301cca694c6def3523b9323f4"}
Oct 30 13:30:21.691722 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:21.691539    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-cda0d9-29889450-bnh6t"
Oct 30 13:30:21.708849 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:21.708824    4364 scope.go:122] "RemoveContainer" containerID="7529f5ab95c88508b806b4cdf3755790380e22f86e008e13391b7569893dbc90"
Oct 30 13:30:21.728971 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:21.728937    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b29df4-29889450-v8kpn"]
Oct 30 13:30:21.835637 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:21.835565    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-cda0d9-29889450-bnh6t"]
Oct 30 13:30:21.838110 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:21.838076    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-cda0d9-29889450-bnh6t"]
Oct 30 13:30:21.872258 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:21.872227    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d833bd-29889450-q6468"]
Oct 30 13:30:21.889918 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:21.889878    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d833bd-29889450-q6468"]
Oct 30 13:30:21.946692 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:21.945847    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4b0ea9-29889450-dc8t7"
Oct 30 13:30:21.984656 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:21.984623    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9c6033-29889450-589x5"
Oct 30 13:30:22.010569 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:22.010124    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889450-4glnd"
Oct 30 13:30:22.013518 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:22.013490    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fa046ef-c365-47a1-976f-18f12312ea4e-kube-api-access-4bj4f\" (UniqueName: \"kubernetes.io/projected/0fa046ef-c365-47a1-976f-18f12312ea4e-kube-api-access-4bj4f\") pod \"0fa046ef-c365-47a1-976f-18f12312ea4e\" (UID: \"0fa046ef-c365-47a1-976f-18f12312ea4e\") "
Oct 30 13:30:22.013756 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:22.013738    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17dc6af9-2a39-48fe-a3dc-405f9f9f1ea0-kube-api-access-9gzs8\" (UniqueName: \"kubernetes.io/projected/17dc6af9-2a39-48fe-a3dc-405f9f9f1ea0-kube-api-access-9gzs8\") pod \"17dc6af9-2a39-48fe-a3dc-405f9f9f1ea0\" (UID: \"17dc6af9-2a39-48fe-a3dc-405f9f9f1ea0\") "
Oct 30 13:30:22.013851 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:22.013798    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fa046ef-c365-47a1-976f-18f12312ea4e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fa046ef-c365-47a1-976f-18f12312ea4e-maas-api-service-ca\") pod \"0fa046ef-c365-47a1-976f-18f12312ea4e\" (UID: \"0fa046ef-c365-47a1-976f-18f12312ea4e\") "
Oct 30 13:30:22.013916 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:22.013866    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17dc6af9-2a39-48fe-a3dc-405f9f9f1ea0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17dc6af9-2a39-48fe-a3dc-405f9f9f1ea0-maas-api-service-ca\") pod \"17dc6af9-2a39-48fe-a3dc-405f9f9f1ea0\" (UID: \"17dc6af9-2a39-48fe-a3dc-405f9f9f1ea0\") "
Oct 30 13:30:22.015638 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:22.015608    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fa046ef-c365-47a1-976f-18f12312ea4e-maas-api-service-ca" pod "0fa046ef-c365-47a1-976f-18f12312ea4e" (UID: "0fa046ef-c365-47a1-976f-18f12312ea4e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:30:22.016186 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:22.016161    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17dc6af9-2a39-48fe-a3dc-405f9f9f1ea0-maas-api-service-ca" pod "17dc6af9-2a39-48fe-a3dc-405f9f9f1ea0" (UID: "17dc6af9-2a39-48fe-a3dc-405f9f9f1ea0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:30:22.017786 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:22.017758    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fa046ef-c365-47a1-976f-18f12312ea4e-kube-api-access-4bj4f" pod "0fa046ef-c365-47a1-976f-18f12312ea4e" (UID: "0fa046ef-c365-47a1-976f-18f12312ea4e"). InnerVolumeSpecName "kube-api-access-4bj4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:30:22.018781 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:22.018760    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17dc6af9-2a39-48fe-a3dc-405f9f9f1ea0-kube-api-access-9gzs8" pod "17dc6af9-2a39-48fe-a3dc-405f9f9f1ea0" (UID: "17dc6af9-2a39-48fe-a3dc-405f9f9f1ea0"). InnerVolumeSpecName "kube-api-access-9gzs8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:30:22.060978 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:22.060951    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b29df4-29889450-v8kpn"
Oct 30 13:30:22.115392 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:22.115349    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0239aaa-e1bf-4ad8-bf54-42920cc8009c-kube-api-access-djtmf\" (UniqueName: \"kubernetes.io/projected/a0239aaa-e1bf-4ad8-bf54-42920cc8009c-kube-api-access-djtmf\") pod \"a0239aaa-e1bf-4ad8-bf54-42920cc8009c\" (UID: \"a0239aaa-e1bf-4ad8-bf54-42920cc8009c\") "
Oct 30 13:30:22.115578 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:22.115407    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f08ebb94-561c-42b6-9430-3c1d78c5fc8a-kube-api-access-j7ktw\" (UniqueName: \"kubernetes.io/projected/f08ebb94-561c-42b6-9430-3c1d78c5fc8a-kube-api-access-j7ktw\") pod \"f08ebb94-561c-42b6-9430-3c1d78c5fc8a\" (UID: \"f08ebb94-561c-42b6-9430-3c1d78c5fc8a\") "
Oct 30 13:30:22.115578 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:22.115469    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0239aaa-e1bf-4ad8-bf54-42920cc8009c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0239aaa-e1bf-4ad8-bf54-42920cc8009c-maas-api-service-ca\") pod \"a0239aaa-e1bf-4ad8-bf54-42920cc8009c\" (UID: \"a0239aaa-e1bf-4ad8-bf54-42920cc8009c\") "
Oct 30 13:30:22.115737 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:22.115634    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f08ebb94-561c-42b6-9430-3c1d78c5fc8a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f08ebb94-561c-42b6-9430-3c1d78c5fc8a-maas-api-service-ca\") pod \"f08ebb94-561c-42b6-9430-3c1d78c5fc8a\" (UID: \"f08ebb94-561c-42b6-9430-3c1d78c5fc8a\") "
Oct 30 13:30:22.116285 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:22.116253    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0239aaa-e1bf-4ad8-bf54-42920cc8009c-maas-api-service-ca" pod "a0239aaa-e1bf-4ad8-bf54-42920cc8009c" (UID: "a0239aaa-e1bf-4ad8-bf54-42920cc8009c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:30:22.116397 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:22.116297    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f08ebb94-561c-42b6-9430-3c1d78c5fc8a-maas-api-service-ca" pod "f08ebb94-561c-42b6-9430-3c1d78c5fc8a" (UID: "f08ebb94-561c-42b6-9430-3c1d78c5fc8a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:30:22.116501 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:22.116471    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gzs8\" (UniqueName: \"kubernetes.io/projected/17dc6af9-2a39-48fe-a3dc-405f9f9f1ea0-kube-api-access-9gzs8\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:30:22.116501 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:22.116500    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fa046ef-c365-47a1-976f-18f12312ea4e-maas-api-service-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:30:22.116704 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:22.116514    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0239aaa-e1bf-4ad8-bf54-42920cc8009c-maas-api-service-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:30:22.116704 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:22.116528    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17dc6af9-2a39-48fe-a3dc-405f9f9f1ea0-maas-api-service-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:30:22.116704 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:22.116541    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f08ebb94-561c-42b6-9430-3c1d78c5fc8a-maas-api-service-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:30:22.116704 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:22.116556    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bj4f\" (UniqueName: \"kubernetes.io/projected/0fa046ef-c365-47a1-976f-18f12312ea4e-kube-api-access-4bj4f\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:30:22.118255 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:22.118225    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f08ebb94-561c-42b6-9430-3c1d78c5fc8a-kube-api-access-j7ktw" pod "f08ebb94-561c-42b6-9430-3c1d78c5fc8a" (UID: "f08ebb94-561c-42b6-9430-3c1d78c5fc8a"). InnerVolumeSpecName "kube-api-access-j7ktw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:30:22.118359 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:22.118327    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0239aaa-e1bf-4ad8-bf54-42920cc8009c-kube-api-access-djtmf" pod "a0239aaa-e1bf-4ad8-bf54-42920cc8009c" (UID: "a0239aaa-e1bf-4ad8-bf54-42920cc8009c"). InnerVolumeSpecName "kube-api-access-djtmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:30:22.217923 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:22.217837    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djtmf\" (UniqueName: \"kubernetes.io/projected/a0239aaa-e1bf-4ad8-bf54-42920cc8009c-kube-api-access-djtmf\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:30:22.217923 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:22.217871    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7ktw\" (UniqueName: \"kubernetes.io/projected/f08ebb94-561c-42b6-9430-3c1d78c5fc8a-kube-api-access-j7ktw\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:30:22.260665 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:22.260625    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:30:22.585343 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:22.585258    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-fdd4d4-78689f8d4-5hq2s"
Oct 30 13:30:22.709018 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:22.708992    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b29df4-29889450-v8kpn"
Oct 30 13:30:22.709480 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:22.708992    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b29df4-29889450-v8kpn" event={"ID":"f08ebb94-561c-42b6-9430-3c1d78c5fc8a","Type":"ContainerDied","Data":"95476c452530334f30c8bad1c67a85c1f5aafbb727eef631d779d8ac5094a87b"}
Oct 30 13:30:22.709480 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:22.709136    4364 scope.go:122] "RemoveContainer" containerID="5d037801fcc45043ebec811b39dc761e5f0b97e233e65fd2e7ac229481deaf09"
Oct 30 13:30:22.710806 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:22.710786    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9c6033-29889450-589x5"
Oct 30 13:30:22.710932 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:22.710819    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9c6033-29889450-589x5" event={"ID":"17dc6af9-2a39-48fe-a3dc-405f9f9f1ea0","Type":"ContainerDied","Data":"975104ee45a5c5f0698fa4752a14361748e13305a35a4f846061a046d608f50f"}
Oct 30 13:30:22.712892 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:22.712870    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889450-4glnd"
Oct 30 13:30:22.713023 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:22.712892    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889450-4glnd" event={"ID":"a0239aaa-e1bf-4ad8-bf54-42920cc8009c","Type":"ContainerDied","Data":"9e12efbcf3591ac6dadce7e9d567d036f7bfb1346acb608abc9700c9500eb5a6"}
Oct 30 13:30:22.715424 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:22.715364    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4b0ea9-29889450-dc8t7"
Oct 30 13:30:22.715424 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:22.715374    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4b0ea9-29889450-dc8t7" event={"ID":"0fa046ef-c365-47a1-976f-18f12312ea4e","Type":"ContainerDied","Data":"1c7ae317a6f8aabda635e606d0d7b1a70b7bbabb64698e81a0aefc280dc5eb2c"}
Oct 30 13:30:22.722889 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:22.722869    4364 scope.go:122] "RemoveContainer" containerID="33f67978b800d01866e87db7b11c07d84e7a5f4dda9274ae754dc5e4e61c4baf"
Oct 30 13:30:22.748435 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:22.748404    4364 scope.go:122] "RemoveContainer" containerID="a45166a070b767066963dd4e4fd26f881badaf5fc4e517b3a20e5a65b4588596"
Oct 30 13:30:22.770927 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:22.770887    4364 scope.go:122] "RemoveContainer" containerID="54c50e58bffbf90ffa7166605eda5fb7b3a15cc00e5cb22d4944e1280dbc0e3d"
Oct 30 13:30:22.790054 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:22.790009    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b29df4-29889450-v8kpn"]
Oct 30 13:30:22.794547 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:22.794520    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b29df4-29889450-v8kpn"]
Oct 30 13:30:22.829685 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:22.829651    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9c6033-29889450-589x5"]
Oct 30 13:30:22.834465 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:22.834437    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9c6033-29889450-589x5"]
Oct 30 13:30:22.881763 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:22.881731    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889450-4glnd"]
Oct 30 13:30:22.889236 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:22.889205    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f07dfe-29889450-4glnd"]
Oct 30 13:30:22.922064 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:22.922027    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4b0ea9-29889450-dc8t7"]
Oct 30 13:30:22.926737 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:22.926710    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4b0ea9-29889450-dc8t7"]
Oct 30 13:30:22.971913 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:22.971879    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fa046ef-c365-47a1-976f-18f12312ea4e" path="/var/lib/kubelet/pods/0fa046ef-c365-47a1-976f-18f12312ea4e/volumes"
Oct 30 13:30:22.972392 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:22.972377    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17dc6af9-2a39-48fe-a3dc-405f9f9f1ea0" path="/var/lib/kubelet/pods/17dc6af9-2a39-48fe-a3dc-405f9f9f1ea0/volumes"
Oct 30 13:30:22.972834 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:22.972820    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0239aaa-e1bf-4ad8-bf54-42920cc8009c" path="/var/lib/kubelet/pods/a0239aaa-e1bf-4ad8-bf54-42920cc8009c/volumes"
Oct 30 13:30:22.973536 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:22.973519    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d10daaa5-7cac-4205-8638-75ede670c28f" path="/var/lib/kubelet/pods/d10daaa5-7cac-4205-8638-75ede670c28f/volumes"
Oct 30 13:30:22.973953 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:22.973939    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f08ebb94-561c-42b6-9430-3c1d78c5fc8a" path="/var/lib/kubelet/pods/f08ebb94-561c-42b6-9430-3c1d78c5fc8a/volumes"
Oct 30 13:30:22.974340 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:22.974328    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff661c30-4a5a-4a98-b640-3b2c3e4143e1" path="/var/lib/kubelet/pods/ff661c30-4a5a-4a98-b640-3b2c3e4143e1/volumes"
Oct 30 13:30:23.035107 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:23.035069    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-306f377f-5pqdt"]
Oct 30 13:30:23.039503 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:23.039475    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-306f377f-5pqdt"]
Oct 30 13:30:25.030621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:25.027855    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea8df410-6452-4d39-966f-feda1430956c" path="/var/lib/kubelet/pods/ea8df410-6452-4d39-966f-feda1430956c/volumes"
Oct 30 13:30:32.851791 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:32.851746    4364 patch_prober.go:29] interesting pod/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:30:32.852366 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:32.852327    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9" podUID="9a967f02-7c93-484e-9bd4-97e1a4f5fdf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:30:36.040541 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:36.040501    4364 generic.go:356] "Generic (PLEG): container finished" podID="9a967f02-7c93-484e-9bd4-97e1a4f5fdf5" containerID="95dc1e29203e28a63d9a2c873fa4bd9873cccc99f836275aa57b1dedabca4994" exitCode=0
Oct 30 13:30:36.041421 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:36.041389    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9" event={"ID":"9a967f02-7c93-484e-9bd4-97e1a4f5fdf5","Type":"ContainerDied","Data":"95dc1e29203e28a63d9a2c873fa4bd9873cccc99f836275aa57b1dedabca4994"}
Oct 30 13:30:36.042203 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:36.042181    4364 scope.go:122] "RemoveContainer" containerID="95dc1e29203e28a63d9a2c873fa4bd9873cccc99f836275aa57b1dedabca4994"
Oct 30 13:30:36.052752 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:36.052715    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:30:36.052897 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:36.052817    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:30:36.052897 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:36.052870    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:30:37.052277 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:37.052239    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9" event={"ID":"9a967f02-7c93-484e-9bd4-97e1a4f5fdf5","Type":"ContainerStarted","Data":"6bd64ed3e7f7b1c54ada326b0f8b22ca74165c53790a0937ddafdccd6d34428e"}
Oct 30 13:30:37.368741 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:37.368714    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9"
Oct 30 13:30:37.369023 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:37.368907    4364 patch_prober.go:29] interesting pod/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 30 13:30:37.369023 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:37.368965    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9" podUID="9a967f02-7c93-484e-9bd4-97e1a4f5fdf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 30 13:30:38.369172 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:38.369137    4364 patch_prober.go:29] interesting pod/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 30 13:30:38.369593 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:38.369195    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9" podUID="9a967f02-7c93-484e-9bd4-97e1a4f5fdf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 30 13:30:39.031412 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:39.031379    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40bd9b0b-54kfk"]
Oct 30 13:30:39.043824 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:39.043790    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40bd9b0b-54kfk"]
Oct 30 13:30:39.369094 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:39.369056    4364 patch_prober.go:29] interesting pod/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 30 13:30:39.369292 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:39.369111    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9" podUID="9a967f02-7c93-484e-9bd4-97e1a4f5fdf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 30 13:30:40.368696 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:40.368663    4364 patch_prober.go:29] interesting pod/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 30 13:30:40.368876 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:40.368723    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9" podUID="9a967f02-7c93-484e-9bd4-97e1a4f5fdf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 30 13:30:40.974921 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:40.974884    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73a2c7d3-f20c-4c98-8416-eec166f20c48" path="/var/lib/kubelet/pods/73a2c7d3-f20c-4c98-8416-eec166f20c48/volumes"
Oct 30 13:30:41.369051 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:41.369024    4364 patch_prober.go:29] interesting pod/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 30 13:30:41.369237 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:41.369082    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9" podUID="9a967f02-7c93-484e-9bd4-97e1a4f5fdf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 30 13:30:41.531450 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:30:41.531406    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:30:42.368737 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:42.368702    4364 patch_prober.go:29] interesting pod/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 30 13:30:42.369216 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:42.369033    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9" podUID="9a967f02-7c93-484e-9bd4-97e1a4f5fdf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 30 13:30:43.277869 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:43.277836    4364 scope.go:122] "RemoveContainer" containerID="94816d09f019e13ca2216dfbd2b861a9fd1015f8deccabe1aaff68663dfb0341"
Oct 30 13:30:43.397076 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:43.397046    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9"
Oct 30 13:30:43.403654 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:43.398714    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9"
Oct 30 13:30:43.403654 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:43.401591    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9"
Oct 30 13:30:47.657651 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.657611    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc"]
Oct 30 13:30:47.658739 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.658693    4364 cpu_manager.go:414] "containerMap: removing container" podUID="17dc6af9-2a39-48fe-a3dc-405f9f9f1ea0" containerName="cleanup"
Oct 30 13:30:47.659922 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.659085    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17dc6af9-2a39-48fe-a3dc-405f9f9f1ea0" containerName="cleanup"
Oct 30 13:30:47.660123 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.660104    4364 cpu_manager.go:414] "containerMap: removing container" podUID="0fa046ef-c365-47a1-976f-18f12312ea4e" containerName="cleanup"
Oct 30 13:30:47.660281 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.660257    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fa046ef-c365-47a1-976f-18f12312ea4e" containerName="cleanup"
Oct 30 13:30:47.660395 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.660384    4364 cpu_manager.go:414] "containerMap: removing container" podUID="f08ebb94-561c-42b6-9430-3c1d78c5fc8a" containerName="cleanup"
Oct 30 13:30:47.660486 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.660475    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f08ebb94-561c-42b6-9430-3c1d78c5fc8a" containerName="cleanup"
Oct 30 13:30:47.660579 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.660570    4364 cpu_manager.go:414] "containerMap: removing container" podUID="a0239aaa-e1bf-4ad8-bf54-42920cc8009c" containerName="cleanup"
Oct 30 13:30:47.660708 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.660695    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0239aaa-e1bf-4ad8-bf54-42920cc8009c" containerName="cleanup"
Oct 30 13:30:47.660803 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.660795    4364 cpu_manager.go:414] "containerMap: removing container" podUID="d10daaa5-7cac-4205-8638-75ede670c28f" containerName="cleanup"
Oct 30 13:30:47.660881 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.660870    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d10daaa5-7cac-4205-8638-75ede670c28f" containerName="cleanup"
Oct 30 13:30:47.660958 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.660949    4364 cpu_manager.go:414] "containerMap: removing container" podUID="ff661c30-4a5a-4a98-b640-3b2c3e4143e1" containerName="cleanup"
Oct 30 13:30:47.661031 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.661021    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff661c30-4a5a-4a98-b640-3b2c3e4143e1" containerName="cleanup"
Oct 30 13:30:47.661114 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.661105    4364 cpu_manager.go:414] "containerMap: removing container" podUID="f9846347-72dc-472f-8bb1-57d39f1be48b" containerName="istio-proxy"
Oct 30 13:30:47.661186 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.661177    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9846347-72dc-472f-8bb1-57d39f1be48b" containerName="istio-proxy"
Oct 30 13:30:47.661271 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.661263    4364 cpu_manager.go:414] "containerMap: removing container" podUID="ff661c30-4a5a-4a98-b640-3b2c3e4143e1" containerName="cleanup"
Oct 30 13:30:47.661355 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.661343    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff661c30-4a5a-4a98-b640-3b2c3e4143e1" containerName="cleanup"
Oct 30 13:30:47.661642 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.661434    4364 cpu_manager.go:414] "containerMap: removing container" podUID="f08ebb94-561c-42b6-9430-3c1d78c5fc8a" containerName="cleanup"
Oct 30 13:30:47.662922 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.662903    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f08ebb94-561c-42b6-9430-3c1d78c5fc8a" containerName="cleanup"
Oct 30 13:30:47.663042 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.663027    4364 cpu_manager.go:414] "containerMap: removing container" podUID="f9846347-72dc-472f-8bb1-57d39f1be48b" containerName="istio-proxy"
Oct 30 13:30:47.663042 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.663041    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9846347-72dc-472f-8bb1-57d39f1be48b" containerName="istio-proxy"
Oct 30 13:30:47.663201 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.663067    4364 cpu_manager.go:414] "containerMap: removing container" podUID="f08ebb94-561c-42b6-9430-3c1d78c5fc8a" containerName="cleanup"
Oct 30 13:30:47.663201 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.663074    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f08ebb94-561c-42b6-9430-3c1d78c5fc8a" containerName="cleanup"
Oct 30 13:30:47.663201 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.663086    4364 cpu_manager.go:414] "containerMap: removing container" podUID="0fa046ef-c365-47a1-976f-18f12312ea4e" containerName="cleanup"
Oct 30 13:30:47.663201 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.663093    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fa046ef-c365-47a1-976f-18f12312ea4e" containerName="cleanup"
Oct 30 13:30:47.663201 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.663107    4364 cpu_manager.go:414] "containerMap: removing container" podUID="0fa046ef-c365-47a1-976f-18f12312ea4e" containerName="cleanup"
Oct 30 13:30:47.663201 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.663114    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fa046ef-c365-47a1-976f-18f12312ea4e" containerName="cleanup"
Oct 30 13:30:47.663201 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.663126    4364 cpu_manager.go:414] "containerMap: removing container" podUID="17dc6af9-2a39-48fe-a3dc-405f9f9f1ea0" containerName="cleanup"
Oct 30 13:30:47.663201 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.663133    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17dc6af9-2a39-48fe-a3dc-405f9f9f1ea0" containerName="cleanup"
Oct 30 13:30:47.663201 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.663152    4364 cpu_manager.go:414] "containerMap: removing container" podUID="d10daaa5-7cac-4205-8638-75ede670c28f" containerName="cleanup"
Oct 30 13:30:47.663201 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.663160    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d10daaa5-7cac-4205-8638-75ede670c28f" containerName="cleanup"
Oct 30 13:30:47.663201 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.663173    4364 cpu_manager.go:414] "containerMap: removing container" podUID="ace885ec-090f-469d-ba45-23bd11e84d9e" containerName="cleanup"
Oct 30 13:30:47.663201 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.663182    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ace885ec-090f-469d-ba45-23bd11e84d9e" containerName="cleanup"
Oct 30 13:30:47.663201 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.663197    4364 cpu_manager.go:414] "containerMap: removing container" podUID="ace885ec-090f-469d-ba45-23bd11e84d9e" containerName="cleanup"
Oct 30 13:30:47.663201 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.663203    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ace885ec-090f-469d-ba45-23bd11e84d9e" containerName="cleanup"
Oct 30 13:30:47.663976 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.663225    4364 cpu_manager.go:414] "containerMap: removing container" podUID="ace885ec-090f-469d-ba45-23bd11e84d9e" containerName="cleanup"
Oct 30 13:30:47.663976 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.663232    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ace885ec-090f-469d-ba45-23bd11e84d9e" containerName="cleanup"
Oct 30 13:30:47.663976 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.663252    4364 cpu_manager.go:414] "containerMap: removing container" podUID="ff661c30-4a5a-4a98-b640-3b2c3e4143e1" containerName="cleanup"
Oct 30 13:30:47.663976 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.663259    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff661c30-4a5a-4a98-b640-3b2c3e4143e1" containerName="cleanup"
Oct 30 13:30:47.663976 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.663269    4364 cpu_manager.go:414] "containerMap: removing container" podUID="17dc6af9-2a39-48fe-a3dc-405f9f9f1ea0" containerName="cleanup"
Oct 30 13:30:47.663976 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.663275    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17dc6af9-2a39-48fe-a3dc-405f9f9f1ea0" containerName="cleanup"
Oct 30 13:30:47.663976 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.663295    4364 cpu_manager.go:414] "containerMap: removing container" podUID="a0239aaa-e1bf-4ad8-bf54-42920cc8009c" containerName="cleanup"
Oct 30 13:30:47.663976 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.663301    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0239aaa-e1bf-4ad8-bf54-42920cc8009c" containerName="cleanup"
Oct 30 13:30:47.663976 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.663512    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="f08ebb94-561c-42b6-9430-3c1d78c5fc8a" containerName="cleanup"
Oct 30 13:30:47.663976 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.663529    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9846347-72dc-472f-8bb1-57d39f1be48b" containerName="istio-proxy"
Oct 30 13:30:47.663976 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.663544    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="f08ebb94-561c-42b6-9430-3c1d78c5fc8a" containerName="cleanup"
Oct 30 13:30:47.663976 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.663554    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fa046ef-c365-47a1-976f-18f12312ea4e" containerName="cleanup"
Oct 30 13:30:47.663976 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.663562    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fa046ef-c365-47a1-976f-18f12312ea4e" containerName="cleanup"
Oct 30 13:30:47.663976 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.663572    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="17dc6af9-2a39-48fe-a3dc-405f9f9f1ea0" containerName="cleanup"
Oct 30 13:30:47.663976 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.663581    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="d10daaa5-7cac-4205-8638-75ede670c28f" containerName="cleanup"
Oct 30 13:30:47.663976 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.663589    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="ace885ec-090f-469d-ba45-23bd11e84d9e" containerName="cleanup"
Oct 30 13:30:47.663976 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.663614    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="ace885ec-090f-469d-ba45-23bd11e84d9e" containerName="cleanup"
Oct 30 13:30:47.663976 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.663624    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0239aaa-e1bf-4ad8-bf54-42920cc8009c" containerName="cleanup"
Oct 30 13:30:47.663976 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.663635    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff661c30-4a5a-4a98-b640-3b2c3e4143e1" containerName="cleanup"
Oct 30 13:30:47.663976 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.663644    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="17dc6af9-2a39-48fe-a3dc-405f9f9f1ea0" containerName="cleanup"
Oct 30 13:30:47.663976 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.663652    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="d10daaa5-7cac-4205-8638-75ede670c28f" containerName="cleanup"
Oct 30 13:30:47.663976 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.663662    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0239aaa-e1bf-4ad8-bf54-42920cc8009c" containerName="cleanup"
Oct 30 13:30:47.663976 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.663674    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0239aaa-e1bf-4ad8-bf54-42920cc8009c" containerName="cleanup"
Oct 30 13:30:47.663976 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.663685    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff661c30-4a5a-4a98-b640-3b2c3e4143e1" containerName="cleanup"
Oct 30 13:30:47.663976 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.663695    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff661c30-4a5a-4a98-b640-3b2c3e4143e1" containerName="cleanup"
Oct 30 13:30:47.670368 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.670337    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc"
Oct 30 13:30:47.685770 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.685750    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc"]
Oct 30 13:30:47.810733 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.810700    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bfddc61-f2f3-4f1a-bb6e-83830a2a058d-istiod-ca-cert\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc\" (UID: \"0bfddc61-f2f3-4f1a-bb6e-83830a2a058d\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc"
Oct 30 13:30:47.810910 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.810743    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bfddc61-f2f3-4f1a-bb6e-83830a2a058d-workload-socket\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc\" (UID: \"0bfddc61-f2f3-4f1a-bb6e-83830a2a058d\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc"
Oct 30 13:30:47.810910 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.810878    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bfddc61-f2f3-4f1a-bb6e-83830a2a058d-istio-envoy\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc\" (UID: \"0bfddc61-f2f3-4f1a-bb6e-83830a2a058d\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc"
Oct 30 13:30:47.810998 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.810933    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bfddc61-f2f3-4f1a-bb6e-83830a2a058d-workload-certs\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc\" (UID: \"0bfddc61-f2f3-4f1a-bb6e-83830a2a058d\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc"
Oct 30 13:30:47.810998 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.810963    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bfddc61-f2f3-4f1a-bb6e-83830a2a058d-istio-token\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc\" (UID: \"0bfddc61-f2f3-4f1a-bb6e-83830a2a058d\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc"
Oct 30 13:30:47.811083 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.811055    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bfddc61-f2f3-4f1a-bb6e-83830a2a058d-credential-socket\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc\" (UID: \"0bfddc61-f2f3-4f1a-bb6e-83830a2a058d\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc"
Oct 30 13:30:47.811123 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.811091    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dsn6\" (UniqueName: \"kubernetes.io/projected/0bfddc61-f2f3-4f1a-bb6e-83830a2a058d-kube-api-access-7dsn6\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc\" (UID: \"0bfddc61-f2f3-4f1a-bb6e-83830a2a058d\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc"
Oct 30 13:30:47.811171 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.811153    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bfddc61-f2f3-4f1a-bb6e-83830a2a058d-istio-data\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc\" (UID: \"0bfddc61-f2f3-4f1a-bb6e-83830a2a058d\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc"
Oct 30 13:30:47.811212 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.811178    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bfddc61-f2f3-4f1a-bb6e-83830a2a058d-istio-podinfo\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc\" (UID: \"0bfddc61-f2f3-4f1a-bb6e-83830a2a058d\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc"
Oct 30 13:30:47.912894 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.912812    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bfddc61-f2f3-4f1a-bb6e-83830a2a058d-workload-certs\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc\" (UID: \"0bfddc61-f2f3-4f1a-bb6e-83830a2a058d\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc"
Oct 30 13:30:47.912894 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.912865    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bfddc61-f2f3-4f1a-bb6e-83830a2a058d-istio-token\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc\" (UID: \"0bfddc61-f2f3-4f1a-bb6e-83830a2a058d\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc"
Oct 30 13:30:47.913137 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.912940    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bfddc61-f2f3-4f1a-bb6e-83830a2a058d-credential-socket\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc\" (UID: \"0bfddc61-f2f3-4f1a-bb6e-83830a2a058d\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc"
Oct 30 13:30:47.913137 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.912980    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dsn6\" (UniqueName: \"kubernetes.io/projected/0bfddc61-f2f3-4f1a-bb6e-83830a2a058d-kube-api-access-7dsn6\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc\" (UID: \"0bfddc61-f2f3-4f1a-bb6e-83830a2a058d\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc"
Oct 30 13:30:47.913137 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.913009    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bfddc61-f2f3-4f1a-bb6e-83830a2a058d-istio-data\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc\" (UID: \"0bfddc61-f2f3-4f1a-bb6e-83830a2a058d\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc"
Oct 30 13:30:47.913319 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.913129    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bfddc61-f2f3-4f1a-bb6e-83830a2a058d-istio-podinfo\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc\" (UID: \"0bfddc61-f2f3-4f1a-bb6e-83830a2a058d\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc"
Oct 30 13:30:47.913319 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.913265    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bfddc61-f2f3-4f1a-bb6e-83830a2a058d-workload-certs\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc\" (UID: \"0bfddc61-f2f3-4f1a-bb6e-83830a2a058d\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc"
Oct 30 13:30:47.913319 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.913289    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bfddc61-f2f3-4f1a-bb6e-83830a2a058d-istiod-ca-cert\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc\" (UID: \"0bfddc61-f2f3-4f1a-bb6e-83830a2a058d\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc"
Oct 30 13:30:47.913450 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.913321    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bfddc61-f2f3-4f1a-bb6e-83830a2a058d-workload-socket\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc\" (UID: \"0bfddc61-f2f3-4f1a-bb6e-83830a2a058d\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc"
Oct 30 13:30:47.913450 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.913323    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bfddc61-f2f3-4f1a-bb6e-83830a2a058d-istio-data\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc\" (UID: \"0bfddc61-f2f3-4f1a-bb6e-83830a2a058d\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc"
Oct 30 13:30:47.913450 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.913381    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bfddc61-f2f3-4f1a-bb6e-83830a2a058d-istio-envoy\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc\" (UID: \"0bfddc61-f2f3-4f1a-bb6e-83830a2a058d\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc"
Oct 30 13:30:47.913589 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.913379    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bfddc61-f2f3-4f1a-bb6e-83830a2a058d-credential-socket\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc\" (UID: \"0bfddc61-f2f3-4f1a-bb6e-83830a2a058d\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc"
Oct 30 13:30:47.913697 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.913637    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bfddc61-f2f3-4f1a-bb6e-83830a2a058d-workload-socket\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc\" (UID: \"0bfddc61-f2f3-4f1a-bb6e-83830a2a058d\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc"
Oct 30 13:30:47.913975 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.913955    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bfddc61-f2f3-4f1a-bb6e-83830a2a058d-istiod-ca-cert\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc\" (UID: \"0bfddc61-f2f3-4f1a-bb6e-83830a2a058d\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc"
Oct 30 13:30:47.915570 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.915548    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bfddc61-f2f3-4f1a-bb6e-83830a2a058d-istio-envoy\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc\" (UID: \"0bfddc61-f2f3-4f1a-bb6e-83830a2a058d\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc"
Oct 30 13:30:47.915698 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.915653    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bfddc61-f2f3-4f1a-bb6e-83830a2a058d-istio-podinfo\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc\" (UID: \"0bfddc61-f2f3-4f1a-bb6e-83830a2a058d\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc"
Oct 30 13:30:47.936385 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.936350    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bfddc61-f2f3-4f1a-bb6e-83830a2a058d-istio-token\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc\" (UID: \"0bfddc61-f2f3-4f1a-bb6e-83830a2a058d\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc"
Oct 30 13:30:47.936508 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.936456    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dsn6\" (UniqueName: \"kubernetes.io/projected/0bfddc61-f2f3-4f1a-bb6e-83830a2a058d-kube-api-access-7dsn6\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc\" (UID: \"0bfddc61-f2f3-4f1a-bb6e-83830a2a058d\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc"
Oct 30 13:30:47.988495 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:47.988469    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc"
Oct 30 13:30:48.423453 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:48.422570    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc"]
Oct 30 13:30:48.434470 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:30:48.434437    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bfddc61_f2f3_4f1a_bb6e_83830a2a058d.slice/crio-14736826b40baf80e716a12295c0298840fc107b32bab63bc87d1f1fc6d9ee8e WatchSource:0}: Error finding container 14736826b40baf80e716a12295c0298840fc107b32bab63bc87d1f1fc6d9ee8e: Status 404 returned error can't find the container with id 14736826b40baf80e716a12295c0298840fc107b32bab63bc87d1f1fc6d9ee8e
Oct 30 13:30:48.446128 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:48.438155    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:30:48.446128 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:48.438285    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:30:48.446128 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:48.438350    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:30:49.202748 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:49.202714    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc" event={"ID":"0bfddc61-f2f3-4f1a-bb6e-83830a2a058d","Type":"ContainerStarted","Data":"d1085ae44d933ca483e2acb81446c148d704921311f4910228c917ed6dadb2cd"}
Oct 30 13:30:49.203218 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:49.203201    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc" event={"ID":"0bfddc61-f2f3-4f1a-bb6e-83830a2a058d","Type":"ContainerStarted","Data":"14736826b40baf80e716a12295c0298840fc107b32bab63bc87d1f1fc6d9ee8e"}
Oct 30 13:30:49.297621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:49.279889    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc" podStartSLOduration=2.279870733 podStartE2EDuration="2.279870733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:49.266514243 +0000 UTC m=+2348.432878891" watchObservedRunningTime="2026-10-30 13:30:49.279870733 +0000 UTC m=+2348.446235382"
Oct 30 13:30:50.005620 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:50.002812    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc"
Oct 30 13:30:50.013624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:50.010904    4364 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body=
Oct 30 13:30:50.013624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:50.010969    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc" podUID="0bfddc61-f2f3-4f1a-bb6e-83830a2a058d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused"
Oct 30 13:30:50.712410 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:50.712360    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv"]
Oct 30 13:30:50.713129 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:50.713113    4364 cpu_manager.go:414] "containerMap: removing container" podUID="d10daaa5-7cac-4205-8638-75ede670c28f" containerName="cleanup"
Oct 30 13:30:50.713175 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:50.713131    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d10daaa5-7cac-4205-8638-75ede670c28f" containerName="cleanup"
Oct 30 13:30:50.713175 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:50.713169    4364 cpu_manager.go:414] "containerMap: removing container" podUID="a0239aaa-e1bf-4ad8-bf54-42920cc8009c" containerName="cleanup"
Oct 30 13:30:50.713175 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:50.713174    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0239aaa-e1bf-4ad8-bf54-42920cc8009c" containerName="cleanup"
Oct 30 13:30:50.713295 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:50.713286    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="f08ebb94-561c-42b6-9430-3c1d78c5fc8a" containerName="cleanup"
Oct 30 13:30:50.713332 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:50.713299    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9846347-72dc-472f-8bb1-57d39f1be48b" containerName="istio-proxy"
Oct 30 13:30:50.713332 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:50.713308    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fa046ef-c365-47a1-976f-18f12312ea4e" containerName="cleanup"
Oct 30 13:30:50.713332 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:50.713316    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="d10daaa5-7cac-4205-8638-75ede670c28f" containerName="cleanup"
Oct 30 13:30:50.713332 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:50.713323    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="ace885ec-090f-469d-ba45-23bd11e84d9e" containerName="cleanup"
Oct 30 13:30:50.713332 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:50.713331    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="17dc6af9-2a39-48fe-a3dc-405f9f9f1ea0" containerName="cleanup"
Oct 30 13:30:50.717273 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:50.717252    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv"
Oct 30 13:30:50.749382 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:50.749351    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv"]
Oct 30 13:30:50.867650 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:50.867586    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7090457f-3bbf-4295-aa35-9dedc02d1b61-istiod-ca-cert\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv\" (UID: \"7090457f-3bbf-4295-aa35-9dedc02d1b61\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv"
Oct 30 13:30:50.867855 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:50.867701    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7090457f-3bbf-4295-aa35-9dedc02d1b61-workload-certs\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv\" (UID: \"7090457f-3bbf-4295-aa35-9dedc02d1b61\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv"
Oct 30 13:30:50.867855 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:50.867778    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7090457f-3bbf-4295-aa35-9dedc02d1b61-workload-socket\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv\" (UID: \"7090457f-3bbf-4295-aa35-9dedc02d1b61\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv"
Oct 30 13:30:50.867855 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:50.867843    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7090457f-3bbf-4295-aa35-9dedc02d1b61-istio-token\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv\" (UID: \"7090457f-3bbf-4295-aa35-9dedc02d1b61\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv"
Oct 30 13:30:50.868035 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:50.867874    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7090457f-3bbf-4295-aa35-9dedc02d1b61-istio-data\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv\" (UID: \"7090457f-3bbf-4295-aa35-9dedc02d1b61\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv"
Oct 30 13:30:50.868035 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:50.867930    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p97n9\" (UniqueName: \"kubernetes.io/projected/7090457f-3bbf-4295-aa35-9dedc02d1b61-kube-api-access-p97n9\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv\" (UID: \"7090457f-3bbf-4295-aa35-9dedc02d1b61\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv"
Oct 30 13:30:50.868158 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:50.868047    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7090457f-3bbf-4295-aa35-9dedc02d1b61-istio-envoy\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv\" (UID: \"7090457f-3bbf-4295-aa35-9dedc02d1b61\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv"
Oct 30 13:30:50.868158 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:50.868141    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7090457f-3bbf-4295-aa35-9dedc02d1b61-credential-socket\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv\" (UID: \"7090457f-3bbf-4295-aa35-9dedc02d1b61\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv"
Oct 30 13:30:50.868264 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:50.868167    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7090457f-3bbf-4295-aa35-9dedc02d1b61-istio-podinfo\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv\" (UID: \"7090457f-3bbf-4295-aa35-9dedc02d1b61\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv"
Oct 30 13:30:50.970261 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:50.970169    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7090457f-3bbf-4295-aa35-9dedc02d1b61-istio-token\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv\" (UID: \"7090457f-3bbf-4295-aa35-9dedc02d1b61\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv"
Oct 30 13:30:50.970261 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:50.970209    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7090457f-3bbf-4295-aa35-9dedc02d1b61-istio-data\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv\" (UID: \"7090457f-3bbf-4295-aa35-9dedc02d1b61\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv"
Oct 30 13:30:50.970261 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:50.970250    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p97n9\" (UniqueName: \"kubernetes.io/projected/7090457f-3bbf-4295-aa35-9dedc02d1b61-kube-api-access-p97n9\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv\" (UID: \"7090457f-3bbf-4295-aa35-9dedc02d1b61\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv"
Oct 30 13:30:50.970574 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:50.970328    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7090457f-3bbf-4295-aa35-9dedc02d1b61-istio-envoy\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv\" (UID: \"7090457f-3bbf-4295-aa35-9dedc02d1b61\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv"
Oct 30 13:30:50.970574 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:50.970390    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7090457f-3bbf-4295-aa35-9dedc02d1b61-credential-socket\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv\" (UID: \"7090457f-3bbf-4295-aa35-9dedc02d1b61\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv"
Oct 30 13:30:50.970574 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:50.970484    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7090457f-3bbf-4295-aa35-9dedc02d1b61-istio-podinfo\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv\" (UID: \"7090457f-3bbf-4295-aa35-9dedc02d1b61\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv"
Oct 30 13:30:50.970774 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:50.970640    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7090457f-3bbf-4295-aa35-9dedc02d1b61-istiod-ca-cert\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv\" (UID: \"7090457f-3bbf-4295-aa35-9dedc02d1b61\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv"
Oct 30 13:30:50.970774 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:50.970722    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7090457f-3bbf-4295-aa35-9dedc02d1b61-workload-certs\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv\" (UID: \"7090457f-3bbf-4295-aa35-9dedc02d1b61\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv"
Oct 30 13:30:50.970774 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:50.970725    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7090457f-3bbf-4295-aa35-9dedc02d1b61-istio-data\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv\" (UID: \"7090457f-3bbf-4295-aa35-9dedc02d1b61\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv"
Oct 30 13:30:50.970934 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:50.970769    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7090457f-3bbf-4295-aa35-9dedc02d1b61-credential-socket\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv\" (UID: \"7090457f-3bbf-4295-aa35-9dedc02d1b61\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv"
Oct 30 13:30:50.970934 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:50.970845    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7090457f-3bbf-4295-aa35-9dedc02d1b61-workload-socket\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv\" (UID: \"7090457f-3bbf-4295-aa35-9dedc02d1b61\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv"
Oct 30 13:30:50.971041 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:50.970991    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7090457f-3bbf-4295-aa35-9dedc02d1b61-workload-certs\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv\" (UID: \"7090457f-3bbf-4295-aa35-9dedc02d1b61\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv"
Oct 30 13:30:50.971200 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:50.971177    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7090457f-3bbf-4295-aa35-9dedc02d1b61-workload-socket\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv\" (UID: \"7090457f-3bbf-4295-aa35-9dedc02d1b61\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv"
Oct 30 13:30:50.971370 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:50.971354    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7090457f-3bbf-4295-aa35-9dedc02d1b61-istiod-ca-cert\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv\" (UID: \"7090457f-3bbf-4295-aa35-9dedc02d1b61\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv"
Oct 30 13:30:50.972785 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:50.972760    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7090457f-3bbf-4295-aa35-9dedc02d1b61-istio-envoy\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv\" (UID: \"7090457f-3bbf-4295-aa35-9dedc02d1b61\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv"
Oct 30 13:30:50.973017 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:50.972998    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7090457f-3bbf-4295-aa35-9dedc02d1b61-istio-podinfo\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv\" (UID: \"7090457f-3bbf-4295-aa35-9dedc02d1b61\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv"
Oct 30 13:30:50.999627 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:50.992532    4364 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body=
Oct 30 13:30:50.999627 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:50.992631    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc" podUID="0bfddc61-f2f3-4f1a-bb6e-83830a2a058d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused"
Oct 30 13:30:51.008040 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:51.007979    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7090457f-3bbf-4295-aa35-9dedc02d1b61-istio-token\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv\" (UID: \"7090457f-3bbf-4295-aa35-9dedc02d1b61\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv"
Oct 30 13:30:51.008831 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:51.008804    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p97n9\" (UniqueName: \"kubernetes.io/projected/7090457f-3bbf-4295-aa35-9dedc02d1b61-kube-api-access-p97n9\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv\" (UID: \"7090457f-3bbf-4295-aa35-9dedc02d1b61\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv"
Oct 30 13:30:51.034205 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:51.034174    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv"
Oct 30 13:30:51.432739 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:51.432707    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv"]
Oct 30 13:30:51.443386 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:30:51.443340    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7090457f_3bbf_4295_aa35_9dedc02d1b61.slice/crio-185cba57705b65253a8f05e0d6a7b90b0bcc35314e703edacfe7893680a11e48 WatchSource:0}: Error finding container 185cba57705b65253a8f05e0d6a7b90b0bcc35314e703edacfe7893680a11e48: Status 404 returned error can't find the container with id 185cba57705b65253a8f05e0d6a7b90b0bcc35314e703edacfe7893680a11e48
Oct 30 13:30:51.446739 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:51.446707    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:30:51.446828 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:51.446786    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:30:51.446946 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:51.446928    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:30:51.993619 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:51.992321    4364 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body=
Oct 30 13:30:51.993619 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:51.992389    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc" podUID="0bfddc61-f2f3-4f1a-bb6e-83830a2a058d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused"
Oct 30 13:30:52.238142 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:52.238100    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv" event={"ID":"7090457f-3bbf-4295-aa35-9dedc02d1b61","Type":"ContainerStarted","Data":"d37963fcb3d6d226584f31b5f028e9f4bd77fc3f6b346e1f40eafd201535a154"}
Oct 30 13:30:52.238321 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:52.238164    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv" event={"ID":"7090457f-3bbf-4295-aa35-9dedc02d1b61","Type":"ContainerStarted","Data":"185cba57705b65253a8f05e0d6a7b90b0bcc35314e703edacfe7893680a11e48"}
Oct 30 13:30:52.313683 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:52.313428    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv" podStartSLOduration=2.3134051270000002 podStartE2EDuration="2.313405127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:52.301916582 +0000 UTC m=+2351.468281231" watchObservedRunningTime="2026-10-30 13:30:52.313405127 +0000 UTC m=+2351.479769775"
Oct 30 13:30:53.022622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:53.016074    4364 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body=
Oct 30 13:30:53.022622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:53.016140    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc" podUID="0bfddc61-f2f3-4f1a-bb6e-83830a2a058d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused"
Oct 30 13:30:53.035657 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:53.034366    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv"
Oct 30 13:30:53.035884 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:53.035844    4364 patch_prober.go:29] interesting pod/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Oct 30 13:30:53.035948 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:53.035925    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv" podUID="7090457f-3bbf-4295-aa35-9dedc02d1b61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Oct 30 13:30:53.995651 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:53.995614    4364 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body=
Oct 30 13:30:53.995941 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:53.995911    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc" podUID="0bfddc61-f2f3-4f1a-bb6e-83830a2a058d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused"
Oct 30 13:30:54.035272 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:54.035239    4364 patch_prober.go:29] interesting pod/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Oct 30 13:30:54.035768 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:54.035293    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv" podUID="7090457f-3bbf-4295-aa35-9dedc02d1b61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Oct 30 13:30:55.010622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:55.009793    4364 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body=
Oct 30 13:30:55.010622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:55.009903    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc" podUID="0bfddc61-f2f3-4f1a-bb6e-83830a2a058d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused"
Oct 30 13:30:55.057291 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:55.057255    4364 patch_prober.go:29] interesting pod/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Oct 30 13:30:55.057968 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:55.057882    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv" podUID="7090457f-3bbf-4295-aa35-9dedc02d1b61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Oct 30 13:30:56.005463 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:56.005426    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc"
Oct 30 13:30:56.005930 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:56.005903    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc"
Oct 30 13:30:56.006793 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:56.006768    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc"
Oct 30 13:30:56.035011 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:56.034986    4364 patch_prober.go:29] interesting pod/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Oct 30 13:30:56.035139 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:56.035028    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv" podUID="7090457f-3bbf-4295-aa35-9dedc02d1b61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Oct 30 13:30:57.034530 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:57.034496    4364 patch_prober.go:29] interesting pod/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Oct 30 13:30:57.034989 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:57.034572    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv" podUID="7090457f-3bbf-4295-aa35-9dedc02d1b61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Oct 30 13:30:58.035099 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:58.035061    4364 patch_prober.go:29] interesting pod/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Oct 30 13:30:58.035508 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:58.035132    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv" podUID="7090457f-3bbf-4295-aa35-9dedc02d1b61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Oct 30 13:30:59.039849 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:59.039819    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv"
Oct 30 13:30:59.040290 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:59.040278    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv"
Oct 30 13:30:59.040955 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:30:59.040935    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv"
Oct 30 13:31:06.589621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:06.589420    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:31:11.085620 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:11.082654    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-280238be-w76vf"]
Oct 30 13:31:11.148629 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:11.147878    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-280238be-w76vf"]
Oct 30 13:31:11.531281 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:31:11.531244    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:31:12.971353 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:12.971320    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5895871b-7b5c-4ba0-8b6e-e761ca5b1adf" path="/var/lib/kubelet/pods/5895871b-7b5c-4ba0-8b6e-e761ca5b1adf/volumes"
Oct 30 13:31:19.244982 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:19.244913    4364 patch_prober.go:29] interesting pod/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body=
Oct 30 13:31:19.245368 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:19.244972    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc" podUID="d675cfa9-7df1-446a-984d-a4e1b0dda352" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused"
Oct 30 13:31:19.588216 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:19.588178    4364 generic.go:356] "Generic (PLEG): container finished" podID="d675cfa9-7df1-446a-984d-a4e1b0dda352" containerID="42c7c544365e5d4fb53bee70de1e7283d4ce4c0a13e7c83c2f88f1c1ce4c966f" exitCode=0
Oct 30 13:31:19.588395 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:19.588251    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc" event={"ID":"d675cfa9-7df1-446a-984d-a4e1b0dda352","Type":"ContainerDied","Data":"42c7c544365e5d4fb53bee70de1e7283d4ce4c0a13e7c83c2f88f1c1ce4c966f"}
Oct 30 13:31:19.588707 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:19.588688    4364 scope.go:122] "RemoveContainer" containerID="42c7c544365e5d4fb53bee70de1e7283d4ce4c0a13e7c83c2f88f1c1ce4c966f"
Oct 30 13:31:19.590401 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:19.590370    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:31:19.590483 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:19.590441    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:31:19.590483 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:19.590476    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:31:20.616258 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:20.616220    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc" event={"ID":"d675cfa9-7df1-446a-984d-a4e1b0dda352","Type":"ContainerStarted","Data":"8e3c0c6a8fc0b33845319cb8efd3eb01abefade605420501182f7c59c42c47ce"}
Oct 30 13:31:21.200668 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:21.199903    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc"
Oct 30 13:31:21.200668 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:21.200430    4364 patch_prober.go:29] interesting pod/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body=
Oct 30 13:31:21.200668 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:21.200510    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc" podUID="d675cfa9-7df1-446a-984d-a4e1b0dda352" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused"
Oct 30 13:31:22.209624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:22.204986    4364 patch_prober.go:29] interesting pod/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body=
Oct 30 13:31:22.209624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:22.205048    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc" podUID="d675cfa9-7df1-446a-984d-a4e1b0dda352" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused"
Oct 30 13:31:23.204018 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:23.203982    4364 patch_prober.go:29] interesting pod/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body=
Oct 30 13:31:23.204200 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:23.204036    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc" podUID="d675cfa9-7df1-446a-984d-a4e1b0dda352" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused"
Oct 30 13:31:24.203176 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:24.202970    4364 patch_prober.go:29] interesting pod/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body=
Oct 30 13:31:24.203884 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:24.203854    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc" podUID="d675cfa9-7df1-446a-984d-a4e1b0dda352" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused"
Oct 30 13:31:25.202622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:25.200998    4364 patch_prober.go:29] interesting pod/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body=
Oct 30 13:31:25.202622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:25.201174    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc" podUID="d675cfa9-7df1-446a-984d-a4e1b0dda352" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused"
Oct 30 13:31:26.204318 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:26.204285    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc"
Oct 30 13:31:26.204713 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:26.204646    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc"
Oct 30 13:31:26.205257 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:26.205240    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc"
Oct 30 13:31:32.497479 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:32.497444    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-8d82d5/sub-test-model-8d82d5-kserve-d4df748cf-hqr5x"]
Oct 30 13:31:34.639334 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:34.639305    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk"]
Oct 30 13:31:34.643842 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:34.643820    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk"
Oct 30 13:31:34.663523 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:34.663502    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-6da60a-openshift-default-dockercfg-njt6m\""
Oct 30 13:31:34.669825 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:34.669798    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fkcn\" (UniqueName: \"kubernetes.io/projected/5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b-kube-api-access-9fkcn\") pod \"e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk\" (UID: \"5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b\") " pod="openshift-ingress/e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk"
Oct 30 13:31:34.669947 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:34.669850    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b-workload-socket\") pod \"e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk\" (UID: \"5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b\") " pod="openshift-ingress/e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk"
Oct 30 13:31:34.670045 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:34.670023    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b-istio-token\") pod \"e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk\" (UID: \"5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b\") " pod="openshift-ingress/e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk"
Oct 30 13:31:34.670230 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:34.670206    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b-istio-podinfo\") pod \"e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk\" (UID: \"5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b\") " pod="openshift-ingress/e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk"
Oct 30 13:31:34.670365 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:34.670346    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b-workload-certs\") pod \"e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk\" (UID: \"5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b\") " pod="openshift-ingress/e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk"
Oct 30 13:31:34.670419 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:34.670394    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b-istiod-ca-cert\") pod \"e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk\" (UID: \"5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b\") " pod="openshift-ingress/e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk"
Oct 30 13:31:34.670498 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:34.670478    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b-istio-data\") pod \"e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk\" (UID: \"5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b\") " pod="openshift-ingress/e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk"
Oct 30 13:31:34.670569 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:34.670535    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b-credential-socket\") pod \"e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk\" (UID: \"5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b\") " pod="openshift-ingress/e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk"
Oct 30 13:31:34.670692 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:34.670667    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b-istio-envoy\") pod \"e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk\" (UID: \"5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b\") " pod="openshift-ingress/e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk"
Oct 30 13:31:34.682677 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:34.682651    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk"]
Oct 30 13:31:34.772724 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:34.772691    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b-workload-certs\") pod \"e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk\" (UID: \"5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b\") " pod="openshift-ingress/e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk"
Oct 30 13:31:34.772903 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:34.772747    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b-istiod-ca-cert\") pod \"e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk\" (UID: \"5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b\") " pod="openshift-ingress/e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk"
Oct 30 13:31:34.772903 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:34.772802    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b-istio-data\") pod \"e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk\" (UID: \"5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b\") " pod="openshift-ingress/e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk"
Oct 30 13:31:34.772903 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:34.772858    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b-credential-socket\") pod \"e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk\" (UID: \"5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b\") " pod="openshift-ingress/e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk"
Oct 30 13:31:34.772903 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:34.772895    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b-istio-envoy\") pod \"e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk\" (UID: \"5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b\") " pod="openshift-ingress/e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk"
Oct 30 13:31:34.773202 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:34.772995    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fkcn\" (UniqueName: \"kubernetes.io/projected/5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b-kube-api-access-9fkcn\") pod \"e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk\" (UID: \"5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b\") " pod="openshift-ingress/e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk"
Oct 30 13:31:34.773202 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:34.773033    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b-workload-socket\") pod \"e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk\" (UID: \"5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b\") " pod="openshift-ingress/e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk"
Oct 30 13:31:34.773202 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:34.773137    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b-istio-token\") pod \"e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk\" (UID: \"5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b\") " pod="openshift-ingress/e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk"
Oct 30 13:31:34.773372 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:34.773209    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b-workload-certs\") pod \"e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk\" (UID: \"5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b\") " pod="openshift-ingress/e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk"
Oct 30 13:31:34.773372 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:34.773236    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b-istio-data\") pod \"e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk\" (UID: \"5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b\") " pod="openshift-ingress/e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk"
Oct 30 13:31:34.773372 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:34.773309    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b-istio-podinfo\") pod \"e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk\" (UID: \"5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b\") " pod="openshift-ingress/e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk"
Oct 30 13:31:34.773660 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:34.773632    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b-workload-socket\") pod \"e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk\" (UID: \"5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b\") " pod="openshift-ingress/e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk"
Oct 30 13:31:34.773660 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:34.773649    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b-credential-socket\") pod \"e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk\" (UID: \"5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b\") " pod="openshift-ingress/e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk"
Oct 30 13:31:34.773924 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:34.773906    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b-istiod-ca-cert\") pod \"e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk\" (UID: \"5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b\") " pod="openshift-ingress/e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk"
Oct 30 13:31:34.775688 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:34.775663    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b-istio-envoy\") pod \"e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk\" (UID: \"5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b\") " pod="openshift-ingress/e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk"
Oct 30 13:31:34.775881 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:34.775865    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b-istio-podinfo\") pod \"e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk\" (UID: \"5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b\") " pod="openshift-ingress/e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk"
Oct 30 13:31:34.796299 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:34.796276    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fkcn\" (UniqueName: \"kubernetes.io/projected/5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b-kube-api-access-9fkcn\") pod \"e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk\" (UID: \"5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b\") " pod="openshift-ingress/e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk"
Oct 30 13:31:34.797276 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:34.797254    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b-istio-token\") pod \"e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk\" (UID: \"5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b\") " pod="openshift-ingress/e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk"
Oct 30 13:31:34.990733 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:34.966540    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk"
Oct 30 13:31:35.450783 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:35.450738    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk"]
Oct 30 13:31:35.451176 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:31:35.451152    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e958d3b_8a4e_41d2_b7ad_fd60d1fc9e9b.slice/crio-898a68892aeaa7fffeac71fa7e7297b03ceafe9618ea2bc0dd2796886f63dc36 WatchSource:0}: Error finding container 898a68892aeaa7fffeac71fa7e7297b03ceafe9618ea2bc0dd2796886f63dc36: Status 404 returned error can't find the container with id 898a68892aeaa7fffeac71fa7e7297b03ceafe9618ea2bc0dd2796886f63dc36
Oct 30 13:31:35.453918 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:35.453884    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:31:35.454007 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:35.453952    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:31:35.454007 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:35.453981    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:31:35.758982 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:35.758900    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk" event={"ID":"5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b","Type":"ContainerStarted","Data":"15629b602c0d92a0beebeec30ea9d71e7a69fceabc26bf40d512995430aadba2"}
Oct 30 13:31:35.758982 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:35.758958    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk" event={"ID":"5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b","Type":"ContainerStarted","Data":"898a68892aeaa7fffeac71fa7e7297b03ceafe9618ea2bc0dd2796886f63dc36"}
Oct 30 13:31:35.809447 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:35.809392    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk" podStartSLOduration=1.809360865 podStartE2EDuration="1.809360865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:31:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:31:35.804001252 +0000 UTC m=+2394.970365905" watchObservedRunningTime="2026-10-30 13:31:35.809360865 +0000 UTC m=+2394.975725514"
Oct 30 13:31:35.974776 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:35.974741    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk"
Oct 30 13:31:37.117631 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:37.114922    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk"
Oct 30 13:31:37.774744 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:37.774715    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk"
Oct 30 13:31:37.775737 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:37.775714    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk"
Oct 30 13:31:39.529751 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:39.499679    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr" podUID="446c12df-de12-4c7c-8c71-dfed93cc33f1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded"
Oct 30 13:31:39.586032 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:39.585733    4364 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-b-w2-5a6281/sub-test-model-5a6281-kserve-694f48b47-4j6cv" podUID="10cce01a-1759-4617-bf74-5433cb240a30" containerName="main" probeResult="failure" output="Get \"https://10.133.0.211:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:31:41.505326 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:31:41.505288    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:31:43.411818 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:43.411782    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq"]
Oct 30 13:31:43.412251 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:43.412051    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq" podUID="1d08eaa5-399d-4d90-8a47-3c060abfb552" containerName="istio-proxy" containerID="cri-o://b8c165e33e34b0ca2ad872c8938134b53338fda62cd9c2439b8af09d8650e51f" gracePeriod=30
Oct 30 13:31:43.442060 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:43.442027    4364 scope.go:122] "RemoveContainer" containerID="41a7b49c2fdd70344ac64bb7bd9f750517b497c6dadd7de00758025db0692658"
Oct 30 13:31:43.453251 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:43.453228    4364 scope.go:122] "RemoveContainer" containerID="3e34b90a68dbbcf23877636968714111cb0e1352daf666dcbacc99053b06d538"
Oct 30 13:31:47.520234 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:47.520197    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-8d82d5/sub-test-model-8d82d5-kserve-d4df748cf-hqr5x" podUID="74468d4c-64a8-4b40-8d14-e9954caadc22" containerName="main" containerID="cri-o://924455d80432ad889fadadcb01c34b75d2251f0c9e839c9b1cb1f953cc7865b0" gracePeriod=45
Oct 30 13:31:47.814756 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:47.814726    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-8d82d5/sub-test-model-8d82d5-kserve-d4df748cf-hqr5x"
Oct 30 13:31:47.932314 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:47.932283    4364 generic.go:356] "Generic (PLEG): container finished" podID="74468d4c-64a8-4b40-8d14-e9954caadc22" containerID="924455d80432ad889fadadcb01c34b75d2251f0c9e839c9b1cb1f953cc7865b0" exitCode=0
Oct 30 13:31:47.932584 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:47.932323    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-8d82d5/sub-test-model-8d82d5-kserve-d4df748cf-hqr5x" event={"ID":"74468d4c-64a8-4b40-8d14-e9954caadc22","Type":"ContainerDied","Data":"924455d80432ad889fadadcb01c34b75d2251f0c9e839c9b1cb1f953cc7865b0"}
Oct 30 13:31:47.932584 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:47.932363    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-8d82d5/sub-test-model-8d82d5-kserve-d4df748cf-hqr5x"
Oct 30 13:31:47.932827 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:47.932626    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-8d82d5/sub-test-model-8d82d5-kserve-d4df748cf-hqr5x" event={"ID":"74468d4c-64a8-4b40-8d14-e9954caadc22","Type":"ContainerDied","Data":"fda0c91a180d604c39b4f3d9b7ecb1b2cff9cec1f4eafc7339dc89aa56381852"}
Oct 30 13:31:47.932827 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:47.932662    4364 scope.go:122] "RemoveContainer" containerID="924455d80432ad889fadadcb01c34b75d2251f0c9e839c9b1cb1f953cc7865b0"
Oct 30 13:31:47.937454 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:47.937428    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74468d4c-64a8-4b40-8d14-e9954caadc22-dshm\" (UniqueName: \"kubernetes.io/empty-dir/74468d4c-64a8-4b40-8d14-e9954caadc22-dshm\") pod \"74468d4c-64a8-4b40-8d14-e9954caadc22\" (UID: \"74468d4c-64a8-4b40-8d14-e9954caadc22\") "
Oct 30 13:31:47.937694 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:47.937670    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74468d4c-64a8-4b40-8d14-e9954caadc22-tls-certs\" (UniqueName: \"kubernetes.io/secret/74468d4c-64a8-4b40-8d14-e9954caadc22-tls-certs\") pod \"74468d4c-64a8-4b40-8d14-e9954caadc22\" (UID: \"74468d4c-64a8-4b40-8d14-e9954caadc22\") "
Oct 30 13:31:47.937770 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:47.937744    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74468d4c-64a8-4b40-8d14-e9954caadc22-home\" (UniqueName: \"kubernetes.io/empty-dir/74468d4c-64a8-4b40-8d14-e9954caadc22-home\") pod \"74468d4c-64a8-4b40-8d14-e9954caadc22\" (UID: \"74468d4c-64a8-4b40-8d14-e9954caadc22\") "
Oct 30 13:31:47.937833 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:47.937801    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74468d4c-64a8-4b40-8d14-e9954caadc22-kube-api-access-s4nnx\" (UniqueName: \"kubernetes.io/projected/74468d4c-64a8-4b40-8d14-e9954caadc22-kube-api-access-s4nnx\") pod \"74468d4c-64a8-4b40-8d14-e9954caadc22\" (UID: \"74468d4c-64a8-4b40-8d14-e9954caadc22\") "
Oct 30 13:31:47.937895 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:47.937878    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74468d4c-64a8-4b40-8d14-e9954caadc22-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74468d4c-64a8-4b40-8d14-e9954caadc22-model-cache\") pod \"74468d4c-64a8-4b40-8d14-e9954caadc22\" (UID: \"74468d4c-64a8-4b40-8d14-e9954caadc22\") "
Oct 30 13:31:47.937962 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:47.937901    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74468d4c-64a8-4b40-8d14-e9954caadc22-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74468d4c-64a8-4b40-8d14-e9954caadc22-tmp-dir\") pod \"74468d4c-64a8-4b40-8d14-e9954caadc22\" (UID: \"74468d4c-64a8-4b40-8d14-e9954caadc22\") "
Oct 30 13:31:47.940653 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:47.938229    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74468d4c-64a8-4b40-8d14-e9954caadc22-model-cache" pod "74468d4c-64a8-4b40-8d14-e9954caadc22" (UID: "74468d4c-64a8-4b40-8d14-e9954caadc22"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:31:47.940653 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:47.938380    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74468d4c-64a8-4b40-8d14-e9954caadc22-tmp-dir" pod "74468d4c-64a8-4b40-8d14-e9954caadc22" (UID: "74468d4c-64a8-4b40-8d14-e9954caadc22"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:31:47.940653 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:47.938749    4364 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74468d4c-64a8-4b40-8d14-e9954caadc22-model-cache\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:31:47.940653 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:47.938771    4364 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74468d4c-64a8-4b40-8d14-e9954caadc22-tmp-dir\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:31:47.940653 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:47.938854    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74468d4c-64a8-4b40-8d14-e9954caadc22-home" pod "74468d4c-64a8-4b40-8d14-e9954caadc22" (UID: "74468d4c-64a8-4b40-8d14-e9954caadc22"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:31:47.940653 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:47.940206    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74468d4c-64a8-4b40-8d14-e9954caadc22-tls-certs" pod "74468d4c-64a8-4b40-8d14-e9954caadc22" (UID: "74468d4c-64a8-4b40-8d14-e9954caadc22"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:31:47.940653 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:47.940307    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74468d4c-64a8-4b40-8d14-e9954caadc22-kube-api-access-s4nnx" pod "74468d4c-64a8-4b40-8d14-e9954caadc22" (UID: "74468d4c-64a8-4b40-8d14-e9954caadc22"). InnerVolumeSpecName "kube-api-access-s4nnx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:31:47.942708 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:47.942688    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74468d4c-64a8-4b40-8d14-e9954caadc22-dshm" pod "74468d4c-64a8-4b40-8d14-e9954caadc22" (UID: "74468d4c-64a8-4b40-8d14-e9954caadc22"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:31:47.946498 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:47.946481    4364 scope.go:122] "RemoveContainer" containerID="924455d80432ad889fadadcb01c34b75d2251f0c9e839c9b1cb1f953cc7865b0"
Oct 30 13:31:47.946799 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:31:47.946774    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"924455d80432ad889fadadcb01c34b75d2251f0c9e839c9b1cb1f953cc7865b0\": container with ID starting with 924455d80432ad889fadadcb01c34b75d2251f0c9e839c9b1cb1f953cc7865b0 not found: ID does not exist" containerID="924455d80432ad889fadadcb01c34b75d2251f0c9e839c9b1cb1f953cc7865b0"
Oct 30 13:31:47.946882 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:47.946805    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"924455d80432ad889fadadcb01c34b75d2251f0c9e839c9b1cb1f953cc7865b0"} err="failed to get container status \"924455d80432ad889fadadcb01c34b75d2251f0c9e839c9b1cb1f953cc7865b0\": rpc error: code = NotFound desc = could not find container \"924455d80432ad889fadadcb01c34b75d2251f0c9e839c9b1cb1f953cc7865b0\": container with ID starting with 924455d80432ad889fadadcb01c34b75d2251f0c9e839c9b1cb1f953cc7865b0 not found: ID does not exist"
Oct 30 13:31:48.045716 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:48.042481    4364 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74468d4c-64a8-4b40-8d14-e9954caadc22-dshm\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:31:48.045716 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:48.042521    4364 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74468d4c-64a8-4b40-8d14-e9954caadc22-tls-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:31:48.045716 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:48.042533    4364 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74468d4c-64a8-4b40-8d14-e9954caadc22-home\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:31:48.045716 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:48.042545    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4nnx\" (UniqueName: \"kubernetes.io/projected/74468d4c-64a8-4b40-8d14-e9954caadc22-kube-api-access-s4nnx\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:31:48.285230 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:48.285203    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-8d82d5/sub-test-model-8d82d5-kserve-d4df748cf-hqr5x"]
Oct 30 13:31:48.295459 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:48.295429    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-8d82d5/sub-test-model-8d82d5-kserve-d4df748cf-hqr5x"]
Oct 30 13:31:48.715044 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:48.715018    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq"
Oct 30 13:31:48.853992 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:48.853961    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d08eaa5-399d-4d90-8a47-3c060abfb552-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d08eaa5-399d-4d90-8a47-3c060abfb552-istio-data\") pod \"1d08eaa5-399d-4d90-8a47-3c060abfb552\" (UID: \"1d08eaa5-399d-4d90-8a47-3c060abfb552\") "
Oct 30 13:31:48.854251 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:48.854224    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d08eaa5-399d-4d90-8a47-3c060abfb552-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d08eaa5-399d-4d90-8a47-3c060abfb552-workload-certs\") pod \"1d08eaa5-399d-4d90-8a47-3c060abfb552\" (UID: \"1d08eaa5-399d-4d90-8a47-3c060abfb552\") "
Oct 30 13:31:48.854301 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:48.854262    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d08eaa5-399d-4d90-8a47-3c060abfb552-istio-data" pod "1d08eaa5-399d-4d90-8a47-3c060abfb552" (UID: "1d08eaa5-399d-4d90-8a47-3c060abfb552"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:31:48.854340 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:48.854304    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d08eaa5-399d-4d90-8a47-3c060abfb552-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d08eaa5-399d-4d90-8a47-3c060abfb552-istio-envoy\") pod \"1d08eaa5-399d-4d90-8a47-3c060abfb552\" (UID: \"1d08eaa5-399d-4d90-8a47-3c060abfb552\") "
Oct 30 13:31:48.854340 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:48.854327    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1d08eaa5-399d-4d90-8a47-3c060abfb552-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d08eaa5-399d-4d90-8a47-3c060abfb552-istio-podinfo\") pod \"1d08eaa5-399d-4d90-8a47-3c060abfb552\" (UID: \"1d08eaa5-399d-4d90-8a47-3c060abfb552\") "
Oct 30 13:31:48.854422 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:48.854348    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d08eaa5-399d-4d90-8a47-3c060abfb552-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d08eaa5-399d-4d90-8a47-3c060abfb552-workload-socket\") pod \"1d08eaa5-399d-4d90-8a47-3c060abfb552\" (UID: \"1d08eaa5-399d-4d90-8a47-3c060abfb552\") "
Oct 30 13:31:48.854422 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:48.854401    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d08eaa5-399d-4d90-8a47-3c060abfb552-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d08eaa5-399d-4d90-8a47-3c060abfb552-istiod-ca-cert\") pod \"1d08eaa5-399d-4d90-8a47-3c060abfb552\" (UID: \"1d08eaa5-399d-4d90-8a47-3c060abfb552\") "
Oct 30 13:31:48.854522 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:48.854467    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d08eaa5-399d-4d90-8a47-3c060abfb552-istio-token\" (UniqueName: \"kubernetes.io/projected/1d08eaa5-399d-4d90-8a47-3c060abfb552-istio-token\") pod \"1d08eaa5-399d-4d90-8a47-3c060abfb552\" (UID: \"1d08eaa5-399d-4d90-8a47-3c060abfb552\") "
Oct 30 13:31:48.854522 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:48.854482    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d08eaa5-399d-4d90-8a47-3c060abfb552-workload-certs" pod "1d08eaa5-399d-4d90-8a47-3c060abfb552" (UID: "1d08eaa5-399d-4d90-8a47-3c060abfb552"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:31:48.854522 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:48.854505    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d08eaa5-399d-4d90-8a47-3c060abfb552-kube-api-access-ccvfb\" (UniqueName: \"kubernetes.io/projected/1d08eaa5-399d-4d90-8a47-3c060abfb552-kube-api-access-ccvfb\") pod \"1d08eaa5-399d-4d90-8a47-3c060abfb552\" (UID: \"1d08eaa5-399d-4d90-8a47-3c060abfb552\") "
Oct 30 13:31:48.854706 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:48.854627    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d08eaa5-399d-4d90-8a47-3c060abfb552-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d08eaa5-399d-4d90-8a47-3c060abfb552-credential-socket\") pod \"1d08eaa5-399d-4d90-8a47-3c060abfb552\" (UID: \"1d08eaa5-399d-4d90-8a47-3c060abfb552\") "
Oct 30 13:31:48.854821 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:48.854800    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d08eaa5-399d-4d90-8a47-3c060abfb552-workload-socket" pod "1d08eaa5-399d-4d90-8a47-3c060abfb552" (UID: "1d08eaa5-399d-4d90-8a47-3c060abfb552"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:31:48.854913 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:48.854888    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d08eaa5-399d-4d90-8a47-3c060abfb552-credential-socket" pod "1d08eaa5-399d-4d90-8a47-3c060abfb552" (UID: "1d08eaa5-399d-4d90-8a47-3c060abfb552"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:31:48.854979 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:48.854904    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d08eaa5-399d-4d90-8a47-3c060abfb552-istiod-ca-cert" pod "1d08eaa5-399d-4d90-8a47-3c060abfb552" (UID: "1d08eaa5-399d-4d90-8a47-3c060abfb552"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:31:48.855713 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:48.855687    4364 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d08eaa5-399d-4d90-8a47-3c060abfb552-workload-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:31:48.855840 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:48.855716    4364 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d08eaa5-399d-4d90-8a47-3c060abfb552-workload-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:31:48.855840 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:48.855732    4364 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d08eaa5-399d-4d90-8a47-3c060abfb552-istiod-ca-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:31:48.855840 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:48.855745    4364 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d08eaa5-399d-4d90-8a47-3c060abfb552-credential-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:31:48.855840 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:48.855758    4364 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d08eaa5-399d-4d90-8a47-3c060abfb552-istio-data\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:31:48.856728 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:48.856702    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d08eaa5-399d-4d90-8a47-3c060abfb552-kube-api-access-ccvfb" pod "1d08eaa5-399d-4d90-8a47-3c060abfb552" (UID: "1d08eaa5-399d-4d90-8a47-3c060abfb552"). InnerVolumeSpecName "kube-api-access-ccvfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:31:48.856873 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:48.856730    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d08eaa5-399d-4d90-8a47-3c060abfb552-istio-envoy" pod "1d08eaa5-399d-4d90-8a47-3c060abfb552" (UID: "1d08eaa5-399d-4d90-8a47-3c060abfb552"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:31:48.856873 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:48.856809    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d08eaa5-399d-4d90-8a47-3c060abfb552-istio-token" pod "1d08eaa5-399d-4d90-8a47-3c060abfb552" (UID: "1d08eaa5-399d-4d90-8a47-3c060abfb552"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:31:48.857023 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:48.856904    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d08eaa5-399d-4d90-8a47-3c060abfb552-istio-podinfo" pod "1d08eaa5-399d-4d90-8a47-3c060abfb552" (UID: "1d08eaa5-399d-4d90-8a47-3c060abfb552"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:31:48.943163 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:48.943090    4364 generic.go:356] "Generic (PLEG): container finished" podID="1d08eaa5-399d-4d90-8a47-3c060abfb552" containerID="b8c165e33e34b0ca2ad872c8938134b53338fda62cd9c2439b8af09d8650e51f" exitCode=0
Oct 30 13:31:48.943163 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:48.943134    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq"
Oct 30 13:31:48.943367 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:48.943169    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq" event={"ID":"1d08eaa5-399d-4d90-8a47-3c060abfb552","Type":"ContainerDied","Data":"b8c165e33e34b0ca2ad872c8938134b53338fda62cd9c2439b8af09d8650e51f"}
Oct 30 13:31:48.943367 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:48.943221    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq" event={"ID":"1d08eaa5-399d-4d90-8a47-3c060abfb552","Type":"ContainerDied","Data":"758bf5ec1acece63a57e5263a3ab5c90c755c5fe6f70a7bbe911d373fc64e556"}
Oct 30 13:31:48.943367 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:48.943249    4364 scope.go:122] "RemoveContainer" containerID="b8c165e33e34b0ca2ad872c8938134b53338fda62cd9c2439b8af09d8650e51f"
Oct 30 13:31:48.957771 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:48.957730    4364 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d08eaa5-399d-4d90-8a47-3c060abfb552-istio-envoy\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:31:48.957771 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:48.957757    4364 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d08eaa5-399d-4d90-8a47-3c060abfb552-istio-podinfo\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:31:48.957771 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:48.957771    4364 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d08eaa5-399d-4d90-8a47-3c060abfb552-istio-token\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:31:48.958017 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:48.957784    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccvfb\" (UniqueName: \"kubernetes.io/projected/1d08eaa5-399d-4d90-8a47-3c060abfb552-kube-api-access-ccvfb\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:31:48.959490 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:48.959467    4364 scope.go:122] "RemoveContainer" containerID="b8c165e33e34b0ca2ad872c8938134b53338fda62cd9c2439b8af09d8650e51f"
Oct 30 13:31:48.959865 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:31:48.959846    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8c165e33e34b0ca2ad872c8938134b53338fda62cd9c2439b8af09d8650e51f\": container with ID starting with b8c165e33e34b0ca2ad872c8938134b53338fda62cd9c2439b8af09d8650e51f not found: ID does not exist" containerID="b8c165e33e34b0ca2ad872c8938134b53338fda62cd9c2439b8af09d8650e51f"
Oct 30 13:31:48.959950 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:48.959870    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8c165e33e34b0ca2ad872c8938134b53338fda62cd9c2439b8af09d8650e51f"} err="failed to get container status \"b8c165e33e34b0ca2ad872c8938134b53338fda62cd9c2439b8af09d8650e51f\": rpc error: code = NotFound desc = could not find container \"b8c165e33e34b0ca2ad872c8938134b53338fda62cd9c2439b8af09d8650e51f\": container with ID starting with b8c165e33e34b0ca2ad872c8938134b53338fda62cd9c2439b8af09d8650e51f not found: ID does not exist"
Oct 30 13:31:48.970985 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:48.970959    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74468d4c-64a8-4b40-8d14-e9954caadc22" path="/var/lib/kubelet/pods/74468d4c-64a8-4b40-8d14-e9954caadc22/volumes"
Oct 30 13:31:48.998202 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:48.998180    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq"]
Oct 30 13:31:49.005344 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:49.005326    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-pbfcq"]
Oct 30 13:31:50.980281 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:50.980245    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d08eaa5-399d-4d90-8a47-3c060abfb552" path="/var/lib/kubelet/pods/1d08eaa5-399d-4d90-8a47-3c060abfb552/volumes"
Oct 30 13:31:53.883463 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:53.883431    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-r9ffg"]
Oct 30 13:31:53.884031 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:53.883671    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-r9ffg" podUID="5afd5fcb-484f-407b-bbeb-6e7263bfac7a" containerName="istio-proxy" containerID="cri-o://0948198ee3d4b0f0298c79afec052f1eea723ebc73915b28e37f37fce302216b" gracePeriod=30
Oct 30 13:31:59.027811 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:59.027771    4364 generic.go:356] "Generic (PLEG): container finished" podID="5afd5fcb-484f-407b-bbeb-6e7263bfac7a" containerID="0948198ee3d4b0f0298c79afec052f1eea723ebc73915b28e37f37fce302216b" exitCode=0
Oct 30 13:31:59.028245 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:59.027850    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-r9ffg" event={"ID":"5afd5fcb-484f-407b-bbeb-6e7263bfac7a","Type":"ContainerDied","Data":"0948198ee3d4b0f0298c79afec052f1eea723ebc73915b28e37f37fce302216b"}
Oct 30 13:31:59.028245 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:59.027937    4364 scope.go:122] "RemoveContainer" containerID="2eec2874af42efd7cb01f6ae53bd78cc6a08c2cd56402b6438ca988b732b831d"
Oct 30 13:31:59.172010 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:59.171988    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-r9ffg"
Oct 30 13:31:59.293138 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:59.293108    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5afd5fcb-484f-407b-bbeb-6e7263bfac7a-istio-token\" (UniqueName: \"kubernetes.io/projected/5afd5fcb-484f-407b-bbeb-6e7263bfac7a-istio-token\") pod \"5afd5fcb-484f-407b-bbeb-6e7263bfac7a\" (UID: \"5afd5fcb-484f-407b-bbeb-6e7263bfac7a\") "
Oct 30 13:31:59.293319 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:59.293161    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5afd5fcb-484f-407b-bbeb-6e7263bfac7a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5afd5fcb-484f-407b-bbeb-6e7263bfac7a-istiod-ca-cert\") pod \"5afd5fcb-484f-407b-bbeb-6e7263bfac7a\" (UID: \"5afd5fcb-484f-407b-bbeb-6e7263bfac7a\") "
Oct 30 13:31:59.293319 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:59.293224    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5afd5fcb-484f-407b-bbeb-6e7263bfac7a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5afd5fcb-484f-407b-bbeb-6e7263bfac7a-istio-envoy\") pod \"5afd5fcb-484f-407b-bbeb-6e7263bfac7a\" (UID: \"5afd5fcb-484f-407b-bbeb-6e7263bfac7a\") "
Oct 30 13:31:59.293319 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:59.293291    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5afd5fcb-484f-407b-bbeb-6e7263bfac7a-kube-api-access-rqbql\" (UniqueName: \"kubernetes.io/projected/5afd5fcb-484f-407b-bbeb-6e7263bfac7a-kube-api-access-rqbql\") pod \"5afd5fcb-484f-407b-bbeb-6e7263bfac7a\" (UID: \"5afd5fcb-484f-407b-bbeb-6e7263bfac7a\") "
Oct 30 13:31:59.293510 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:59.293382    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5afd5fcb-484f-407b-bbeb-6e7263bfac7a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5afd5fcb-484f-407b-bbeb-6e7263bfac7a-credential-socket\") pod \"5afd5fcb-484f-407b-bbeb-6e7263bfac7a\" (UID: \"5afd5fcb-484f-407b-bbeb-6e7263bfac7a\") "
Oct 30 13:31:59.293510 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:59.293458    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5afd5fcb-484f-407b-bbeb-6e7263bfac7a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5afd5fcb-484f-407b-bbeb-6e7263bfac7a-workload-socket\") pod \"5afd5fcb-484f-407b-bbeb-6e7263bfac7a\" (UID: \"5afd5fcb-484f-407b-bbeb-6e7263bfac7a\") "
Oct 30 13:31:59.293510 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:59.293503    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5afd5fcb-484f-407b-bbeb-6e7263bfac7a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5afd5fcb-484f-407b-bbeb-6e7263bfac7a-istio-podinfo\") pod \"5afd5fcb-484f-407b-bbeb-6e7263bfac7a\" (UID: \"5afd5fcb-484f-407b-bbeb-6e7263bfac7a\") "
Oct 30 13:31:59.293729 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:59.293651    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5afd5fcb-484f-407b-bbeb-6e7263bfac7a-istiod-ca-cert" pod "5afd5fcb-484f-407b-bbeb-6e7263bfac7a" (UID: "5afd5fcb-484f-407b-bbeb-6e7263bfac7a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:31:59.293798 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:59.293732    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5afd5fcb-484f-407b-bbeb-6e7263bfac7a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5afd5fcb-484f-407b-bbeb-6e7263bfac7a-workload-certs\") pod \"5afd5fcb-484f-407b-bbeb-6e7263bfac7a\" (UID: \"5afd5fcb-484f-407b-bbeb-6e7263bfac7a\") "
Oct 30 13:31:59.293798 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:59.293767    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5afd5fcb-484f-407b-bbeb-6e7263bfac7a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5afd5fcb-484f-407b-bbeb-6e7263bfac7a-istio-data\") pod \"5afd5fcb-484f-407b-bbeb-6e7263bfac7a\" (UID: \"5afd5fcb-484f-407b-bbeb-6e7263bfac7a\") "
Oct 30 13:31:59.293919 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:59.293857    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5afd5fcb-484f-407b-bbeb-6e7263bfac7a-credential-socket" pod "5afd5fcb-484f-407b-bbeb-6e7263bfac7a" (UID: "5afd5fcb-484f-407b-bbeb-6e7263bfac7a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:31:59.293986 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:59.293928    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5afd5fcb-484f-407b-bbeb-6e7263bfac7a-workload-socket" pod "5afd5fcb-484f-407b-bbeb-6e7263bfac7a" (UID: "5afd5fcb-484f-407b-bbeb-6e7263bfac7a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:31:59.294174 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:59.294147    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5afd5fcb-484f-407b-bbeb-6e7263bfac7a-workload-certs" pod "5afd5fcb-484f-407b-bbeb-6e7263bfac7a" (UID: "5afd5fcb-484f-407b-bbeb-6e7263bfac7a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:31:59.294743 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:59.294344    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5afd5fcb-484f-407b-bbeb-6e7263bfac7a-istio-data" pod "5afd5fcb-484f-407b-bbeb-6e7263bfac7a" (UID: "5afd5fcb-484f-407b-bbeb-6e7263bfac7a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:31:59.294743 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:59.294658    4364 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5afd5fcb-484f-407b-bbeb-6e7263bfac7a-workload-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:31:59.294743 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:59.294678    4364 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5afd5fcb-484f-407b-bbeb-6e7263bfac7a-istio-data\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:31:59.294743 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:59.294693    4364 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5afd5fcb-484f-407b-bbeb-6e7263bfac7a-istiod-ca-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:31:59.294743 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:59.294707    4364 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5afd5fcb-484f-407b-bbeb-6e7263bfac7a-credential-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:31:59.294743 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:59.294722    4364 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5afd5fcb-484f-407b-bbeb-6e7263bfac7a-workload-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:31:59.296326 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:59.296253    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5afd5fcb-484f-407b-bbeb-6e7263bfac7a-istio-envoy" pod "5afd5fcb-484f-407b-bbeb-6e7263bfac7a" (UID: "5afd5fcb-484f-407b-bbeb-6e7263bfac7a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:31:59.296326 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:59.296275    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5afd5fcb-484f-407b-bbeb-6e7263bfac7a-istio-podinfo" pod "5afd5fcb-484f-407b-bbeb-6e7263bfac7a" (UID: "5afd5fcb-484f-407b-bbeb-6e7263bfac7a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:31:59.296326 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:59.296305    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5afd5fcb-484f-407b-bbeb-6e7263bfac7a-kube-api-access-rqbql" pod "5afd5fcb-484f-407b-bbeb-6e7263bfac7a" (UID: "5afd5fcb-484f-407b-bbeb-6e7263bfac7a"). InnerVolumeSpecName "kube-api-access-rqbql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:31:59.297505 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:59.297477    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5afd5fcb-484f-407b-bbeb-6e7263bfac7a-istio-token" pod "5afd5fcb-484f-407b-bbeb-6e7263bfac7a" (UID: "5afd5fcb-484f-407b-bbeb-6e7263bfac7a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:31:59.395947 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:59.395912    4364 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5afd5fcb-484f-407b-bbeb-6e7263bfac7a-istio-podinfo\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:31:59.395947 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:59.395939    4364 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5afd5fcb-484f-407b-bbeb-6e7263bfac7a-istio-token\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:31:59.395947 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:59.395948    4364 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5afd5fcb-484f-407b-bbeb-6e7263bfac7a-istio-envoy\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:31:59.395947 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:31:59.395957    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqbql\" (UniqueName: \"kubernetes.io/projected/5afd5fcb-484f-407b-bbeb-6e7263bfac7a-kube-api-access-rqbql\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:32:00.038174 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:00.038143    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-r9ffg"
Oct 30 13:32:00.038664 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:00.038181    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-r9ffg" event={"ID":"5afd5fcb-484f-407b-bbeb-6e7263bfac7a","Type":"ContainerDied","Data":"801d4df035a7a93765e1c3182b96c87e9aa286cf739087ffde98dda994aa1af5"}
Oct 30 13:32:00.038664 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:00.038255    4364 scope.go:122] "RemoveContainer" containerID="0948198ee3d4b0f0298c79afec052f1eea723ebc73915b28e37f37fce302216b"
Oct 30 13:32:00.104848 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:00.104822    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-r9ffg"]
Oct 30 13:32:00.116002 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:00.115977    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-r9ffg"]
Oct 30 13:32:00.971404 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:00.971373    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5afd5fcb-484f-407b-bbeb-6e7263bfac7a" path="/var/lib/kubelet/pods/5afd5fcb-484f-407b-bbeb-6e7263bfac7a/volumes"
Oct 30 13:32:01.748031 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:01.747998    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-4b0ea9/auth-test-model-4b0ea9-kserve-86458657c-m5qrl"]
Oct 30 13:32:01.748867 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:01.748840    4364 cpu_manager.go:414] "containerMap: removing container" podUID="1d08eaa5-399d-4d90-8a47-3c060abfb552" containerName="istio-proxy"
Oct 30 13:32:01.748867 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:01.748867    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d08eaa5-399d-4d90-8a47-3c060abfb552" containerName="istio-proxy"
Oct 30 13:32:01.749060 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:01.748899    4364 cpu_manager.go:414] "containerMap: removing container" podUID="74468d4c-64a8-4b40-8d14-e9954caadc22" containerName="main"
Oct 30 13:32:01.749060 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:01.748907    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74468d4c-64a8-4b40-8d14-e9954caadc22" containerName="main"
Oct 30 13:32:01.749060 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:01.748934    4364 cpu_manager.go:414] "containerMap: removing container" podUID="5afd5fcb-484f-407b-bbeb-6e7263bfac7a" containerName="istio-proxy"
Oct 30 13:32:01.749060 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:01.748940    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5afd5fcb-484f-407b-bbeb-6e7263bfac7a" containerName="istio-proxy"
Oct 30 13:32:01.749060 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:01.748948    4364 cpu_manager.go:414] "containerMap: removing container" podUID="5afd5fcb-484f-407b-bbeb-6e7263bfac7a" containerName="istio-proxy"
Oct 30 13:32:01.749060 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:01.748953    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5afd5fcb-484f-407b-bbeb-6e7263bfac7a" containerName="istio-proxy"
Oct 30 13:32:01.749401 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:01.749125    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="5afd5fcb-484f-407b-bbeb-6e7263bfac7a" containerName="istio-proxy"
Oct 30 13:32:01.749401 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:01.749143    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="5afd5fcb-484f-407b-bbeb-6e7263bfac7a" containerName="istio-proxy"
Oct 30 13:32:01.749401 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:01.749162    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d08eaa5-399d-4d90-8a47-3c060abfb552" containerName="istio-proxy"
Oct 30 13:32:01.749401 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:01.749172    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="74468d4c-64a8-4b40-8d14-e9954caadc22" containerName="main"
Oct 30 13:32:01.753079 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:01.753053    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4b0ea9/auth-test-model-4b0ea9-kserve-86458657c-m5qrl"
Oct 30 13:32:01.779144 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:01.779120    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4b0ea9/auth-test-model-4b0ea9-kserve-86458657c-m5qrl"]
Oct 30 13:32:01.783828 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:01.783796    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-4b0ea9\"/\"openshift-service-ca.crt\""
Oct 30 13:32:01.783828 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:01.783804    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-4b0ea9\"/\"default-dockercfg-hdsmv\""
Oct 30 13:32:01.784068 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:01.783893    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-4b0ea9\"/\"kube-root-ca.crt\""
Oct 30 13:32:01.784068 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:01.784032    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-4b0ea9\"/\"auth-test-model-4b0ea9-kserve-self-signed-certs\""
Oct 30 13:32:01.823479 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:01.823457    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/825bb91f-8422-4b4d-8e13-ad421aa3c20f-model-cache\") pod \"auth-test-model-4b0ea9-kserve-86458657c-m5qrl\" (UID: \"825bb91f-8422-4b4d-8e13-ad421aa3c20f\") " pod="ai-tenant-e2e-shared-b-w3-4b0ea9/auth-test-model-4b0ea9-kserve-86458657c-m5qrl"
Oct 30 13:32:01.823628 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:01.823493    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqndb\" (UniqueName: \"kubernetes.io/projected/825bb91f-8422-4b4d-8e13-ad421aa3c20f-kube-api-access-hqndb\") pod \"auth-test-model-4b0ea9-kserve-86458657c-m5qrl\" (UID: \"825bb91f-8422-4b4d-8e13-ad421aa3c20f\") " pod="ai-tenant-e2e-shared-b-w3-4b0ea9/auth-test-model-4b0ea9-kserve-86458657c-m5qrl"
Oct 30 13:32:01.823837 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:01.823810    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/825bb91f-8422-4b4d-8e13-ad421aa3c20f-home\") pod \"auth-test-model-4b0ea9-kserve-86458657c-m5qrl\" (UID: \"825bb91f-8422-4b4d-8e13-ad421aa3c20f\") " pod="ai-tenant-e2e-shared-b-w3-4b0ea9/auth-test-model-4b0ea9-kserve-86458657c-m5qrl"
Oct 30 13:32:01.823958 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:01.823872    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/825bb91f-8422-4b4d-8e13-ad421aa3c20f-dshm\") pod \"auth-test-model-4b0ea9-kserve-86458657c-m5qrl\" (UID: \"825bb91f-8422-4b4d-8e13-ad421aa3c20f\") " pod="ai-tenant-e2e-shared-b-w3-4b0ea9/auth-test-model-4b0ea9-kserve-86458657c-m5qrl"
Oct 30 13:32:01.824026 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:01.824006    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/825bb91f-8422-4b4d-8e13-ad421aa3c20f-tls-certs\") pod \"auth-test-model-4b0ea9-kserve-86458657c-m5qrl\" (UID: \"825bb91f-8422-4b4d-8e13-ad421aa3c20f\") " pod="ai-tenant-e2e-shared-b-w3-4b0ea9/auth-test-model-4b0ea9-kserve-86458657c-m5qrl"
Oct 30 13:32:01.824077 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:01.824063    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/825bb91f-8422-4b4d-8e13-ad421aa3c20f-tmp-dir\") pod \"auth-test-model-4b0ea9-kserve-86458657c-m5qrl\" (UID: \"825bb91f-8422-4b4d-8e13-ad421aa3c20f\") " pod="ai-tenant-e2e-shared-b-w3-4b0ea9/auth-test-model-4b0ea9-kserve-86458657c-m5qrl"
Oct 30 13:32:01.924766 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:01.924738    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/825bb91f-8422-4b4d-8e13-ad421aa3c20f-home\") pod \"auth-test-model-4b0ea9-kserve-86458657c-m5qrl\" (UID: \"825bb91f-8422-4b4d-8e13-ad421aa3c20f\") " pod="ai-tenant-e2e-shared-b-w3-4b0ea9/auth-test-model-4b0ea9-kserve-86458657c-m5qrl"
Oct 30 13:32:01.924924 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:01.924782    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/825bb91f-8422-4b4d-8e13-ad421aa3c20f-dshm\") pod \"auth-test-model-4b0ea9-kserve-86458657c-m5qrl\" (UID: \"825bb91f-8422-4b4d-8e13-ad421aa3c20f\") " pod="ai-tenant-e2e-shared-b-w3-4b0ea9/auth-test-model-4b0ea9-kserve-86458657c-m5qrl"
Oct 30 13:32:01.924924 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:01.924881    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/825bb91f-8422-4b4d-8e13-ad421aa3c20f-tls-certs\") pod \"auth-test-model-4b0ea9-kserve-86458657c-m5qrl\" (UID: \"825bb91f-8422-4b4d-8e13-ad421aa3c20f\") " pod="ai-tenant-e2e-shared-b-w3-4b0ea9/auth-test-model-4b0ea9-kserve-86458657c-m5qrl"
Oct 30 13:32:01.924924 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:01.924911    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/825bb91f-8422-4b4d-8e13-ad421aa3c20f-tmp-dir\") pod \"auth-test-model-4b0ea9-kserve-86458657c-m5qrl\" (UID: \"825bb91f-8422-4b4d-8e13-ad421aa3c20f\") " pod="ai-tenant-e2e-shared-b-w3-4b0ea9/auth-test-model-4b0ea9-kserve-86458657c-m5qrl"
Oct 30 13:32:01.925090 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:01.924956    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/825bb91f-8422-4b4d-8e13-ad421aa3c20f-model-cache\") pod \"auth-test-model-4b0ea9-kserve-86458657c-m5qrl\" (UID: \"825bb91f-8422-4b4d-8e13-ad421aa3c20f\") " pod="ai-tenant-e2e-shared-b-w3-4b0ea9/auth-test-model-4b0ea9-kserve-86458657c-m5qrl"
Oct 30 13:32:01.925090 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:01.924982    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqndb\" (UniqueName: \"kubernetes.io/projected/825bb91f-8422-4b4d-8e13-ad421aa3c20f-kube-api-access-hqndb\") pod \"auth-test-model-4b0ea9-kserve-86458657c-m5qrl\" (UID: \"825bb91f-8422-4b4d-8e13-ad421aa3c20f\") " pod="ai-tenant-e2e-shared-b-w3-4b0ea9/auth-test-model-4b0ea9-kserve-86458657c-m5qrl"
Oct 30 13:32:01.925220 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:01.925186    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/825bb91f-8422-4b4d-8e13-ad421aa3c20f-home\") pod \"auth-test-model-4b0ea9-kserve-86458657c-m5qrl\" (UID: \"825bb91f-8422-4b4d-8e13-ad421aa3c20f\") " pod="ai-tenant-e2e-shared-b-w3-4b0ea9/auth-test-model-4b0ea9-kserve-86458657c-m5qrl"
Oct 30 13:32:01.925319 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:01.925294    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/825bb91f-8422-4b4d-8e13-ad421aa3c20f-tmp-dir\") pod \"auth-test-model-4b0ea9-kserve-86458657c-m5qrl\" (UID: \"825bb91f-8422-4b4d-8e13-ad421aa3c20f\") " pod="ai-tenant-e2e-shared-b-w3-4b0ea9/auth-test-model-4b0ea9-kserve-86458657c-m5qrl"
Oct 30 13:32:01.925418 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:01.925392    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/825bb91f-8422-4b4d-8e13-ad421aa3c20f-model-cache\") pod \"auth-test-model-4b0ea9-kserve-86458657c-m5qrl\" (UID: \"825bb91f-8422-4b4d-8e13-ad421aa3c20f\") " pod="ai-tenant-e2e-shared-b-w3-4b0ea9/auth-test-model-4b0ea9-kserve-86458657c-m5qrl"
Oct 30 13:32:01.927075 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:01.927054    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/825bb91f-8422-4b4d-8e13-ad421aa3c20f-dshm\") pod \"auth-test-model-4b0ea9-kserve-86458657c-m5qrl\" (UID: \"825bb91f-8422-4b4d-8e13-ad421aa3c20f\") " pod="ai-tenant-e2e-shared-b-w3-4b0ea9/auth-test-model-4b0ea9-kserve-86458657c-m5qrl"
Oct 30 13:32:01.927333 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:01.927315    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/825bb91f-8422-4b4d-8e13-ad421aa3c20f-tls-certs\") pod \"auth-test-model-4b0ea9-kserve-86458657c-m5qrl\" (UID: \"825bb91f-8422-4b4d-8e13-ad421aa3c20f\") " pod="ai-tenant-e2e-shared-b-w3-4b0ea9/auth-test-model-4b0ea9-kserve-86458657c-m5qrl"
Oct 30 13:32:01.951232 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:01.951212    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqndb\" (UniqueName: \"kubernetes.io/projected/825bb91f-8422-4b4d-8e13-ad421aa3c20f-kube-api-access-hqndb\") pod \"auth-test-model-4b0ea9-kserve-86458657c-m5qrl\" (UID: \"825bb91f-8422-4b4d-8e13-ad421aa3c20f\") " pod="ai-tenant-e2e-shared-b-w3-4b0ea9/auth-test-model-4b0ea9-kserve-86458657c-m5qrl"
Oct 30 13:32:02.074669 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:02.068734    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4b0ea9/auth-test-model-4b0ea9-kserve-86458657c-m5qrl"
Oct 30 13:32:02.456388 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:02.456360    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4b0ea9/auth-test-model-4b0ea9-kserve-86458657c-m5qrl"]
Oct 30 13:32:02.456998 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:32:02.456972    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod825bb91f_8422_4b4d_8e13_ad421aa3c20f.slice/crio-f8c9619ccfe9e26804870767fe682ed345d229922b4bc6b3f2e21b2ccdbbb6e9 WatchSource:0}: Error finding container f8c9619ccfe9e26804870767fe682ed345d229922b4bc6b3f2e21b2ccdbbb6e9: Status 404 returned error can't find the container with id f8c9619ccfe9e26804870767fe682ed345d229922b4bc6b3f2e21b2ccdbbb6e9
Oct 30 13:32:03.076463 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:03.076433    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4b0ea9/auth-test-model-4b0ea9-kserve-86458657c-m5qrl" event={"ID":"825bb91f-8422-4b4d-8e13-ad421aa3c20f","Type":"ContainerStarted","Data":"9a82173e937d061bd8daf22e63f31c803e8cd578b31bdc5ad3cef5323527e857"}
Oct 30 13:32:03.076928 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:03.076473    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4b0ea9/auth-test-model-4b0ea9-kserve-86458657c-m5qrl" event={"ID":"825bb91f-8422-4b4d-8e13-ad421aa3c20f","Type":"ContainerStarted","Data":"f8c9619ccfe9e26804870767fe682ed345d229922b4bc6b3f2e21b2ccdbbb6e9"}
Oct 30 13:32:03.137887 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:03.137842    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-4b0ea9/auth-test-model-4b0ea9-kserve-86458657c-m5qrl" podStartSLOduration=2.137829297 podStartE2EDuration="2.137829297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:32:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:32:03.134125361 +0000 UTC m=+2422.300490009" watchObservedRunningTime="2026-10-30 13:32:03.137829297 +0000 UTC m=+2422.304193944"
Oct 30 13:32:04.069900 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:04.069864    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-4b0ea9/auth-test-model-4b0ea9-kserve-86458657c-m5qrl"
Oct 30 13:32:05.471066 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:05.471031    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w6-cda0d9/rate-test-model-cda0d9-kserve-9d99f655c-swqhw"]
Oct 30 13:32:05.480865 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:05.480839    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-cda0d9/rate-test-model-cda0d9-kserve-9d99f655c-swqhw"
Oct 30 13:32:05.497257 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:05.497227    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-cda0d9/rate-test-model-cda0d9-kserve-9d99f655c-swqhw"]
Oct 30 13:32:05.501096 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:05.501079    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-cda0d9\"/\"default-dockercfg-7r8ms\""
Oct 30 13:32:05.512837 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:05.512813    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-cda0d9\"/\"rate-test-model-cda0d9-kserve-self-signed-certs\""
Oct 30 13:32:05.513068 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:05.513050    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-cda0d9\"/\"openshift-service-ca.crt\""
Oct 30 13:32:05.513162 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:05.513087    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-cda0d9\"/\"kube-root-ca.crt\""
Oct 30 13:32:05.515828 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:05.515804    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d21066d6-fdb1-4c53-8e3b-b8b254112229-model-cache\") pod \"rate-test-model-cda0d9-kserve-9d99f655c-swqhw\" (UID: \"d21066d6-fdb1-4c53-8e3b-b8b254112229\") " pod="ai-tenant-e2e-shared-b-w6-cda0d9/rate-test-model-cda0d9-kserve-9d99f655c-swqhw"
Oct 30 13:32:05.515902 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:05.515840    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d21066d6-fdb1-4c53-8e3b-b8b254112229-tls-certs\") pod \"rate-test-model-cda0d9-kserve-9d99f655c-swqhw\" (UID: \"d21066d6-fdb1-4c53-8e3b-b8b254112229\") " pod="ai-tenant-e2e-shared-b-w6-cda0d9/rate-test-model-cda0d9-kserve-9d99f655c-swqhw"
Oct 30 13:32:05.515902 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:05.515858    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d21066d6-fdb1-4c53-8e3b-b8b254112229-tmp-dir\") pod \"rate-test-model-cda0d9-kserve-9d99f655c-swqhw\" (UID: \"d21066d6-fdb1-4c53-8e3b-b8b254112229\") " pod="ai-tenant-e2e-shared-b-w6-cda0d9/rate-test-model-cda0d9-kserve-9d99f655c-swqhw"
Oct 30 13:32:05.516013 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:05.515996    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d21066d6-fdb1-4c53-8e3b-b8b254112229-home\") pod \"rate-test-model-cda0d9-kserve-9d99f655c-swqhw\" (UID: \"d21066d6-fdb1-4c53-8e3b-b8b254112229\") " pod="ai-tenant-e2e-shared-b-w6-cda0d9/rate-test-model-cda0d9-kserve-9d99f655c-swqhw"
Oct 30 13:32:05.516110 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:05.516097    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d21066d6-fdb1-4c53-8e3b-b8b254112229-dshm\") pod \"rate-test-model-cda0d9-kserve-9d99f655c-swqhw\" (UID: \"d21066d6-fdb1-4c53-8e3b-b8b254112229\") " pod="ai-tenant-e2e-shared-b-w6-cda0d9/rate-test-model-cda0d9-kserve-9d99f655c-swqhw"
Oct 30 13:32:05.516224 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:05.516207    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrjrf\" (UniqueName: \"kubernetes.io/projected/d21066d6-fdb1-4c53-8e3b-b8b254112229-kube-api-access-lrjrf\") pod \"rate-test-model-cda0d9-kserve-9d99f655c-swqhw\" (UID: \"d21066d6-fdb1-4c53-8e3b-b8b254112229\") " pod="ai-tenant-e2e-shared-b-w6-cda0d9/rate-test-model-cda0d9-kserve-9d99f655c-swqhw"
Oct 30 13:32:05.617742 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:05.617715    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d21066d6-fdb1-4c53-8e3b-b8b254112229-model-cache\") pod \"rate-test-model-cda0d9-kserve-9d99f655c-swqhw\" (UID: \"d21066d6-fdb1-4c53-8e3b-b8b254112229\") " pod="ai-tenant-e2e-shared-b-w6-cda0d9/rate-test-model-cda0d9-kserve-9d99f655c-swqhw"
Oct 30 13:32:05.617881 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:05.617745    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d21066d6-fdb1-4c53-8e3b-b8b254112229-tls-certs\") pod \"rate-test-model-cda0d9-kserve-9d99f655c-swqhw\" (UID: \"d21066d6-fdb1-4c53-8e3b-b8b254112229\") " pod="ai-tenant-e2e-shared-b-w6-cda0d9/rate-test-model-cda0d9-kserve-9d99f655c-swqhw"
Oct 30 13:32:05.617881 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:05.617765    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d21066d6-fdb1-4c53-8e3b-b8b254112229-tmp-dir\") pod \"rate-test-model-cda0d9-kserve-9d99f655c-swqhw\" (UID: \"d21066d6-fdb1-4c53-8e3b-b8b254112229\") " pod="ai-tenant-e2e-shared-b-w6-cda0d9/rate-test-model-cda0d9-kserve-9d99f655c-swqhw"
Oct 30 13:32:05.617881 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:05.617810    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d21066d6-fdb1-4c53-8e3b-b8b254112229-home\") pod \"rate-test-model-cda0d9-kserve-9d99f655c-swqhw\" (UID: \"d21066d6-fdb1-4c53-8e3b-b8b254112229\") " pod="ai-tenant-e2e-shared-b-w6-cda0d9/rate-test-model-cda0d9-kserve-9d99f655c-swqhw"
Oct 30 13:32:05.617881 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:05.617860    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d21066d6-fdb1-4c53-8e3b-b8b254112229-dshm\") pod \"rate-test-model-cda0d9-kserve-9d99f655c-swqhw\" (UID: \"d21066d6-fdb1-4c53-8e3b-b8b254112229\") " pod="ai-tenant-e2e-shared-b-w6-cda0d9/rate-test-model-cda0d9-kserve-9d99f655c-swqhw"
Oct 30 13:32:05.618057 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:05.617919    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrjrf\" (UniqueName: \"kubernetes.io/projected/d21066d6-fdb1-4c53-8e3b-b8b254112229-kube-api-access-lrjrf\") pod \"rate-test-model-cda0d9-kserve-9d99f655c-swqhw\" (UID: \"d21066d6-fdb1-4c53-8e3b-b8b254112229\") " pod="ai-tenant-e2e-shared-b-w6-cda0d9/rate-test-model-cda0d9-kserve-9d99f655c-swqhw"
Oct 30 13:32:05.618161 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:05.618135    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d21066d6-fdb1-4c53-8e3b-b8b254112229-model-cache\") pod \"rate-test-model-cda0d9-kserve-9d99f655c-swqhw\" (UID: \"d21066d6-fdb1-4c53-8e3b-b8b254112229\") " pod="ai-tenant-e2e-shared-b-w6-cda0d9/rate-test-model-cda0d9-kserve-9d99f655c-swqhw"
Oct 30 13:32:05.618228 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:05.618207    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d21066d6-fdb1-4c53-8e3b-b8b254112229-home\") pod \"rate-test-model-cda0d9-kserve-9d99f655c-swqhw\" (UID: \"d21066d6-fdb1-4c53-8e3b-b8b254112229\") " pod="ai-tenant-e2e-shared-b-w6-cda0d9/rate-test-model-cda0d9-kserve-9d99f655c-swqhw"
Oct 30 13:32:05.618298 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:05.618232    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d21066d6-fdb1-4c53-8e3b-b8b254112229-tmp-dir\") pod \"rate-test-model-cda0d9-kserve-9d99f655c-swqhw\" (UID: \"d21066d6-fdb1-4c53-8e3b-b8b254112229\") " pod="ai-tenant-e2e-shared-b-w6-cda0d9/rate-test-model-cda0d9-kserve-9d99f655c-swqhw"
Oct 30 13:32:05.620067 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:05.620041    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d21066d6-fdb1-4c53-8e3b-b8b254112229-dshm\") pod \"rate-test-model-cda0d9-kserve-9d99f655c-swqhw\" (UID: \"d21066d6-fdb1-4c53-8e3b-b8b254112229\") " pod="ai-tenant-e2e-shared-b-w6-cda0d9/rate-test-model-cda0d9-kserve-9d99f655c-swqhw"
Oct 30 13:32:05.620322 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:05.620304    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d21066d6-fdb1-4c53-8e3b-b8b254112229-tls-certs\") pod \"rate-test-model-cda0d9-kserve-9d99f655c-swqhw\" (UID: \"d21066d6-fdb1-4c53-8e3b-b8b254112229\") " pod="ai-tenant-e2e-shared-b-w6-cda0d9/rate-test-model-cda0d9-kserve-9d99f655c-swqhw"
Oct 30 13:32:05.642179 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:05.642155    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrjrf\" (UniqueName: \"kubernetes.io/projected/d21066d6-fdb1-4c53-8e3b-b8b254112229-kube-api-access-lrjrf\") pod \"rate-test-model-cda0d9-kserve-9d99f655c-swqhw\" (UID: \"d21066d6-fdb1-4c53-8e3b-b8b254112229\") " pod="ai-tenant-e2e-shared-b-w6-cda0d9/rate-test-model-cda0d9-kserve-9d99f655c-swqhw"
Oct 30 13:32:05.808687 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:05.807939    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-cda0d9/rate-test-model-cda0d9-kserve-9d99f655c-swqhw"
Oct 30 13:32:05.956106 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:05.956078    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp"]
Oct 30 13:32:05.960249 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:05.960223    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp"
Oct 30 13:32:05.986156 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:05.986124    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp"]
Oct 30 13:32:06.049786 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:06.049758    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09bcc5ff-a33b-4699-a797-58b18a5f63bd-workload-socket\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp\" (UID: \"09bcc5ff-a33b-4699-a797-58b18a5f63bd\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp"
Oct 30 13:32:06.049914 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:06.049855    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09bcc5ff-a33b-4699-a797-58b18a5f63bd-istio-envoy\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp\" (UID: \"09bcc5ff-a33b-4699-a797-58b18a5f63bd\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp"
Oct 30 13:32:06.049914 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:06.049907    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09bcc5ff-a33b-4699-a797-58b18a5f63bd-istio-data\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp\" (UID: \"09bcc5ff-a33b-4699-a797-58b18a5f63bd\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp"
Oct 30 13:32:06.050032 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:06.049972    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09bcc5ff-a33b-4699-a797-58b18a5f63bd-credential-socket\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp\" (UID: \"09bcc5ff-a33b-4699-a797-58b18a5f63bd\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp"
Oct 30 13:32:06.050032 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:06.050002    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09bcc5ff-a33b-4699-a797-58b18a5f63bd-istiod-ca-cert\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp\" (UID: \"09bcc5ff-a33b-4699-a797-58b18a5f63bd\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp"
Oct 30 13:32:06.050114 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:06.050087    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09bcc5ff-a33b-4699-a797-58b18a5f63bd-workload-certs\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp\" (UID: \"09bcc5ff-a33b-4699-a797-58b18a5f63bd\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp"
Oct 30 13:32:06.050230 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:06.050214    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09bcc5ff-a33b-4699-a797-58b18a5f63bd-istio-podinfo\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp\" (UID: \"09bcc5ff-a33b-4699-a797-58b18a5f63bd\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp"
Oct 30 13:32:06.050447 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:06.050427    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5sdd\" (UniqueName: \"kubernetes.io/projected/09bcc5ff-a33b-4699-a797-58b18a5f63bd-kube-api-access-f5sdd\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp\" (UID: \"09bcc5ff-a33b-4699-a797-58b18a5f63bd\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp"
Oct 30 13:32:06.050510 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:06.050482    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09bcc5ff-a33b-4699-a797-58b18a5f63bd-istio-token\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp\" (UID: \"09bcc5ff-a33b-4699-a797-58b18a5f63bd\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp"
Oct 30 13:32:06.152334 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:06.152300    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09bcc5ff-a33b-4699-a797-58b18a5f63bd-workload-certs\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp\" (UID: \"09bcc5ff-a33b-4699-a797-58b18a5f63bd\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp"
Oct 30 13:32:06.152492 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:06.152357    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09bcc5ff-a33b-4699-a797-58b18a5f63bd-istio-podinfo\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp\" (UID: \"09bcc5ff-a33b-4699-a797-58b18a5f63bd\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp"
Oct 30 13:32:06.152492 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:06.152446    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5sdd\" (UniqueName: \"kubernetes.io/projected/09bcc5ff-a33b-4699-a797-58b18a5f63bd-kube-api-access-f5sdd\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp\" (UID: \"09bcc5ff-a33b-4699-a797-58b18a5f63bd\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp"
Oct 30 13:32:06.152492 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:06.152486    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09bcc5ff-a33b-4699-a797-58b18a5f63bd-istio-token\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp\" (UID: \"09bcc5ff-a33b-4699-a797-58b18a5f63bd\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp"
Oct 30 13:32:06.152704 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:06.152649    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09bcc5ff-a33b-4699-a797-58b18a5f63bd-workload-socket\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp\" (UID: \"09bcc5ff-a33b-4699-a797-58b18a5f63bd\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp"
Oct 30 13:32:06.152704 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:06.152687    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09bcc5ff-a33b-4699-a797-58b18a5f63bd-workload-certs\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp\" (UID: \"09bcc5ff-a33b-4699-a797-58b18a5f63bd\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp"
Oct 30 13:32:06.152704 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:06.152700    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09bcc5ff-a33b-4699-a797-58b18a5f63bd-istio-envoy\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp\" (UID: \"09bcc5ff-a33b-4699-a797-58b18a5f63bd\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp"
Oct 30 13:32:06.152882 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:06.152743    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09bcc5ff-a33b-4699-a797-58b18a5f63bd-istio-data\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp\" (UID: \"09bcc5ff-a33b-4699-a797-58b18a5f63bd\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp"
Oct 30 13:32:06.152882 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:06.152774    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09bcc5ff-a33b-4699-a797-58b18a5f63bd-credential-socket\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp\" (UID: \"09bcc5ff-a33b-4699-a797-58b18a5f63bd\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp"
Oct 30 13:32:06.152882 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:06.152800    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09bcc5ff-a33b-4699-a797-58b18a5f63bd-istiod-ca-cert\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp\" (UID: \"09bcc5ff-a33b-4699-a797-58b18a5f63bd\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp"
Oct 30 13:32:06.153037 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:06.152943    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09bcc5ff-a33b-4699-a797-58b18a5f63bd-workload-socket\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp\" (UID: \"09bcc5ff-a33b-4699-a797-58b18a5f63bd\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp"
Oct 30 13:32:06.153258 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:06.153235    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09bcc5ff-a33b-4699-a797-58b18a5f63bd-istio-data\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp\" (UID: \"09bcc5ff-a33b-4699-a797-58b18a5f63bd\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp"
Oct 30 13:32:06.153380 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:06.153269    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09bcc5ff-a33b-4699-a797-58b18a5f63bd-credential-socket\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp\" (UID: \"09bcc5ff-a33b-4699-a797-58b18a5f63bd\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp"
Oct 30 13:32:06.153488 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:06.153468    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09bcc5ff-a33b-4699-a797-58b18a5f63bd-istiod-ca-cert\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp\" (UID: \"09bcc5ff-a33b-4699-a797-58b18a5f63bd\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp"
Oct 30 13:32:06.154993 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:06.154968    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09bcc5ff-a33b-4699-a797-58b18a5f63bd-istio-envoy\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp\" (UID: \"09bcc5ff-a33b-4699-a797-58b18a5f63bd\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp"
Oct 30 13:32:06.155287 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:06.155268    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09bcc5ff-a33b-4699-a797-58b18a5f63bd-istio-podinfo\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp\" (UID: \"09bcc5ff-a33b-4699-a797-58b18a5f63bd\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp"
Oct 30 13:32:06.168562 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:06.168533    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-cda0d9/rate-test-model-cda0d9-kserve-9d99f655c-swqhw"]
Oct 30 13:32:06.176240 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:06.176219    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09bcc5ff-a33b-4699-a797-58b18a5f63bd-istio-token\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp\" (UID: \"09bcc5ff-a33b-4699-a797-58b18a5f63bd\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp"
Oct 30 13:32:06.176657 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:06.176639    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5sdd\" (UniqueName: \"kubernetes.io/projected/09bcc5ff-a33b-4699-a797-58b18a5f63bd-kube-api-access-f5sdd\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp\" (UID: \"09bcc5ff-a33b-4699-a797-58b18a5f63bd\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp"
Oct 30 13:32:06.176802 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:32:06.176783    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd21066d6_fdb1_4c53_8e3b_b8b254112229.slice/crio-f68408d50e68cbc1cf524b00af6533710dcb67d32f8c736e74458128f084762e WatchSource:0}: Error finding container f68408d50e68cbc1cf524b00af6533710dcb67d32f8c736e74458128f084762e: Status 404 returned error can't find the container with id f68408d50e68cbc1cf524b00af6533710dcb67d32f8c736e74458128f084762e
Oct 30 13:32:06.284033 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:06.284007    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp"
Oct 30 13:32:06.615758 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:06.615731    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp"]
Oct 30 13:32:06.616367 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:32:06.616344    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09bcc5ff_a33b_4699_a797_58b18a5f63bd.slice/crio-b863f27ba21dac577c59806167067db7a02c727cda2a36bd96f315c0195c38b0 WatchSource:0}: Error finding container b863f27ba21dac577c59806167067db7a02c727cda2a36bd96f315c0195c38b0: Status 404 returned error can't find the container with id b863f27ba21dac577c59806167067db7a02c727cda2a36bd96f315c0195c38b0
Oct 30 13:32:06.619235 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:06.619203    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:32:06.619290 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:06.619270    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:32:06.619334 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:06.619299    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:32:07.108438 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:07.108400    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-cda0d9/rate-test-model-cda0d9-kserve-9d99f655c-swqhw" event={"ID":"d21066d6-fdb1-4c53-8e3b-b8b254112229","Type":"ContainerStarted","Data":"28611909a0fd3f9b745f018b3f4e78717ff98dbebe07111a5e889fd8b06a6aa5"}
Oct 30 13:32:07.108657 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:07.108451    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-cda0d9/rate-test-model-cda0d9-kserve-9d99f655c-swqhw" event={"ID":"d21066d6-fdb1-4c53-8e3b-b8b254112229","Type":"ContainerStarted","Data":"f68408d50e68cbc1cf524b00af6533710dcb67d32f8c736e74458128f084762e"}
Oct 30 13:32:07.110277 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:07.110250    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp" event={"ID":"09bcc5ff-a33b-4699-a797-58b18a5f63bd","Type":"ContainerStarted","Data":"5e78e34aa30d512d552790b9f84b026d6c4a5dca1b4ea04c843cefa29daad33e"}
Oct 30 13:32:07.110414 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:07.110298    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp" event={"ID":"09bcc5ff-a33b-4699-a797-58b18a5f63bd","Type":"ContainerStarted","Data":"b863f27ba21dac577c59806167067db7a02c727cda2a36bd96f315c0195c38b0"}
Oct 30 13:32:07.204619 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:07.203735    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w6-cda0d9/rate-test-model-cda0d9-kserve-9d99f655c-swqhw" podStartSLOduration=2.203715671 podStartE2EDuration="2.203715671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:32:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:32:07.155711352 +0000 UTC m=+2426.322076003" watchObservedRunningTime="2026-10-30 13:32:07.203715671 +0000 UTC m=+2426.370080322"
Oct 30 13:32:07.207967 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:07.207910    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp" podStartSLOduration=2.20789869 podStartE2EDuration="2.20789869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:32:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:32:07.203920002 +0000 UTC m=+2426.370284652" watchObservedRunningTime="2026-10-30 13:32:07.20789869 +0000 UTC m=+2426.374263335"
Oct 30 13:32:07.288590 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:07.286976    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp"
Oct 30 13:32:07.295626 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:07.295540    4364 patch_prober.go:29] interesting pod/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body=
Oct 30 13:32:07.295626 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:07.295614    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp" podUID="09bcc5ff-a33b-4699-a797-58b18a5f63bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused"
Oct 30 13:32:07.809377 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:07.809340    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w6-cda0d9/rate-test-model-cda0d9-kserve-9d99f655c-swqhw"
Oct 30 13:32:08.299176 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:08.299140    4364 patch_prober.go:29] interesting pod/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body=
Oct 30 13:32:08.299695 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:08.299654    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp" podUID="09bcc5ff-a33b-4699-a797-58b18a5f63bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused"
Oct 30 13:32:09.298624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:09.295922    4364 patch_prober.go:29] interesting pod/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body=
Oct 30 13:32:09.298624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:09.295978    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp" podUID="09bcc5ff-a33b-4699-a797-58b18a5f63bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused"
Oct 30 13:32:10.287620 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:10.286893    4364 patch_prober.go:29] interesting pod/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body=
Oct 30 13:32:10.287620 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:10.287007    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp" podUID="09bcc5ff-a33b-4699-a797-58b18a5f63bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused"
Oct 30 13:32:11.303891 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:11.303850    4364 patch_prober.go:29] interesting pod/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body=
Oct 30 13:32:11.304354 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:11.303913    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp" podUID="09bcc5ff-a33b-4699-a797-58b18a5f63bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused"
Oct 30 13:32:11.528250 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:32:11.528217    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:32:12.069886 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:12.069859    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-4b0ea9/auth-test-model-4b0ea9-kserve-86458657c-m5qrl"
Oct 30 13:32:12.107244 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:12.107210    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-4b0ea9/auth-test-model-4b0ea9-kserve-86458657c-m5qrl"
Oct 30 13:32:12.164844 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:12.164815    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-4b0ea9/auth-test-model-4b0ea9-kserve-86458657c-m5qrl"
Oct 30 13:32:12.290272 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:12.290230    4364 patch_prober.go:29] interesting pod/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body=
Oct 30 13:32:12.290957 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:12.290924    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp" podUID="09bcc5ff-a33b-4699-a797-58b18a5f63bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused"
Oct 30 13:32:13.289995 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:13.289964    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp"
Oct 30 13:32:13.290412 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:13.290338    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp"
Oct 30 13:32:13.291107 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:13.291083    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp"
Oct 30 13:32:15.809665 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:15.809632    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w6-cda0d9/rate-test-model-cda0d9-kserve-9d99f655c-swqhw"
Oct 30 13:32:15.822716 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:15.822691    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w6-cda0d9/rate-test-model-cda0d9-kserve-9d99f655c-swqhw"
Oct 30 13:32:16.197562 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:16.197536    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w6-cda0d9/rate-test-model-cda0d9-kserve-9d99f655c-swqhw"
Oct 30 13:32:18.869394 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:18.869359    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-8b77b0/sub-test-model-8b77b0-kserve-5f5b478b5c-dcqbx"]
Oct 30 13:32:21.201109 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:21.201078    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6da60a-589f9f6ddb-vcldm"]
Oct 30 13:32:21.204431 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:21.204413    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6da60a-589f9f6ddb-vcldm"
Oct 30 13:32:21.225735 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:21.225711    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6da60a-589f9f6ddb-vcldm"]
Oct 30 13:32:21.232036 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:21.231992    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-6da60a-serving-cert\""
Oct 30 13:32:21.251473 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:21.251441    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29ab696e-02cd-40c3-a170-32545b8407f3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-6da60a-589f9f6ddb-vcldm\" (UID: \"29ab696e-02cd-40c3-a170-32545b8407f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6da60a-589f9f6ddb-vcldm"
Oct 30 13:32:21.251623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:21.251504    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29ab696e-02cd-40c3-a170-32545b8407f3-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-6da60a-589f9f6ddb-vcldm\" (UID: \"29ab696e-02cd-40c3-a170-32545b8407f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6da60a-589f9f6ddb-vcldm"
Oct 30 13:32:21.251764 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:21.251743    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29ab696e-02cd-40c3-a170-32545b8407f3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-6da60a-589f9f6ddb-vcldm\" (UID: \"29ab696e-02cd-40c3-a170-32545b8407f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6da60a-589f9f6ddb-vcldm"
Oct 30 13:32:21.251829 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:21.251816    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b95g\" (UniqueName: \"kubernetes.io/projected/29ab696e-02cd-40c3-a170-32545b8407f3-kube-api-access-7b95g\") pod \"maas-api-e2e-shared-b-w0-6da60a-589f9f6ddb-vcldm\" (UID: \"29ab696e-02cd-40c3-a170-32545b8407f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6da60a-589f9f6ddb-vcldm"
Oct 30 13:32:21.353670 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:21.353639    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29ab696e-02cd-40c3-a170-32545b8407f3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-6da60a-589f9f6ddb-vcldm\" (UID: \"29ab696e-02cd-40c3-a170-32545b8407f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6da60a-589f9f6ddb-vcldm"
Oct 30 13:32:21.353847 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:21.353722    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b95g\" (UniqueName: \"kubernetes.io/projected/29ab696e-02cd-40c3-a170-32545b8407f3-kube-api-access-7b95g\") pod \"maas-api-e2e-shared-b-w0-6da60a-589f9f6ddb-vcldm\" (UID: \"29ab696e-02cd-40c3-a170-32545b8407f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6da60a-589f9f6ddb-vcldm"
Oct 30 13:32:21.353947 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:32:21.353888    4364 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6da60a-serving-cert: secret "maas-api-e2e-shared-b-w0-6da60a-serving-cert" not found
Oct 30 13:32:21.354008 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:32:21.353980    4364 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29ab696e-02cd-40c3-a170-32545b8407f3-maas-api-tls podName:29ab696e-02cd-40c3-a170-32545b8407f3 nodeName:}" failed. No retries permitted until 2026-10-30 13:32:21.853947349 +0000 UTC m=+2441.020311978 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/29ab696e-02cd-40c3-a170-32545b8407f3-maas-api-tls") pod "maas-api-e2e-shared-b-w0-6da60a-589f9f6ddb-vcldm" (UID: "29ab696e-02cd-40c3-a170-32545b8407f3") : secret "maas-api-e2e-shared-b-w0-6da60a-serving-cert" not found
Oct 30 13:32:21.354074 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:21.354002    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29ab696e-02cd-40c3-a170-32545b8407f3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-6da60a-589f9f6ddb-vcldm\" (UID: \"29ab696e-02cd-40c3-a170-32545b8407f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6da60a-589f9f6ddb-vcldm"
Oct 30 13:32:21.354284 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:21.354198    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29ab696e-02cd-40c3-a170-32545b8407f3-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-6da60a-589f9f6ddb-vcldm\" (UID: \"29ab696e-02cd-40c3-a170-32545b8407f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6da60a-589f9f6ddb-vcldm"
Oct 30 13:32:21.355191 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:21.355165    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29ab696e-02cd-40c3-a170-32545b8407f3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-6da60a-589f9f6ddb-vcldm\" (UID: \"29ab696e-02cd-40c3-a170-32545b8407f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6da60a-589f9f6ddb-vcldm"
Oct 30 13:32:21.358570 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:21.358548    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29ab696e-02cd-40c3-a170-32545b8407f3-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-6da60a-589f9f6ddb-vcldm\" (UID: \"29ab696e-02cd-40c3-a170-32545b8407f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6da60a-589f9f6ddb-vcldm"
Oct 30 13:32:21.377741 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:21.377718    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b95g\" (UniqueName: \"kubernetes.io/projected/29ab696e-02cd-40c3-a170-32545b8407f3-kube-api-access-7b95g\") pod \"maas-api-e2e-shared-b-w0-6da60a-589f9f6ddb-vcldm\" (UID: \"29ab696e-02cd-40c3-a170-32545b8407f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6da60a-589f9f6ddb-vcldm"
Oct 30 13:32:21.440239 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:21.440183    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:32:21.862196 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:21.862159    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29ab696e-02cd-40c3-a170-32545b8407f3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-6da60a-589f9f6ddb-vcldm\" (UID: \"29ab696e-02cd-40c3-a170-32545b8407f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6da60a-589f9f6ddb-vcldm"
Oct 30 13:32:21.864537 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:21.864516    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29ab696e-02cd-40c3-a170-32545b8407f3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-6da60a-589f9f6ddb-vcldm\" (UID: \"29ab696e-02cd-40c3-a170-32545b8407f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6da60a-589f9f6ddb-vcldm"
Oct 30 13:32:22.117424 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:22.117401    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6da60a-589f9f6ddb-vcldm"
Oct 30 13:32:22.427044 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:22.427014    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6da60a-589f9f6ddb-vcldm"]
Oct 30 13:32:22.427804 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:32:22.427779    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29ab696e_02cd_40c3_a170_32545b8407f3.slice/crio-4bab3638f76027a461a3f567d74b891269dab44088098f02bbd1b754c32aba40 WatchSource:0}: Error finding container 4bab3638f76027a461a3f567d74b891269dab44088098f02bbd1b754c32aba40: Status 404 returned error can't find the container with id 4bab3638f76027a461a3f567d74b891269dab44088098f02bbd1b754c32aba40
Oct 30 13:32:23.379961 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:23.379900    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6da60a-589f9f6ddb-vcldm" event={"ID":"29ab696e-02cd-40c3-a170-32545b8407f3","Type":"ContainerStarted","Data":"74f1e147dc5b3a66551e6586e81bd3863e4a3e426f84e630d846cd4f11090fed"}
Oct 30 13:32:23.380143 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:23.380052    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6da60a-589f9f6ddb-vcldm" event={"ID":"29ab696e-02cd-40c3-a170-32545b8407f3","Type":"ContainerStarted","Data":"4bab3638f76027a461a3f567d74b891269dab44088098f02bbd1b754c32aba40"}
Oct 30 13:32:23.380143 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:23.380091    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6da60a-589f9f6ddb-vcldm"
Oct 30 13:32:23.541620 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:23.534921    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6da60a-589f9f6ddb-vcldm" podStartSLOduration=2.050504147 podStartE2EDuration="2.534899612s" totalImagesPullingTime="484.395465ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:32:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:32:23.464076521 +0000 UTC m=+2442.630441169" watchObservedRunningTime="2026-10-30 13:32:23.534899612 +0000 UTC m=+2442.701264261"
Oct 30 13:32:24.471628 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:24.468350    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6" podUID="b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:32:29.510246 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:29.510205    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6da60a-589f9f6ddb-vcldm"
Oct 30 13:32:33.895212 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:33.895164    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-8b77b0/sub-test-model-8b77b0-kserve-5f5b478b5c-dcqbx" podUID="eb255481-3535-4bbe-a2c5-b4c1ba925fe4" containerName="main" containerID="cri-o://53486d32b0639e288121c74089dce280c3d588465574eceda30b3797bda049e9" gracePeriod=45
Oct 30 13:32:34.173898 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:34.173874    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-8b77b0/sub-test-model-8b77b0-kserve-5f5b478b5c-dcqbx"
Oct 30 13:32:34.210913 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:34.210883    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb255481-3535-4bbe-a2c5-b4c1ba925fe4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb255481-3535-4bbe-a2c5-b4c1ba925fe4-dshm\") pod \"eb255481-3535-4bbe-a2c5-b4c1ba925fe4\" (UID: \"eb255481-3535-4bbe-a2c5-b4c1ba925fe4\") "
Oct 30 13:32:34.211060 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:34.211016    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb255481-3535-4bbe-a2c5-b4c1ba925fe4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb255481-3535-4bbe-a2c5-b4c1ba925fe4-tmp-dir\") pod \"eb255481-3535-4bbe-a2c5-b4c1ba925fe4\" (UID: \"eb255481-3535-4bbe-a2c5-b4c1ba925fe4\") "
Oct 30 13:32:34.211126 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:34.211110    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb255481-3535-4bbe-a2c5-b4c1ba925fe4-kube-api-access-srskp\" (UniqueName: \"kubernetes.io/projected/eb255481-3535-4bbe-a2c5-b4c1ba925fe4-kube-api-access-srskp\") pod \"eb255481-3535-4bbe-a2c5-b4c1ba925fe4\" (UID: \"eb255481-3535-4bbe-a2c5-b4c1ba925fe4\") "
Oct 30 13:32:34.211206 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:34.211185    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb255481-3535-4bbe-a2c5-b4c1ba925fe4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb255481-3535-4bbe-a2c5-b4c1ba925fe4-model-cache\") pod \"eb255481-3535-4bbe-a2c5-b4c1ba925fe4\" (UID: \"eb255481-3535-4bbe-a2c5-b4c1ba925fe4\") "
Oct 30 13:32:34.211692 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:34.211264    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb255481-3535-4bbe-a2c5-b4c1ba925fe4-tmp-dir" pod "eb255481-3535-4bbe-a2c5-b4c1ba925fe4" (UID: "eb255481-3535-4bbe-a2c5-b4c1ba925fe4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:32:34.211692 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:34.211408    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb255481-3535-4bbe-a2c5-b4c1ba925fe4-tls-certs\" (UniqueName: \"kubernetes.io/secret/eb255481-3535-4bbe-a2c5-b4c1ba925fe4-tls-certs\") pod \"eb255481-3535-4bbe-a2c5-b4c1ba925fe4\" (UID: \"eb255481-3535-4bbe-a2c5-b4c1ba925fe4\") "
Oct 30 13:32:34.211692 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:34.211550    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb255481-3535-4bbe-a2c5-b4c1ba925fe4-home\" (UniqueName: \"kubernetes.io/empty-dir/eb255481-3535-4bbe-a2c5-b4c1ba925fe4-home\") pod \"eb255481-3535-4bbe-a2c5-b4c1ba925fe4\" (UID: \"eb255481-3535-4bbe-a2c5-b4c1ba925fe4\") "
Oct 30 13:32:34.211958 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:34.211937    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb255481-3535-4bbe-a2c5-b4c1ba925fe4-home" pod "eb255481-3535-4bbe-a2c5-b4c1ba925fe4" (UID: "eb255481-3535-4bbe-a2c5-b4c1ba925fe4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:32:34.212023 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:34.211998    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb255481-3535-4bbe-a2c5-b4c1ba925fe4-model-cache" pod "eb255481-3535-4bbe-a2c5-b4c1ba925fe4" (UID: "eb255481-3535-4bbe-a2c5-b4c1ba925fe4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:32:34.212283 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:34.212267    4364 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb255481-3535-4bbe-a2c5-b4c1ba925fe4-tmp-dir\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:32:34.212283 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:34.212284    4364 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb255481-3535-4bbe-a2c5-b4c1ba925fe4-model-cache\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:32:34.212283 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:34.212293    4364 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb255481-3535-4bbe-a2c5-b4c1ba925fe4-home\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:32:34.213673 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:34.213634    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb255481-3535-4bbe-a2c5-b4c1ba925fe4-kube-api-access-srskp" pod "eb255481-3535-4bbe-a2c5-b4c1ba925fe4" (UID: "eb255481-3535-4bbe-a2c5-b4c1ba925fe4"). InnerVolumeSpecName "kube-api-access-srskp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:32:34.213780 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:34.213712    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb255481-3535-4bbe-a2c5-b4c1ba925fe4-dshm" pod "eb255481-3535-4bbe-a2c5-b4c1ba925fe4" (UID: "eb255481-3535-4bbe-a2c5-b4c1ba925fe4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:32:34.213953 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:34.213927    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb255481-3535-4bbe-a2c5-b4c1ba925fe4-tls-certs" pod "eb255481-3535-4bbe-a2c5-b4c1ba925fe4" (UID: "eb255481-3535-4bbe-a2c5-b4c1ba925fe4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:32:34.313509 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:34.313477    4364 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb255481-3535-4bbe-a2c5-b4c1ba925fe4-tls-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:32:34.313509 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:34.313503    4364 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb255481-3535-4bbe-a2c5-b4c1ba925fe4-dshm\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:32:34.313509 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:34.313514    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srskp\" (UniqueName: \"kubernetes.io/projected/eb255481-3535-4bbe-a2c5-b4c1ba925fe4-kube-api-access-srskp\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:32:34.609167 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:34.609079    4364 generic.go:356] "Generic (PLEG): container finished" podID="eb255481-3535-4bbe-a2c5-b4c1ba925fe4" containerID="53486d32b0639e288121c74089dce280c3d588465574eceda30b3797bda049e9" exitCode=0
Oct 30 13:32:34.609341 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:34.609281    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-8b77b0/sub-test-model-8b77b0-kserve-5f5b478b5c-dcqbx" event={"ID":"eb255481-3535-4bbe-a2c5-b4c1ba925fe4","Type":"ContainerDied","Data":"53486d32b0639e288121c74089dce280c3d588465574eceda30b3797bda049e9"}
Oct 30 13:32:34.609341 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:34.609321    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-8b77b0/sub-test-model-8b77b0-kserve-5f5b478b5c-dcqbx" event={"ID":"eb255481-3535-4bbe-a2c5-b4c1ba925fe4","Type":"ContainerDied","Data":"2e1a4b25958a238d3edc298c4cbb4fea39c072e673ba09b87b554bffbd3b0a9c"}
Oct 30 13:32:34.609463 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:34.609345    4364 scope.go:122] "RemoveContainer" containerID="53486d32b0639e288121c74089dce280c3d588465574eceda30b3797bda049e9"
Oct 30 13:32:34.609544 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:34.609527    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-8b77b0/sub-test-model-8b77b0-kserve-5f5b478b5c-dcqbx"
Oct 30 13:32:34.623044 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:34.623025    4364 scope.go:122] "RemoveContainer" containerID="53486d32b0639e288121c74089dce280c3d588465574eceda30b3797bda049e9"
Oct 30 13:32:34.623310 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:32:34.623288    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53486d32b0639e288121c74089dce280c3d588465574eceda30b3797bda049e9\": container with ID starting with 53486d32b0639e288121c74089dce280c3d588465574eceda30b3797bda049e9 not found: ID does not exist" containerID="53486d32b0639e288121c74089dce280c3d588465574eceda30b3797bda049e9"
Oct 30 13:32:34.623359 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:34.623319    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53486d32b0639e288121c74089dce280c3d588465574eceda30b3797bda049e9"} err="failed to get container status \"53486d32b0639e288121c74089dce280c3d588465574eceda30b3797bda049e9\": rpc error: code = NotFound desc = could not find container \"53486d32b0639e288121c74089dce280c3d588465574eceda30b3797bda049e9\": container with ID starting with 53486d32b0639e288121c74089dce280c3d588465574eceda30b3797bda049e9 not found: ID does not exist"
Oct 30 13:32:34.664083 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:34.664057    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-8b77b0/sub-test-model-8b77b0-kserve-5f5b478b5c-dcqbx"]
Oct 30 13:32:34.675987 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:34.675967    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-8b77b0/sub-test-model-8b77b0-kserve-5f5b478b5c-dcqbx"]
Oct 30 13:32:34.971505 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:34.971475    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb255481-3535-4bbe-a2c5-b4c1ba925fe4" path="/var/lib/kubelet/pods/eb255481-3535-4bbe-a2c5-b4c1ba925fe4/volumes"
Oct 30 13:32:40.344955 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:40.344913    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-5a6281/sub-test-model-5a6281-kserve-694f48b47-4j6cv"]
Oct 30 13:32:41.506120 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:32:41.506062    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:32:42.471721 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:42.471692    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb"]
Oct 30 13:32:42.471937 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:42.471900    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb" podUID="59c001b5-b0f7-4898-ba90-aa8a0b96a6e7" containerName="istio-proxy" containerID="cri-o://9388d1a63e7622e5bdab5846d099b319327e2a2d5686a27207ef54b1dd247a86" gracePeriod=30
Oct 30 13:32:43.487846 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:43.487812    4364 scope.go:122] "RemoveContainer" containerID="7c9b0f2c2fa0e22e9f34203bda8d399630f5023140bb530bd45d6b63709156f9"
Oct 30 13:32:43.503140 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:43.503116    4364 scope.go:122] "RemoveContainer" containerID="d4bd800a0da8f0df9a85a23307d387b612e2445c1c55bd147ac03d9a7ac9147e"
Oct 30 13:32:47.768405 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:47.768168    4364 generic.go:356] "Generic (PLEG): container finished" podID="59c001b5-b0f7-4898-ba90-aa8a0b96a6e7" containerID="9388d1a63e7622e5bdab5846d099b319327e2a2d5686a27207ef54b1dd247a86" exitCode=0
Oct 30 13:32:47.768405 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:47.768276    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb" event={"ID":"59c001b5-b0f7-4898-ba90-aa8a0b96a6e7","Type":"ContainerDied","Data":"9388d1a63e7622e5bdab5846d099b319327e2a2d5686a27207ef54b1dd247a86"}
Oct 30 13:32:48.073608 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:48.073570    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb"
Oct 30 13:32:48.114683 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:48.114645    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59c001b5-b0f7-4898-ba90-aa8a0b96a6e7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59c001b5-b0f7-4898-ba90-aa8a0b96a6e7-istio-data\") pod \"59c001b5-b0f7-4898-ba90-aa8a0b96a6e7\" (UID: \"59c001b5-b0f7-4898-ba90-aa8a0b96a6e7\") "
Oct 30 13:32:48.114896 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:48.114876    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59c001b5-b0f7-4898-ba90-aa8a0b96a6e7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59c001b5-b0f7-4898-ba90-aa8a0b96a6e7-workload-socket\") pod \"59c001b5-b0f7-4898-ba90-aa8a0b96a6e7\" (UID: \"59c001b5-b0f7-4898-ba90-aa8a0b96a6e7\") "
Oct 30 13:32:48.114943 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:48.114920    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59c001b5-b0f7-4898-ba90-aa8a0b96a6e7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59c001b5-b0f7-4898-ba90-aa8a0b96a6e7-credential-socket\") pod \"59c001b5-b0f7-4898-ba90-aa8a0b96a6e7\" (UID: \"59c001b5-b0f7-4898-ba90-aa8a0b96a6e7\") "
Oct 30 13:32:48.114980 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:48.114880    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59c001b5-b0f7-4898-ba90-aa8a0b96a6e7-istio-data" pod "59c001b5-b0f7-4898-ba90-aa8a0b96a6e7" (UID: "59c001b5-b0f7-4898-ba90-aa8a0b96a6e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:32:48.115051 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:48.115035    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59c001b5-b0f7-4898-ba90-aa8a0b96a6e7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59c001b5-b0f7-4898-ba90-aa8a0b96a6e7-istiod-ca-cert\") pod \"59c001b5-b0f7-4898-ba90-aa8a0b96a6e7\" (UID: \"59c001b5-b0f7-4898-ba90-aa8a0b96a6e7\") "
Oct 30 13:32:48.115093 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:48.115065    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59c001b5-b0f7-4898-ba90-aa8a0b96a6e7-workload-socket" pod "59c001b5-b0f7-4898-ba90-aa8a0b96a6e7" (UID: "59c001b5-b0f7-4898-ba90-aa8a0b96a6e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:32:48.115144 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:48.115132    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59c001b5-b0f7-4898-ba90-aa8a0b96a6e7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59c001b5-b0f7-4898-ba90-aa8a0b96a6e7-istio-envoy\") pod \"59c001b5-b0f7-4898-ba90-aa8a0b96a6e7\" (UID: \"59c001b5-b0f7-4898-ba90-aa8a0b96a6e7\") "
Oct 30 13:32:48.115181 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:48.115150    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59c001b5-b0f7-4898-ba90-aa8a0b96a6e7-credential-socket" pod "59c001b5-b0f7-4898-ba90-aa8a0b96a6e7" (UID: "59c001b5-b0f7-4898-ba90-aa8a0b96a6e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:32:48.115221 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:48.115206    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/59c001b5-b0f7-4898-ba90-aa8a0b96a6e7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59c001b5-b0f7-4898-ba90-aa8a0b96a6e7-istio-podinfo\") pod \"59c001b5-b0f7-4898-ba90-aa8a0b96a6e7\" (UID: \"59c001b5-b0f7-4898-ba90-aa8a0b96a6e7\") "
Oct 30 13:32:48.115285 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:48.115272    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59c001b5-b0f7-4898-ba90-aa8a0b96a6e7-istio-token\" (UniqueName: \"kubernetes.io/projected/59c001b5-b0f7-4898-ba90-aa8a0b96a6e7-istio-token\") pod \"59c001b5-b0f7-4898-ba90-aa8a0b96a6e7\" (UID: \"59c001b5-b0f7-4898-ba90-aa8a0b96a6e7\") "
Oct 30 13:32:48.115345 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:48.115296    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59c001b5-b0f7-4898-ba90-aa8a0b96a6e7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59c001b5-b0f7-4898-ba90-aa8a0b96a6e7-workload-certs\") pod \"59c001b5-b0f7-4898-ba90-aa8a0b96a6e7\" (UID: \"59c001b5-b0f7-4898-ba90-aa8a0b96a6e7\") "
Oct 30 13:32:48.115345 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:48.115315    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59c001b5-b0f7-4898-ba90-aa8a0b96a6e7-kube-api-access-mdn7s\" (UniqueName: \"kubernetes.io/projected/59c001b5-b0f7-4898-ba90-aa8a0b96a6e7-kube-api-access-mdn7s\") pod \"59c001b5-b0f7-4898-ba90-aa8a0b96a6e7\" (UID: \"59c001b5-b0f7-4898-ba90-aa8a0b96a6e7\") "
Oct 30 13:32:48.115458 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:48.115364    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59c001b5-b0f7-4898-ba90-aa8a0b96a6e7-istiod-ca-cert" pod "59c001b5-b0f7-4898-ba90-aa8a0b96a6e7" (UID: "59c001b5-b0f7-4898-ba90-aa8a0b96a6e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:32:48.115797 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:48.115768    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59c001b5-b0f7-4898-ba90-aa8a0b96a6e7-workload-certs" pod "59c001b5-b0f7-4898-ba90-aa8a0b96a6e7" (UID: "59c001b5-b0f7-4898-ba90-aa8a0b96a6e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:32:48.116179 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:48.116152    4364 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59c001b5-b0f7-4898-ba90-aa8a0b96a6e7-istiod-ca-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:32:48.116326 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:48.116182    4364 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59c001b5-b0f7-4898-ba90-aa8a0b96a6e7-workload-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:32:48.116326 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:48.116197    4364 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59c001b5-b0f7-4898-ba90-aa8a0b96a6e7-istio-data\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:32:48.116326 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:48.116210    4364 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59c001b5-b0f7-4898-ba90-aa8a0b96a6e7-workload-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:32:48.116326 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:48.116225    4364 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59c001b5-b0f7-4898-ba90-aa8a0b96a6e7-credential-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:32:48.117450 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:48.117423    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/59c001b5-b0f7-4898-ba90-aa8a0b96a6e7-istio-podinfo" pod "59c001b5-b0f7-4898-ba90-aa8a0b96a6e7" (UID: "59c001b5-b0f7-4898-ba90-aa8a0b96a6e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:32:48.117558 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:48.117459    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59c001b5-b0f7-4898-ba90-aa8a0b96a6e7-istio-envoy" pod "59c001b5-b0f7-4898-ba90-aa8a0b96a6e7" (UID: "59c001b5-b0f7-4898-ba90-aa8a0b96a6e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:32:48.117558 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:48.117539    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59c001b5-b0f7-4898-ba90-aa8a0b96a6e7-kube-api-access-mdn7s" pod "59c001b5-b0f7-4898-ba90-aa8a0b96a6e7" (UID: "59c001b5-b0f7-4898-ba90-aa8a0b96a6e7"). InnerVolumeSpecName "kube-api-access-mdn7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:32:48.117682 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:48.117665    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59c001b5-b0f7-4898-ba90-aa8a0b96a6e7-istio-token" pod "59c001b5-b0f7-4898-ba90-aa8a0b96a6e7" (UID: "59c001b5-b0f7-4898-ba90-aa8a0b96a6e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:32:48.217540 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:48.217503    4364 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59c001b5-b0f7-4898-ba90-aa8a0b96a6e7-istio-envoy\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:32:48.217540 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:48.217537    4364 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59c001b5-b0f7-4898-ba90-aa8a0b96a6e7-istio-podinfo\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:32:48.217540 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:48.217545    4364 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59c001b5-b0f7-4898-ba90-aa8a0b96a6e7-istio-token\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:32:48.217835 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:48.217554    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdn7s\" (UniqueName: \"kubernetes.io/projected/59c001b5-b0f7-4898-ba90-aa8a0b96a6e7-kube-api-access-mdn7s\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:32:48.777440 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:48.777406    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb" event={"ID":"59c001b5-b0f7-4898-ba90-aa8a0b96a6e7","Type":"ContainerDied","Data":"7f352153d5ad5d4b6bbf89f4b703b649ab5261ba5a95bc9ac130890439deb7db"}
Oct 30 13:32:48.777440 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:48.777427    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb"
Oct 30 13:32:48.777994 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:48.777467    4364 scope.go:122] "RemoveContainer" containerID="9388d1a63e7622e5bdab5846d099b319327e2a2d5686a27207ef54b1dd247a86"
Oct 30 13:32:48.833318 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:48.833285    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb"]
Oct 30 13:32:48.844882 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:48.844837    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-z99tb"]
Oct 30 13:32:48.972998 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:48.972961    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59c001b5-b0f7-4898-ba90-aa8a0b96a6e7" path="/var/lib/kubelet/pods/59c001b5-b0f7-4898-ba90-aa8a0b96a6e7/volumes"
Oct 30 13:32:51.034021 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:51.033984    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b97b9de-kvsj7"]
Oct 30 13:32:51.045350 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:51.045315    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b97b9de-kvsj7"]
Oct 30 13:32:52.972806 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:52.972774    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adc1ed51-9afb-4fe4-80ef-92d0109836e0" path="/var/lib/kubelet/pods/adc1ed51-9afb-4fe4-80ef-92d0109836e0/volumes"
Oct 30 13:32:55.369470 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:55.369380    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-5a6281/sub-test-model-5a6281-kserve-694f48b47-4j6cv" podUID="10cce01a-1759-4617-bf74-5433cb240a30" containerName="main" containerID="cri-o://1e6ef79452d544f8edd2bf46a2957db2ff671772df03b0cafed7739150aec696" gracePeriod=45
Oct 30 13:32:55.675061 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:55.675035    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-5a6281/sub-test-model-5a6281-kserve-694f48b47-4j6cv"
Oct 30 13:32:55.853873 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:55.853831    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10cce01a-1759-4617-bf74-5433cb240a30-kube-api-access-ccgvh\" (UniqueName: \"kubernetes.io/projected/10cce01a-1759-4617-bf74-5433cb240a30-kube-api-access-ccgvh\") pod \"10cce01a-1759-4617-bf74-5433cb240a30\" (UID: \"10cce01a-1759-4617-bf74-5433cb240a30\") "
Oct 30 13:32:55.854048 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:55.853961    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10cce01a-1759-4617-bf74-5433cb240a30-home\" (UniqueName: \"kubernetes.io/empty-dir/10cce01a-1759-4617-bf74-5433cb240a30-home\") pod \"10cce01a-1759-4617-bf74-5433cb240a30\" (UID: \"10cce01a-1759-4617-bf74-5433cb240a30\") "
Oct 30 13:32:55.854048 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:55.853999    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10cce01a-1759-4617-bf74-5433cb240a30-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10cce01a-1759-4617-bf74-5433cb240a30-model-cache\") pod \"10cce01a-1759-4617-bf74-5433cb240a30\" (UID: \"10cce01a-1759-4617-bf74-5433cb240a30\") "
Oct 30 13:32:55.854189 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:55.854094    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10cce01a-1759-4617-bf74-5433cb240a30-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10cce01a-1759-4617-bf74-5433cb240a30-tmp-dir\") pod \"10cce01a-1759-4617-bf74-5433cb240a30\" (UID: \"10cce01a-1759-4617-bf74-5433cb240a30\") "
Oct 30 13:32:55.854189 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:55.854139    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10cce01a-1759-4617-bf74-5433cb240a30-tls-certs\" (UniqueName: \"kubernetes.io/secret/10cce01a-1759-4617-bf74-5433cb240a30-tls-certs\") pod \"10cce01a-1759-4617-bf74-5433cb240a30\" (UID: \"10cce01a-1759-4617-bf74-5433cb240a30\") "
Oct 30 13:32:55.854317 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:55.854221    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10cce01a-1759-4617-bf74-5433cb240a30-dshm\" (UniqueName: \"kubernetes.io/empty-dir/10cce01a-1759-4617-bf74-5433cb240a30-dshm\") pod \"10cce01a-1759-4617-bf74-5433cb240a30\" (UID: \"10cce01a-1759-4617-bf74-5433cb240a30\") "
Oct 30 13:32:55.854317 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:55.854223    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10cce01a-1759-4617-bf74-5433cb240a30-home" pod "10cce01a-1759-4617-bf74-5433cb240a30" (UID: "10cce01a-1759-4617-bf74-5433cb240a30"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:32:55.854425 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:55.854345    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10cce01a-1759-4617-bf74-5433cb240a30-model-cache" pod "10cce01a-1759-4617-bf74-5433cb240a30" (UID: "10cce01a-1759-4617-bf74-5433cb240a30"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:32:55.854425 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:55.854373    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10cce01a-1759-4617-bf74-5433cb240a30-tmp-dir" pod "10cce01a-1759-4617-bf74-5433cb240a30" (UID: "10cce01a-1759-4617-bf74-5433cb240a30"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:32:55.855126 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:55.855100    4364 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10cce01a-1759-4617-bf74-5433cb240a30-home\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:32:55.855257 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:55.855130    4364 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10cce01a-1759-4617-bf74-5433cb240a30-model-cache\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:32:55.855257 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:55.855146    4364 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10cce01a-1759-4617-bf74-5433cb240a30-tmp-dir\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:32:55.856651 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:55.856626    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10cce01a-1759-4617-bf74-5433cb240a30-kube-api-access-ccgvh" pod "10cce01a-1759-4617-bf74-5433cb240a30" (UID: "10cce01a-1759-4617-bf74-5433cb240a30"). InnerVolumeSpecName "kube-api-access-ccgvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:32:55.856780 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:55.856658    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10cce01a-1759-4617-bf74-5433cb240a30-dshm" pod "10cce01a-1759-4617-bf74-5433cb240a30" (UID: "10cce01a-1759-4617-bf74-5433cb240a30"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:32:55.856780 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:55.856676    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10cce01a-1759-4617-bf74-5433cb240a30-tls-certs" pod "10cce01a-1759-4617-bf74-5433cb240a30" (UID: "10cce01a-1759-4617-bf74-5433cb240a30"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:32:55.857331 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:55.857310    4364 generic.go:356] "Generic (PLEG): container finished" podID="10cce01a-1759-4617-bf74-5433cb240a30" containerID="1e6ef79452d544f8edd2bf46a2957db2ff671772df03b0cafed7739150aec696" exitCode=0
Oct 30 13:32:55.857418 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:55.857360    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-5a6281/sub-test-model-5a6281-kserve-694f48b47-4j6cv"
Oct 30 13:32:55.857418 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:55.857394    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-5a6281/sub-test-model-5a6281-kserve-694f48b47-4j6cv" event={"ID":"10cce01a-1759-4617-bf74-5433cb240a30","Type":"ContainerDied","Data":"1e6ef79452d544f8edd2bf46a2957db2ff671772df03b0cafed7739150aec696"}
Oct 30 13:32:55.857517 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:55.857447    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-5a6281/sub-test-model-5a6281-kserve-694f48b47-4j6cv" event={"ID":"10cce01a-1759-4617-bf74-5433cb240a30","Type":"ContainerDied","Data":"446d2a7739e732f9a16e887189be383f69b1655d759befe589e316f23faea915"}
Oct 30 13:32:55.857517 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:55.857468    4364 scope.go:122] "RemoveContainer" containerID="1e6ef79452d544f8edd2bf46a2957db2ff671772df03b0cafed7739150aec696"
Oct 30 13:32:55.885179 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:55.885158    4364 scope.go:122] "RemoveContainer" containerID="1e6ef79452d544f8edd2bf46a2957db2ff671772df03b0cafed7739150aec696"
Oct 30 13:32:55.885488 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:32:55.885467    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e6ef79452d544f8edd2bf46a2957db2ff671772df03b0cafed7739150aec696\": container with ID starting with 1e6ef79452d544f8edd2bf46a2957db2ff671772df03b0cafed7739150aec696 not found: ID does not exist" containerID="1e6ef79452d544f8edd2bf46a2957db2ff671772df03b0cafed7739150aec696"
Oct 30 13:32:55.885562 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:55.885501    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e6ef79452d544f8edd2bf46a2957db2ff671772df03b0cafed7739150aec696"} err="failed to get container status \"1e6ef79452d544f8edd2bf46a2957db2ff671772df03b0cafed7739150aec696\": rpc error: code = NotFound desc = could not find container \"1e6ef79452d544f8edd2bf46a2957db2ff671772df03b0cafed7739150aec696\": container with ID starting with 1e6ef79452d544f8edd2bf46a2957db2ff671772df03b0cafed7739150aec696 not found: ID does not exist"
Oct 30 13:32:55.919571 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:55.919487    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-5a6281/sub-test-model-5a6281-kserve-694f48b47-4j6cv"]
Oct 30 13:32:55.924376 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:55.924355    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-5a6281/sub-test-model-5a6281-kserve-694f48b47-4j6cv"]
Oct 30 13:32:55.956506 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:55.956465    4364 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10cce01a-1759-4617-bf74-5433cb240a30-tls-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:32:55.956506 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:55.956502    4364 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10cce01a-1759-4617-bf74-5433cb240a30-dshm\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:32:55.956506 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:55.956512    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccgvh\" (UniqueName: \"kubernetes.io/projected/10cce01a-1759-4617-bf74-5433cb240a30-kube-api-access-ccgvh\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:32:56.973154 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:32:56.973116    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10cce01a-1759-4617-bf74-5433cb240a30" path="/var/lib/kubelet/pods/10cce01a-1759-4617-bf74-5433cb240a30/volumes"
Oct 30 13:33:03.781583 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:03.781547    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-8d82d5/rate-test-model-8d82d5-kserve-66649ffff7-dndh9"]
Oct 30 13:33:03.782312 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:03.782292    4364 cpu_manager.go:414] "containerMap: removing container" podUID="10cce01a-1759-4617-bf74-5433cb240a30" containerName="main"
Oct 30 13:33:03.782312 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:03.782309    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10cce01a-1759-4617-bf74-5433cb240a30" containerName="main"
Oct 30 13:33:03.782475 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:03.782332    4364 cpu_manager.go:414] "containerMap: removing container" podUID="eb255481-3535-4bbe-a2c5-b4c1ba925fe4" containerName="main"
Oct 30 13:33:03.782475 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:03.782337    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb255481-3535-4bbe-a2c5-b4c1ba925fe4" containerName="main"
Oct 30 13:33:03.782475 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:03.782357    4364 cpu_manager.go:414] "containerMap: removing container" podUID="59c001b5-b0f7-4898-ba90-aa8a0b96a6e7" containerName="istio-proxy"
Oct 30 13:33:03.782475 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:03.782361    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59c001b5-b0f7-4898-ba90-aa8a0b96a6e7" containerName="istio-proxy"
Oct 30 13:33:03.782728 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:03.782495    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb255481-3535-4bbe-a2c5-b4c1ba925fe4" containerName="main"
Oct 30 13:33:03.782728 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:03.782506    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="59c001b5-b0f7-4898-ba90-aa8a0b96a6e7" containerName="istio-proxy"
Oct 30 13:33:03.782728 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:03.782518    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="10cce01a-1759-4617-bf74-5433cb240a30" containerName="main"
Oct 30 13:33:03.787988 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:03.787968    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-8d82d5/rate-test-model-8d82d5-kserve-66649ffff7-dndh9"
Oct 30 13:33:03.821492 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:03.821464    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-8d82d5\"/\"openshift-service-ca.crt\""
Oct 30 13:33:03.821492 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:03.821482    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-8d82d5\"/\"kube-root-ca.crt\""
Oct 30 13:33:03.833283 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:03.833257    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-8d82d5\"/\"rate-test-model-8d82d5-kserve-self-signed-certs\""
Oct 30 13:33:03.833398 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:03.833365    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-8d82d5\"/\"default-dockercfg-792k5\""
Oct 30 13:33:03.846562 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:03.846535    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-8d82d5/rate-test-model-8d82d5-kserve-66649ffff7-dndh9"]
Oct 30 13:33:03.896519 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:03.896482    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8781a4e-4171-4db3-808e-a8f412894661-dshm\") pod \"rate-test-model-8d82d5-kserve-66649ffff7-dndh9\" (UID: \"d8781a4e-4171-4db3-808e-a8f412894661\") " pod="ai-tenant-e2e-shared-b-w1-8d82d5/rate-test-model-8d82d5-kserve-66649ffff7-dndh9"
Oct 30 13:33:03.896736 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:03.896531    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8781a4e-4171-4db3-808e-a8f412894661-tmp-dir\") pod \"rate-test-model-8d82d5-kserve-66649ffff7-dndh9\" (UID: \"d8781a4e-4171-4db3-808e-a8f412894661\") " pod="ai-tenant-e2e-shared-b-w1-8d82d5/rate-test-model-8d82d5-kserve-66649ffff7-dndh9"
Oct 30 13:33:03.896838 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:03.896817    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8781a4e-4171-4db3-808e-a8f412894661-model-cache\") pod \"rate-test-model-8d82d5-kserve-66649ffff7-dndh9\" (UID: \"d8781a4e-4171-4db3-808e-a8f412894661\") " pod="ai-tenant-e2e-shared-b-w1-8d82d5/rate-test-model-8d82d5-kserve-66649ffff7-dndh9"
Oct 30 13:33:03.897008 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:03.896990    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8781a4e-4171-4db3-808e-a8f412894661-home\") pod \"rate-test-model-8d82d5-kserve-66649ffff7-dndh9\" (UID: \"d8781a4e-4171-4db3-808e-a8f412894661\") " pod="ai-tenant-e2e-shared-b-w1-8d82d5/rate-test-model-8d82d5-kserve-66649ffff7-dndh9"
Oct 30 13:33:03.897074 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:03.897039    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws6l2\" (UniqueName: \"kubernetes.io/projected/d8781a4e-4171-4db3-808e-a8f412894661-kube-api-access-ws6l2\") pod \"rate-test-model-8d82d5-kserve-66649ffff7-dndh9\" (UID: \"d8781a4e-4171-4db3-808e-a8f412894661\") " pod="ai-tenant-e2e-shared-b-w1-8d82d5/rate-test-model-8d82d5-kserve-66649ffff7-dndh9"
Oct 30 13:33:03.897185 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:03.897170    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8781a4e-4171-4db3-808e-a8f412894661-tls-certs\") pod \"rate-test-model-8d82d5-kserve-66649ffff7-dndh9\" (UID: \"d8781a4e-4171-4db3-808e-a8f412894661\") " pod="ai-tenant-e2e-shared-b-w1-8d82d5/rate-test-model-8d82d5-kserve-66649ffff7-dndh9"
Oct 30 13:33:04.032672 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:04.031802    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8781a4e-4171-4db3-808e-a8f412894661-model-cache\") pod \"rate-test-model-8d82d5-kserve-66649ffff7-dndh9\" (UID: \"d8781a4e-4171-4db3-808e-a8f412894661\") " pod="ai-tenant-e2e-shared-b-w1-8d82d5/rate-test-model-8d82d5-kserve-66649ffff7-dndh9"
Oct 30 13:33:04.032672 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:04.032009    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8781a4e-4171-4db3-808e-a8f412894661-home\") pod \"rate-test-model-8d82d5-kserve-66649ffff7-dndh9\" (UID: \"d8781a4e-4171-4db3-808e-a8f412894661\") " pod="ai-tenant-e2e-shared-b-w1-8d82d5/rate-test-model-8d82d5-kserve-66649ffff7-dndh9"
Oct 30 13:33:04.032672 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:04.032066    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws6l2\" (UniqueName: \"kubernetes.io/projected/d8781a4e-4171-4db3-808e-a8f412894661-kube-api-access-ws6l2\") pod \"rate-test-model-8d82d5-kserve-66649ffff7-dndh9\" (UID: \"d8781a4e-4171-4db3-808e-a8f412894661\") " pod="ai-tenant-e2e-shared-b-w1-8d82d5/rate-test-model-8d82d5-kserve-66649ffff7-dndh9"
Oct 30 13:33:04.032672 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:04.032172    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8781a4e-4171-4db3-808e-a8f412894661-tls-certs\") pod \"rate-test-model-8d82d5-kserve-66649ffff7-dndh9\" (UID: \"d8781a4e-4171-4db3-808e-a8f412894661\") " pod="ai-tenant-e2e-shared-b-w1-8d82d5/rate-test-model-8d82d5-kserve-66649ffff7-dndh9"
Oct 30 13:33:04.032672 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:04.032224    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8781a4e-4171-4db3-808e-a8f412894661-dshm\") pod \"rate-test-model-8d82d5-kserve-66649ffff7-dndh9\" (UID: \"d8781a4e-4171-4db3-808e-a8f412894661\") " pod="ai-tenant-e2e-shared-b-w1-8d82d5/rate-test-model-8d82d5-kserve-66649ffff7-dndh9"
Oct 30 13:33:04.032672 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:04.032263    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8781a4e-4171-4db3-808e-a8f412894661-tmp-dir\") pod \"rate-test-model-8d82d5-kserve-66649ffff7-dndh9\" (UID: \"d8781a4e-4171-4db3-808e-a8f412894661\") " pod="ai-tenant-e2e-shared-b-w1-8d82d5/rate-test-model-8d82d5-kserve-66649ffff7-dndh9"
Oct 30 13:33:04.033170 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:04.032835    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8781a4e-4171-4db3-808e-a8f412894661-tmp-dir\") pod \"rate-test-model-8d82d5-kserve-66649ffff7-dndh9\" (UID: \"d8781a4e-4171-4db3-808e-a8f412894661\") " pod="ai-tenant-e2e-shared-b-w1-8d82d5/rate-test-model-8d82d5-kserve-66649ffff7-dndh9"
Oct 30 13:33:04.033170 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:04.033097    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8781a4e-4171-4db3-808e-a8f412894661-model-cache\") pod \"rate-test-model-8d82d5-kserve-66649ffff7-dndh9\" (UID: \"d8781a4e-4171-4db3-808e-a8f412894661\") " pod="ai-tenant-e2e-shared-b-w1-8d82d5/rate-test-model-8d82d5-kserve-66649ffff7-dndh9"
Oct 30 13:33:04.040660 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:04.033341    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8781a4e-4171-4db3-808e-a8f412894661-home\") pod \"rate-test-model-8d82d5-kserve-66649ffff7-dndh9\" (UID: \"d8781a4e-4171-4db3-808e-a8f412894661\") " pod="ai-tenant-e2e-shared-b-w1-8d82d5/rate-test-model-8d82d5-kserve-66649ffff7-dndh9"
Oct 30 13:33:04.060630 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:04.052234    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8781a4e-4171-4db3-808e-a8f412894661-tls-certs\") pod \"rate-test-model-8d82d5-kserve-66649ffff7-dndh9\" (UID: \"d8781a4e-4171-4db3-808e-a8f412894661\") " pod="ai-tenant-e2e-shared-b-w1-8d82d5/rate-test-model-8d82d5-kserve-66649ffff7-dndh9"
Oct 30 13:33:04.074624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:04.064916    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8781a4e-4171-4db3-808e-a8f412894661-dshm\") pod \"rate-test-model-8d82d5-kserve-66649ffff7-dndh9\" (UID: \"d8781a4e-4171-4db3-808e-a8f412894661\") " pod="ai-tenant-e2e-shared-b-w1-8d82d5/rate-test-model-8d82d5-kserve-66649ffff7-dndh9"
Oct 30 13:33:04.083622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:04.080859    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws6l2\" (UniqueName: \"kubernetes.io/projected/d8781a4e-4171-4db3-808e-a8f412894661-kube-api-access-ws6l2\") pod \"rate-test-model-8d82d5-kserve-66649ffff7-dndh9\" (UID: \"d8781a4e-4171-4db3-808e-a8f412894661\") " pod="ai-tenant-e2e-shared-b-w1-8d82d5/rate-test-model-8d82d5-kserve-66649ffff7-dndh9"
Oct 30 13:33:04.107998 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:04.107958    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-8d82d5/rate-test-model-8d82d5-kserve-66649ffff7-dndh9"
Oct 30 13:33:04.463493 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:04.463468    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-8d82d5/rate-test-model-8d82d5-kserve-66649ffff7-dndh9"]
Oct 30 13:33:04.465432 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:33:04.465405    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd8781a4e_4171_4db3_808e_a8f412894661.slice/crio-f35a970d39d82bc67cc54d320d589dd2f852c127983d82580ad636670b368ebe WatchSource:0}: Error finding container f35a970d39d82bc67cc54d320d589dd2f852c127983d82580ad636670b368ebe: Status 404 returned error can't find the container with id f35a970d39d82bc67cc54d320d589dd2f852c127983d82580ad636670b368ebe
Oct 30 13:33:04.972165 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:04.972130    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-8d82d5/rate-test-model-8d82d5-kserve-66649ffff7-dndh9" event={"ID":"d8781a4e-4171-4db3-808e-a8f412894661","Type":"ContainerStarted","Data":"fa9376c2c207f9d058327167d9032e0fbc89d3cedc2bba9d3f938697976d6fd4"}
Oct 30 13:33:04.972567 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:04.972176    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-8d82d5/rate-test-model-8d82d5-kserve-66649ffff7-dndh9" event={"ID":"d8781a4e-4171-4db3-808e-a8f412894661","Type":"ContainerStarted","Data":"f35a970d39d82bc67cc54d320d589dd2f852c127983d82580ad636670b368ebe"}
Oct 30 13:33:05.011433 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:05.011380    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-8d82d5/rate-test-model-8d82d5-kserve-66649ffff7-dndh9" podStartSLOduration=2.01136503 podStartE2EDuration="2.01136503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:33:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:33:05.0103373 +0000 UTC m=+2484.176701948" watchObservedRunningTime="2026-10-30 13:33:05.01136503 +0000 UTC m=+2484.177729683"
Oct 30 13:33:05.108884 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:05.108853    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-8d82d5/rate-test-model-8d82d5-kserve-66649ffff7-dndh9"
Oct 30 13:33:08.375587 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:08.375550    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w5-82796d/auth-test-model-82796d-kserve-fd6dd7b57-fc2sg"]
Oct 30 13:33:08.380488 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:08.380461    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-82796d/auth-test-model-82796d-kserve-fd6dd7b57-fc2sg"
Oct 30 13:33:08.408441 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:08.407917    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-82796d/auth-test-model-82796d-kserve-fd6dd7b57-fc2sg"]
Oct 30 13:33:08.414246 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:08.414205    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-82796d\"/\"default-dockercfg-7gqg8\""
Oct 30 13:33:08.414421 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:08.414335    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-82796d\"/\"kube-root-ca.crt\""
Oct 30 13:33:08.414421 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:08.414394    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-82796d\"/\"openshift-service-ca.crt\""
Oct 30 13:33:08.414551 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:08.414487    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-82796d\"/\"auth-test-model-82796d-kserve-self-signed-certs\""
Oct 30 13:33:08.431470 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:08.431429    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56e242de-da4d-4e53-b4b3-0e4ecd40ef8a-tls-certs\") pod \"auth-test-model-82796d-kserve-fd6dd7b57-fc2sg\" (UID: \"56e242de-da4d-4e53-b4b3-0e4ecd40ef8a\") " pod="ai-tenant-e2e-shared-b-w5-82796d/auth-test-model-82796d-kserve-fd6dd7b57-fc2sg"
Oct 30 13:33:08.431711 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:08.431486    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56e242de-da4d-4e53-b4b3-0e4ecd40ef8a-dshm\") pod \"auth-test-model-82796d-kserve-fd6dd7b57-fc2sg\" (UID: \"56e242de-da4d-4e53-b4b3-0e4ecd40ef8a\") " pod="ai-tenant-e2e-shared-b-w5-82796d/auth-test-model-82796d-kserve-fd6dd7b57-fc2sg"
Oct 30 13:33:08.431711 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:08.431519    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56e242de-da4d-4e53-b4b3-0e4ecd40ef8a-home\") pod \"auth-test-model-82796d-kserve-fd6dd7b57-fc2sg\" (UID: \"56e242de-da4d-4e53-b4b3-0e4ecd40ef8a\") " pod="ai-tenant-e2e-shared-b-w5-82796d/auth-test-model-82796d-kserve-fd6dd7b57-fc2sg"
Oct 30 13:33:08.431845 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:08.431721    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56e242de-da4d-4e53-b4b3-0e4ecd40ef8a-model-cache\") pod \"auth-test-model-82796d-kserve-fd6dd7b57-fc2sg\" (UID: \"56e242de-da4d-4e53-b4b3-0e4ecd40ef8a\") " pod="ai-tenant-e2e-shared-b-w5-82796d/auth-test-model-82796d-kserve-fd6dd7b57-fc2sg"
Oct 30 13:33:08.431845 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:08.431802    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56e242de-da4d-4e53-b4b3-0e4ecd40ef8a-tmp-dir\") pod \"auth-test-model-82796d-kserve-fd6dd7b57-fc2sg\" (UID: \"56e242de-da4d-4e53-b4b3-0e4ecd40ef8a\") " pod="ai-tenant-e2e-shared-b-w5-82796d/auth-test-model-82796d-kserve-fd6dd7b57-fc2sg"
Oct 30 13:33:08.431954 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:08.431843    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44fvq\" (UniqueName: \"kubernetes.io/projected/56e242de-da4d-4e53-b4b3-0e4ecd40ef8a-kube-api-access-44fvq\") pod \"auth-test-model-82796d-kserve-fd6dd7b57-fc2sg\" (UID: \"56e242de-da4d-4e53-b4b3-0e4ecd40ef8a\") " pod="ai-tenant-e2e-shared-b-w5-82796d/auth-test-model-82796d-kserve-fd6dd7b57-fc2sg"
Oct 30 13:33:08.533754 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:08.533711    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44fvq\" (UniqueName: \"kubernetes.io/projected/56e242de-da4d-4e53-b4b3-0e4ecd40ef8a-kube-api-access-44fvq\") pod \"auth-test-model-82796d-kserve-fd6dd7b57-fc2sg\" (UID: \"56e242de-da4d-4e53-b4b3-0e4ecd40ef8a\") " pod="ai-tenant-e2e-shared-b-w5-82796d/auth-test-model-82796d-kserve-fd6dd7b57-fc2sg"
Oct 30 13:33:08.533977 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:08.533958    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56e242de-da4d-4e53-b4b3-0e4ecd40ef8a-tls-certs\") pod \"auth-test-model-82796d-kserve-fd6dd7b57-fc2sg\" (UID: \"56e242de-da4d-4e53-b4b3-0e4ecd40ef8a\") " pod="ai-tenant-e2e-shared-b-w5-82796d/auth-test-model-82796d-kserve-fd6dd7b57-fc2sg"
Oct 30 13:33:08.534055 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:08.534005    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56e242de-da4d-4e53-b4b3-0e4ecd40ef8a-dshm\") pod \"auth-test-model-82796d-kserve-fd6dd7b57-fc2sg\" (UID: \"56e242de-da4d-4e53-b4b3-0e4ecd40ef8a\") " pod="ai-tenant-e2e-shared-b-w5-82796d/auth-test-model-82796d-kserve-fd6dd7b57-fc2sg"
Oct 30 13:33:08.534055 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:08.534045    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56e242de-da4d-4e53-b4b3-0e4ecd40ef8a-home\") pod \"auth-test-model-82796d-kserve-fd6dd7b57-fc2sg\" (UID: \"56e242de-da4d-4e53-b4b3-0e4ecd40ef8a\") " pod="ai-tenant-e2e-shared-b-w5-82796d/auth-test-model-82796d-kserve-fd6dd7b57-fc2sg"
Oct 30 13:33:08.534174 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:08.534148    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56e242de-da4d-4e53-b4b3-0e4ecd40ef8a-model-cache\") pod \"auth-test-model-82796d-kserve-fd6dd7b57-fc2sg\" (UID: \"56e242de-da4d-4e53-b4b3-0e4ecd40ef8a\") " pod="ai-tenant-e2e-shared-b-w5-82796d/auth-test-model-82796d-kserve-fd6dd7b57-fc2sg"
Oct 30 13:33:08.534271 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:08.534256    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56e242de-da4d-4e53-b4b3-0e4ecd40ef8a-tmp-dir\") pod \"auth-test-model-82796d-kserve-fd6dd7b57-fc2sg\" (UID: \"56e242de-da4d-4e53-b4b3-0e4ecd40ef8a\") " pod="ai-tenant-e2e-shared-b-w5-82796d/auth-test-model-82796d-kserve-fd6dd7b57-fc2sg"
Oct 30 13:33:08.534750 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:08.534719    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56e242de-da4d-4e53-b4b3-0e4ecd40ef8a-tmp-dir\") pod \"auth-test-model-82796d-kserve-fd6dd7b57-fc2sg\" (UID: \"56e242de-da4d-4e53-b4b3-0e4ecd40ef8a\") " pod="ai-tenant-e2e-shared-b-w5-82796d/auth-test-model-82796d-kserve-fd6dd7b57-fc2sg"
Oct 30 13:33:08.535000 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:08.534980    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56e242de-da4d-4e53-b4b3-0e4ecd40ef8a-home\") pod \"auth-test-model-82796d-kserve-fd6dd7b57-fc2sg\" (UID: \"56e242de-da4d-4e53-b4b3-0e4ecd40ef8a\") " pod="ai-tenant-e2e-shared-b-w5-82796d/auth-test-model-82796d-kserve-fd6dd7b57-fc2sg"
Oct 30 13:33:08.535261 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:08.535236    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56e242de-da4d-4e53-b4b3-0e4ecd40ef8a-model-cache\") pod \"auth-test-model-82796d-kserve-fd6dd7b57-fc2sg\" (UID: \"56e242de-da4d-4e53-b4b3-0e4ecd40ef8a\") " pod="ai-tenant-e2e-shared-b-w5-82796d/auth-test-model-82796d-kserve-fd6dd7b57-fc2sg"
Oct 30 13:33:08.536413 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:08.536391    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56e242de-da4d-4e53-b4b3-0e4ecd40ef8a-dshm\") pod \"auth-test-model-82796d-kserve-fd6dd7b57-fc2sg\" (UID: \"56e242de-da4d-4e53-b4b3-0e4ecd40ef8a\") " pod="ai-tenant-e2e-shared-b-w5-82796d/auth-test-model-82796d-kserve-fd6dd7b57-fc2sg"
Oct 30 13:33:08.536977 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:08.536956    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56e242de-da4d-4e53-b4b3-0e4ecd40ef8a-tls-certs\") pod \"auth-test-model-82796d-kserve-fd6dd7b57-fc2sg\" (UID: \"56e242de-da4d-4e53-b4b3-0e4ecd40ef8a\") " pod="ai-tenant-e2e-shared-b-w5-82796d/auth-test-model-82796d-kserve-fd6dd7b57-fc2sg"
Oct 30 13:33:08.561331 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:08.561298    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44fvq\" (UniqueName: \"kubernetes.io/projected/56e242de-da4d-4e53-b4b3-0e4ecd40ef8a-kube-api-access-44fvq\") pod \"auth-test-model-82796d-kserve-fd6dd7b57-fc2sg\" (UID: \"56e242de-da4d-4e53-b4b3-0e4ecd40ef8a\") " pod="ai-tenant-e2e-shared-b-w5-82796d/auth-test-model-82796d-kserve-fd6dd7b57-fc2sg"
Oct 30 13:33:08.703126 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:08.703031    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-82796d/auth-test-model-82796d-kserve-fd6dd7b57-fc2sg"
Oct 30 13:33:09.324680 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:09.308483    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-82796d/auth-test-model-82796d-kserve-fd6dd7b57-fc2sg"]
Oct 30 13:33:10.073796 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:10.071221    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d25db49-xrw2t"]
Oct 30 13:33:10.098701 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:10.098662    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d25db49-xrw2t"]
Oct 30 13:33:10.103767 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:10.103731    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-82796d/auth-test-model-82796d-kserve-fd6dd7b57-fc2sg" event={"ID":"56e242de-da4d-4e53-b4b3-0e4ecd40ef8a","Type":"ContainerStarted","Data":"39cbb0c9f1231b21a1ce74e65cf4be03949fa99b9e166da0becd7b85d8168017"}
Oct 30 13:33:10.103999 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:10.103979    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-82796d/auth-test-model-82796d-kserve-fd6dd7b57-fc2sg" event={"ID":"56e242de-da4d-4e53-b4b3-0e4ecd40ef8a","Type":"ContainerStarted","Data":"cb681c0c75c29bd708d3897a265aeeafe72daa5d36dbb2e054dc215142da3d67"}
Oct 30 13:33:10.705633 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:10.703713    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w5-82796d/auth-test-model-82796d-kserve-fd6dd7b57-fc2sg"
Oct 30 13:33:10.973169 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:10.973095    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4dda6f80-8485-49d9-99cf-82a602c79918" path="/var/lib/kubelet/pods/4dda6f80-8485-49d9-99cf-82a602c79918/volumes"
Oct 30 13:33:11.527411 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:33:11.527376    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:33:14.109219 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:14.109183    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-8d82d5/rate-test-model-8d82d5-kserve-66649ffff7-dndh9"
Oct 30 13:33:14.122422 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:14.122396    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-8d82d5/rate-test-model-8d82d5-kserve-66649ffff7-dndh9"
Oct 30 13:33:14.152545 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:14.152511    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-8d82d5/rate-test-model-8d82d5-kserve-66649ffff7-dndh9"
Oct 30 13:33:14.166891 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:14.166844    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w5-82796d/auth-test-model-82796d-kserve-fd6dd7b57-fc2sg" podStartSLOduration=6.166829743 podStartE2EDuration="6.166829743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:33:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:33:10.174024824 +0000 UTC m=+2489.340389474" watchObservedRunningTime="2026-10-30 13:33:14.166829743 +0000 UTC m=+2493.333194432"
Oct 30 13:33:18.703763 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:18.703726    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w5-82796d/auth-test-model-82796d-kserve-fd6dd7b57-fc2sg"
Oct 30 13:33:18.722474 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:18.722443    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w5-82796d/auth-test-model-82796d-kserve-fd6dd7b57-fc2sg"
Oct 30 13:33:19.198469 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:19.198437    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w5-82796d/auth-test-model-82796d-kserve-fd6dd7b57-fc2sg"
Oct 30 13:33:20.210369 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:20.210328    4364 generic.go:356] "Generic (PLEG): container finished" podID="5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b" containerID="15629b602c0d92a0beebeec30ea9d71e7a69fceabc26bf40d512995430aadba2" exitCode=0
Oct 30 13:33:20.210833 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:20.210394    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk" event={"ID":"5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b","Type":"ContainerDied","Data":"15629b602c0d92a0beebeec30ea9d71e7a69fceabc26bf40d512995430aadba2"}
Oct 30 13:33:20.210977 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:20.210960    4364 scope.go:122] "RemoveContainer" containerID="15629b602c0d92a0beebeec30ea9d71e7a69fceabc26bf40d512995430aadba2"
Oct 30 13:33:20.212803 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:20.212777    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:33:20.212877 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:20.212842    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:33:20.212915 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:20.212875    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:33:21.221093 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:21.221058    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk" event={"ID":"5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b","Type":"ContainerStarted","Data":"fed1ac4064786654b61c4c54e7b0f18094223daf7de251f5114b989759c18992"}
Oct 30 13:33:21.973104 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:21.973070    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk"
Oct 30 13:33:21.974447 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:21.974417    4364 patch_prober.go:29] interesting pod/e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Oct 30 13:33:21.974583 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:21.974466    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk" podUID="5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Oct 30 13:33:22.774687 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:22.774658    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk"
Oct 30 13:33:22.972305 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:22.972268    4364 patch_prober.go:29] interesting pod/e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Oct 30 13:33:22.972486 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:22.972317    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk" podUID="5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Oct 30 13:33:23.973311 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:23.973271    4364 patch_prober.go:29] interesting pod/e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Oct 30 13:33:23.973743 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:23.973325    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk" podUID="5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Oct 30 13:33:24.972430 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:24.972390    4364 patch_prober.go:29] interesting pod/e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Oct 30 13:33:24.972657 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:24.972446    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk" podUID="5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Oct 30 13:33:25.973094 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:25.973055    4364 patch_prober.go:29] interesting pod/e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Oct 30 13:33:25.973514 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:25.973111    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk" podUID="5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Oct 30 13:33:26.977316 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:26.977286    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk"
Oct 30 13:33:26.978355 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:26.978331    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk"
Oct 30 13:33:27.221684 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:27.221653    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-082748-8fcd5c46-zpgwg"]
Oct 30 13:33:27.225245 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:27.225219    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-082748-8fcd5c46-zpgwg"
Oct 30 13:33:27.245484 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:27.245453    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-082748-serving-cert\""
Oct 30 13:33:27.249654 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:27.249625    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-082748-8fcd5c46-zpgwg"]
Oct 30 13:33:27.356267 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:27.356233    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f77f214e-e14c-46b1-b1a7-68c693f3a055-metrics-certs\") pod \"maas-api-e2e-ipp-a-082748-8fcd5c46-zpgwg\" (UID: \"f77f214e-e14c-46b1-b1a7-68c693f3a055\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-082748-8fcd5c46-zpgwg"
Oct 30 13:33:27.356442 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:27.356419    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f77f214e-e14c-46b1-b1a7-68c693f3a055-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-082748-8fcd5c46-zpgwg\" (UID: \"f77f214e-e14c-46b1-b1a7-68c693f3a055\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-082748-8fcd5c46-zpgwg"
Oct 30 13:33:27.356501 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:27.356488    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvvld\" (UniqueName: \"kubernetes.io/projected/f77f214e-e14c-46b1-b1a7-68c693f3a055-kube-api-access-bvvld\") pod \"maas-api-e2e-ipp-a-082748-8fcd5c46-zpgwg\" (UID: \"f77f214e-e14c-46b1-b1a7-68c693f3a055\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-082748-8fcd5c46-zpgwg"
Oct 30 13:33:27.356779 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:27.356760    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f77f214e-e14c-46b1-b1a7-68c693f3a055-maas-api-tls\") pod \"maas-api-e2e-ipp-a-082748-8fcd5c46-zpgwg\" (UID: \"f77f214e-e14c-46b1-b1a7-68c693f3a055\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-082748-8fcd5c46-zpgwg"
Oct 30 13:33:27.458408 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:27.458372    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f77f214e-e14c-46b1-b1a7-68c693f3a055-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-082748-8fcd5c46-zpgwg\" (UID: \"f77f214e-e14c-46b1-b1a7-68c693f3a055\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-082748-8fcd5c46-zpgwg"
Oct 30 13:33:27.458635 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:27.458432    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvvld\" (UniqueName: \"kubernetes.io/projected/f77f214e-e14c-46b1-b1a7-68c693f3a055-kube-api-access-bvvld\") pod \"maas-api-e2e-ipp-a-082748-8fcd5c46-zpgwg\" (UID: \"f77f214e-e14c-46b1-b1a7-68c693f3a055\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-082748-8fcd5c46-zpgwg"
Oct 30 13:33:27.458635 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:27.458567    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f77f214e-e14c-46b1-b1a7-68c693f3a055-maas-api-tls\") pod \"maas-api-e2e-ipp-a-082748-8fcd5c46-zpgwg\" (UID: \"f77f214e-e14c-46b1-b1a7-68c693f3a055\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-082748-8fcd5c46-zpgwg"
Oct 30 13:33:27.458785 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:27.458719    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f77f214e-e14c-46b1-b1a7-68c693f3a055-metrics-certs\") pod \"maas-api-e2e-ipp-a-082748-8fcd5c46-zpgwg\" (UID: \"f77f214e-e14c-46b1-b1a7-68c693f3a055\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-082748-8fcd5c46-zpgwg"
Oct 30 13:33:27.459177 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:27.459155    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f77f214e-e14c-46b1-b1a7-68c693f3a055-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-082748-8fcd5c46-zpgwg\" (UID: \"f77f214e-e14c-46b1-b1a7-68c693f3a055\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-082748-8fcd5c46-zpgwg"
Oct 30 13:33:27.461209 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:27.461189    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f77f214e-e14c-46b1-b1a7-68c693f3a055-metrics-certs\") pod \"maas-api-e2e-ipp-a-082748-8fcd5c46-zpgwg\" (UID: \"f77f214e-e14c-46b1-b1a7-68c693f3a055\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-082748-8fcd5c46-zpgwg"
Oct 30 13:33:27.461320 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:27.461260    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f77f214e-e14c-46b1-b1a7-68c693f3a055-maas-api-tls\") pod \"maas-api-e2e-ipp-a-082748-8fcd5c46-zpgwg\" (UID: \"f77f214e-e14c-46b1-b1a7-68c693f3a055\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-082748-8fcd5c46-zpgwg"
Oct 30 13:33:27.489341 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:27.489269    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvvld\" (UniqueName: \"kubernetes.io/projected/f77f214e-e14c-46b1-b1a7-68c693f3a055-kube-api-access-bvvld\") pod \"maas-api-e2e-ipp-a-082748-8fcd5c46-zpgwg\" (UID: \"f77f214e-e14c-46b1-b1a7-68c693f3a055\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-082748-8fcd5c46-zpgwg"
Oct 30 13:33:27.543063 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:27.543025    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-082748-8fcd5c46-zpgwg"
Oct 30 13:33:27.930162 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:27.930132    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-082748-8fcd5c46-zpgwg"]
Oct 30 13:33:27.931484 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:33:27.931461    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf77f214e_e14c_46b1_b1a7_68c693f3a055.slice/crio-18d37eb8b10afee5a352d965171bb67ded3441e7f048c5b8d6e69e7befa265a9 WatchSource:0}: Error finding container 18d37eb8b10afee5a352d965171bb67ded3441e7f048c5b8d6e69e7befa265a9: Status 404 returned error can't find the container with id 18d37eb8b10afee5a352d965171bb67ded3441e7f048c5b8d6e69e7befa265a9
Oct 30 13:33:27.932948 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:27.932929    4364 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 13:33:28.291593 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:28.291514    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-082748-8fcd5c46-zpgwg" event={"ID":"f77f214e-e14c-46b1-b1a7-68c693f3a055","Type":"ContainerStarted","Data":"18d37eb8b10afee5a352d965171bb67ded3441e7f048c5b8d6e69e7befa265a9"}
Oct 30 13:33:29.353638 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:29.353564    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-082748-8fcd5c46-zpgwg" event={"ID":"f77f214e-e14c-46b1-b1a7-68c693f3a055","Type":"ContainerStarted","Data":"3884b0921b079c7497005e7a27735da7ebad64e586106c2f99022ed3a50d6875"}
Oct 30 13:33:29.354162 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:29.354137    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-082748-8fcd5c46-zpgwg"
Oct 30 13:33:29.397935 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:29.397878    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-082748-8fcd5c46-zpgwg" podStartSLOduration=2.030420678 podStartE2EDuration="2.397860083s" totalImagesPullingTime="367.439405ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:33:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:33:29.395474485 +0000 UTC m=+2508.561839134" watchObservedRunningTime="2026-10-30 13:33:29.397860083 +0000 UTC m=+2508.564224732"
Oct 30 13:33:34.492504 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:34.492467    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg"]
Oct 30 13:33:34.496295 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:34.496273    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg"
Oct 30 13:33:34.529122 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:34.529094    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-9cbcbc-openshift-default-dockercfg-hkxps\""
Oct 30 13:33:34.534813 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:34.534785    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg"]
Oct 30 13:33:34.594115 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:34.594086    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c71d1db6-aeed-434f-9829-a6589831daac-istio-envoy\") pod \"e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg\" (UID: \"c71d1db6-aeed-434f-9829-a6589831daac\") " pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg"
Oct 30 13:33:34.594290 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:34.594224    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c71d1db6-aeed-434f-9829-a6589831daac-istio-podinfo\") pod \"e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg\" (UID: \"c71d1db6-aeed-434f-9829-a6589831daac\") " pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg"
Oct 30 13:33:34.594290 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:34.594281    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svr28\" (UniqueName: \"kubernetes.io/projected/c71d1db6-aeed-434f-9829-a6589831daac-kube-api-access-svr28\") pod \"e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg\" (UID: \"c71d1db6-aeed-434f-9829-a6589831daac\") " pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg"
Oct 30 13:33:34.594408 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:34.594310    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c71d1db6-aeed-434f-9829-a6589831daac-workload-socket\") pod \"e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg\" (UID: \"c71d1db6-aeed-434f-9829-a6589831daac\") " pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg"
Oct 30 13:33:34.594460 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:34.594447    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c71d1db6-aeed-434f-9829-a6589831daac-workload-certs\") pod \"e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg\" (UID: \"c71d1db6-aeed-434f-9829-a6589831daac\") " pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg"
Oct 30 13:33:34.594499 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:34.594484    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c71d1db6-aeed-434f-9829-a6589831daac-credential-socket\") pod \"e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg\" (UID: \"c71d1db6-aeed-434f-9829-a6589831daac\") " pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg"
Oct 30 13:33:34.594645 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:34.594593    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c71d1db6-aeed-434f-9829-a6589831daac-istio-token\") pod \"e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg\" (UID: \"c71d1db6-aeed-434f-9829-a6589831daac\") " pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg"
Oct 30 13:33:34.594725 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:34.594702    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c71d1db6-aeed-434f-9829-a6589831daac-istio-data\") pod \"e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg\" (UID: \"c71d1db6-aeed-434f-9829-a6589831daac\") " pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg"
Oct 30 13:33:34.594781 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:34.594745    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c71d1db6-aeed-434f-9829-a6589831daac-istiod-ca-cert\") pod \"e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg\" (UID: \"c71d1db6-aeed-434f-9829-a6589831daac\") " pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg"
Oct 30 13:33:34.695989 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:34.695952    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c71d1db6-aeed-434f-9829-a6589831daac-istio-podinfo\") pod \"e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg\" (UID: \"c71d1db6-aeed-434f-9829-a6589831daac\") " pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg"
Oct 30 13:33:34.696185 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:34.696013    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svr28\" (UniqueName: \"kubernetes.io/projected/c71d1db6-aeed-434f-9829-a6589831daac-kube-api-access-svr28\") pod \"e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg\" (UID: \"c71d1db6-aeed-434f-9829-a6589831daac\") " pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg"
Oct 30 13:33:34.696185 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:34.696121    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c71d1db6-aeed-434f-9829-a6589831daac-workload-socket\") pod \"e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg\" (UID: \"c71d1db6-aeed-434f-9829-a6589831daac\") " pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg"
Oct 30 13:33:34.696316 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:34.696199    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c71d1db6-aeed-434f-9829-a6589831daac-workload-certs\") pod \"e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg\" (UID: \"c71d1db6-aeed-434f-9829-a6589831daac\") " pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg"
Oct 30 13:33:34.696316 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:34.696236    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c71d1db6-aeed-434f-9829-a6589831daac-credential-socket\") pod \"e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg\" (UID: \"c71d1db6-aeed-434f-9829-a6589831daac\") " pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg"
Oct 30 13:33:34.696431 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:34.696316    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c71d1db6-aeed-434f-9829-a6589831daac-istio-token\") pod \"e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg\" (UID: \"c71d1db6-aeed-434f-9829-a6589831daac\") " pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg"
Oct 30 13:33:34.696431 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:34.696355    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c71d1db6-aeed-434f-9829-a6589831daac-istio-data\") pod \"e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg\" (UID: \"c71d1db6-aeed-434f-9829-a6589831daac\") " pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg"
Oct 30 13:33:34.696431 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:34.696378    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c71d1db6-aeed-434f-9829-a6589831daac-istiod-ca-cert\") pod \"e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg\" (UID: \"c71d1db6-aeed-434f-9829-a6589831daac\") " pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg"
Oct 30 13:33:34.696638 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:34.696456    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c71d1db6-aeed-434f-9829-a6589831daac-istio-envoy\") pod \"e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg\" (UID: \"c71d1db6-aeed-434f-9829-a6589831daac\") " pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg"
Oct 30 13:33:34.696638 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:34.696544    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c71d1db6-aeed-434f-9829-a6589831daac-workload-socket\") pod \"e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg\" (UID: \"c71d1db6-aeed-434f-9829-a6589831daac\") " pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg"
Oct 30 13:33:34.696638 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:34.696626    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c71d1db6-aeed-434f-9829-a6589831daac-workload-certs\") pod \"e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg\" (UID: \"c71d1db6-aeed-434f-9829-a6589831daac\") " pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg"
Oct 30 13:33:34.696811 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:34.696676    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c71d1db6-aeed-434f-9829-a6589831daac-credential-socket\") pod \"e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg\" (UID: \"c71d1db6-aeed-434f-9829-a6589831daac\") " pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg"
Oct 30 13:33:34.696811 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:34.696778    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c71d1db6-aeed-434f-9829-a6589831daac-istio-data\") pod \"e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg\" (UID: \"c71d1db6-aeed-434f-9829-a6589831daac\") " pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg"
Oct 30 13:33:34.697192 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:34.697173    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c71d1db6-aeed-434f-9829-a6589831daac-istiod-ca-cert\") pod \"e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg\" (UID: \"c71d1db6-aeed-434f-9829-a6589831daac\") " pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg"
Oct 30 13:33:34.698587 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:34.698566    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c71d1db6-aeed-434f-9829-a6589831daac-istio-envoy\") pod \"e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg\" (UID: \"c71d1db6-aeed-434f-9829-a6589831daac\") " pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg"
Oct 30 13:33:34.698715 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:34.698694    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c71d1db6-aeed-434f-9829-a6589831daac-istio-podinfo\") pod \"e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg\" (UID: \"c71d1db6-aeed-434f-9829-a6589831daac\") " pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg"
Oct 30 13:33:34.719568 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:34.719535    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svr28\" (UniqueName: \"kubernetes.io/projected/c71d1db6-aeed-434f-9829-a6589831daac-kube-api-access-svr28\") pod \"e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg\" (UID: \"c71d1db6-aeed-434f-9829-a6589831daac\") " pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg"
Oct 30 13:33:34.719726 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:34.719703    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c71d1db6-aeed-434f-9829-a6589831daac-istio-token\") pod \"e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg\" (UID: \"c71d1db6-aeed-434f-9829-a6589831daac\") " pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg"
Oct 30 13:33:34.811344 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:34.811275    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg"
Oct 30 13:33:35.168296 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:35.168270    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg"]
Oct 30 13:33:35.169947 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:33:35.169915    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc71d1db6_aeed_434f_9829_a6589831daac.slice/crio-30d94a83baefa04b256170c0adbe2608882d7ce5e6fade46735b4fc6bc5040d1 WatchSource:0}: Error finding container 30d94a83baefa04b256170c0adbe2608882d7ce5e6fade46735b4fc6bc5040d1: Status 404 returned error can't find the container with id 30d94a83baefa04b256170c0adbe2608882d7ce5e6fade46735b4fc6bc5040d1
Oct 30 13:33:35.173021 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:35.172989    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:33:35.173107 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:35.173057    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:33:35.173107 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:35.173086    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:33:35.367219 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:35.367189    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-082748-8fcd5c46-zpgwg"
Oct 30 13:33:35.419016 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:35.418928    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg" event={"ID":"c71d1db6-aeed-434f-9829-a6589831daac","Type":"ContainerStarted","Data":"09871ad2dad15e8ddfae62ad1934266e457fdabcf163535980f6fd21955c6d0c"}
Oct 30 13:33:35.419016 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:35.418980    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg" event={"ID":"c71d1db6-aeed-434f-9829-a6589831daac","Type":"ContainerStarted","Data":"30d94a83baefa04b256170c0adbe2608882d7ce5e6fade46735b4fc6bc5040d1"}
Oct 30 13:33:35.541167 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:35.541108    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg" podStartSLOduration=1.5410919889999999 podStartE2EDuration="1.541091989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:33:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:33:35.536148666 +0000 UTC m=+2514.702513327" watchObservedRunningTime="2026-10-30 13:33:35.541091989 +0000 UTC m=+2514.707456636"
Oct 30 13:33:35.811726 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:35.811645    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg"
Oct 30 13:33:36.815239 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:36.815201    4364 patch_prober.go:29] interesting pod/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Oct 30 13:33:36.815696 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:36.815264    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg" podUID="c71d1db6-aeed-434f-9829-a6589831daac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Oct 30 13:33:37.812344 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:37.812310    4364 patch_prober.go:29] interesting pod/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Oct 30 13:33:37.812505 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:37.812365    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg" podUID="c71d1db6-aeed-434f-9829-a6589831daac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Oct 30 13:33:38.104681 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:38.104571    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:33:38.817619 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:38.815007    4364 patch_prober.go:29] interesting pod/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Oct 30 13:33:38.817619 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:38.815072    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg" podUID="c71d1db6-aeed-434f-9829-a6589831daac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Oct 30 13:33:39.812308 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:39.812263    4364 patch_prober.go:29] interesting pod/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Oct 30 13:33:39.812818 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:39.812331    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg" podUID="c71d1db6-aeed-434f-9829-a6589831daac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Oct 30 13:33:40.188750 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:40.188699    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w6-cda0d9/rate-test-model-cda0d9-kserve-9d99f655c-swqhw" podUID="d21066d6-fdb1-4c53-8e3b-b8b254112229" containerName="main" probeResult="failure" output="Get \"https://10.133.0.225:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:33:40.236622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:40.234704    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w5-82796d/auth-test-model-82796d-kserve-fd6dd7b57-fc2sg" podUID="56e242de-da4d-4e53-b4b3-0e4ecd40ef8a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.229:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:33:40.245621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:40.244894    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74" podUID="4f2b06ac-9bf8-4b57-bd22-85532576f9c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded"
Oct 30 13:33:40.302623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:40.301659    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz" podUID="a568f1ce-602d-4069-a41c-9fe1a7635c5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 30 13:33:40.359269 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:40.359225    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4" podUID="0cbe6af7-d884-4613-90b8-192e5ffdb027" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:33:40.812622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:40.811837    4364 patch_prober.go:29] interesting pod/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Oct 30 13:33:40.812622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:40.811893    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg" podUID="c71d1db6-aeed-434f-9829-a6589831daac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Oct 30 13:33:41.702318 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:33:41.702275    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:33:41.812090 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:41.812055    4364 patch_prober.go:29] interesting pod/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Oct 30 13:33:41.812266 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:41.812116    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg" podUID="c71d1db6-aeed-434f-9829-a6589831daac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Oct 30 13:33:42.883819 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:42.883781    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg"
Oct 30 13:33:42.884229 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:42.884176    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg"
Oct 30 13:33:42.884872 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:33:42.884853    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg"
Oct 30 13:34:11.506938 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:34:11.506898    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:34:38.570061 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:38.570028    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4b0ea9/auth-test-model-4b0ea9-kserve-86458657c-m5qrl"]
Oct 30 13:34:39.048484 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:39.048445    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-cda0d9/rate-test-model-cda0d9-kserve-9d99f655c-swqhw"]
Oct 30 13:34:39.731334 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:39.731294    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg"]
Oct 30 13:34:39.735929 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:39.735907    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg"
Oct 30 13:34:39.751362 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:39.751341    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-b4fb25-openshift-default-dockercfg-5469m\""
Oct 30 13:34:39.770392 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:39.770364    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg"]
Oct 30 13:34:39.819839 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:39.819802    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ce05a81-71f3-4dab-96e1-44c9ea628a7b-istio-envoy\") pod \"e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg\" (UID: \"5ce05a81-71f3-4dab-96e1-44c9ea628a7b\") " pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg"
Oct 30 13:34:39.820124 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:39.820101    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ce05a81-71f3-4dab-96e1-44c9ea628a7b-workload-socket\") pod \"e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg\" (UID: \"5ce05a81-71f3-4dab-96e1-44c9ea628a7b\") " pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg"
Oct 30 13:34:39.820259 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:39.820243    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ce05a81-71f3-4dab-96e1-44c9ea628a7b-istio-data\") pod \"e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg\" (UID: \"5ce05a81-71f3-4dab-96e1-44c9ea628a7b\") " pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg"
Oct 30 13:34:39.820496 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:39.820413    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ce05a81-71f3-4dab-96e1-44c9ea628a7b-istio-token\") pod \"e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg\" (UID: \"5ce05a81-71f3-4dab-96e1-44c9ea628a7b\") " pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg"
Oct 30 13:34:39.820712 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:39.820692    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ce05a81-71f3-4dab-96e1-44c9ea628a7b-credential-socket\") pod \"e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg\" (UID: \"5ce05a81-71f3-4dab-96e1-44c9ea628a7b\") " pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg"
Oct 30 13:34:39.821482 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:39.821012    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ce05a81-71f3-4dab-96e1-44c9ea628a7b-workload-certs\") pod \"e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg\" (UID: \"5ce05a81-71f3-4dab-96e1-44c9ea628a7b\") " pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg"
Oct 30 13:34:39.821866 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:39.821842    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ce05a81-71f3-4dab-96e1-44c9ea628a7b-istiod-ca-cert\") pod \"e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg\" (UID: \"5ce05a81-71f3-4dab-96e1-44c9ea628a7b\") " pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg"
Oct 30 13:34:39.822292 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:39.822246    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ce05a81-71f3-4dab-96e1-44c9ea628a7b-istio-podinfo\") pod \"e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg\" (UID: \"5ce05a81-71f3-4dab-96e1-44c9ea628a7b\") " pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg"
Oct 30 13:34:39.822425 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:39.822299    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzdpj\" (UniqueName: \"kubernetes.io/projected/5ce05a81-71f3-4dab-96e1-44c9ea628a7b-kube-api-access-gzdpj\") pod \"e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg\" (UID: \"5ce05a81-71f3-4dab-96e1-44c9ea628a7b\") " pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg"
Oct 30 13:34:39.923320 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:39.923283    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ce05a81-71f3-4dab-96e1-44c9ea628a7b-istio-token\") pod \"e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg\" (UID: \"5ce05a81-71f3-4dab-96e1-44c9ea628a7b\") " pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg"
Oct 30 13:34:39.923517 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:39.923325    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ce05a81-71f3-4dab-96e1-44c9ea628a7b-credential-socket\") pod \"e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg\" (UID: \"5ce05a81-71f3-4dab-96e1-44c9ea628a7b\") " pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg"
Oct 30 13:34:39.923517 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:39.923463    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ce05a81-71f3-4dab-96e1-44c9ea628a7b-workload-certs\") pod \"e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg\" (UID: \"5ce05a81-71f3-4dab-96e1-44c9ea628a7b\") " pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg"
Oct 30 13:34:39.923706 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:39.923578    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ce05a81-71f3-4dab-96e1-44c9ea628a7b-istiod-ca-cert\") pod \"e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg\" (UID: \"5ce05a81-71f3-4dab-96e1-44c9ea628a7b\") " pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg"
Oct 30 13:34:39.923706 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:39.923646    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ce05a81-71f3-4dab-96e1-44c9ea628a7b-istio-podinfo\") pod \"e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg\" (UID: \"5ce05a81-71f3-4dab-96e1-44c9ea628a7b\") " pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg"
Oct 30 13:34:39.923706 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:39.923690    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzdpj\" (UniqueName: \"kubernetes.io/projected/5ce05a81-71f3-4dab-96e1-44c9ea628a7b-kube-api-access-gzdpj\") pod \"e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg\" (UID: \"5ce05a81-71f3-4dab-96e1-44c9ea628a7b\") " pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg"
Oct 30 13:34:39.923871 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:39.923781    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ce05a81-71f3-4dab-96e1-44c9ea628a7b-istio-envoy\") pod \"e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg\" (UID: \"5ce05a81-71f3-4dab-96e1-44c9ea628a7b\") " pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg"
Oct 30 13:34:39.923871 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:39.923853    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ce05a81-71f3-4dab-96e1-44c9ea628a7b-workload-socket\") pod \"e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg\" (UID: \"5ce05a81-71f3-4dab-96e1-44c9ea628a7b\") " pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg"
Oct 30 13:34:39.923871 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:39.923855    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ce05a81-71f3-4dab-96e1-44c9ea628a7b-credential-socket\") pod \"e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg\" (UID: \"5ce05a81-71f3-4dab-96e1-44c9ea628a7b\") " pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg"
Oct 30 13:34:39.924033 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:39.923885    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ce05a81-71f3-4dab-96e1-44c9ea628a7b-istio-data\") pod \"e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg\" (UID: \"5ce05a81-71f3-4dab-96e1-44c9ea628a7b\") " pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg"
Oct 30 13:34:39.924288 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:39.924263    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ce05a81-71f3-4dab-96e1-44c9ea628a7b-istio-data\") pod \"e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg\" (UID: \"5ce05a81-71f3-4dab-96e1-44c9ea628a7b\") " pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg"
Oct 30 13:34:39.924397 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:39.924286    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ce05a81-71f3-4dab-96e1-44c9ea628a7b-istiod-ca-cert\") pod \"e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg\" (UID: \"5ce05a81-71f3-4dab-96e1-44c9ea628a7b\") " pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg"
Oct 30 13:34:39.924491 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:39.924465    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ce05a81-71f3-4dab-96e1-44c9ea628a7b-workload-socket\") pod \"e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg\" (UID: \"5ce05a81-71f3-4dab-96e1-44c9ea628a7b\") " pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg"
Oct 30 13:34:39.924693 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:39.924677    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ce05a81-71f3-4dab-96e1-44c9ea628a7b-workload-certs\") pod \"e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg\" (UID: \"5ce05a81-71f3-4dab-96e1-44c9ea628a7b\") " pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg"
Oct 30 13:34:39.927112 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:39.927090    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ce05a81-71f3-4dab-96e1-44c9ea628a7b-istio-podinfo\") pod \"e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg\" (UID: \"5ce05a81-71f3-4dab-96e1-44c9ea628a7b\") " pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg"
Oct 30 13:34:39.927684 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:39.927661    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ce05a81-71f3-4dab-96e1-44c9ea628a7b-istio-envoy\") pod \"e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg\" (UID: \"5ce05a81-71f3-4dab-96e1-44c9ea628a7b\") " pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg"
Oct 30 13:34:39.952894 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:39.952864    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzdpj\" (UniqueName: \"kubernetes.io/projected/5ce05a81-71f3-4dab-96e1-44c9ea628a7b-kube-api-access-gzdpj\") pod \"e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg\" (UID: \"5ce05a81-71f3-4dab-96e1-44c9ea628a7b\") " pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg"
Oct 30 13:34:39.954317 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:39.954292    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ce05a81-71f3-4dab-96e1-44c9ea628a7b-istio-token\") pod \"e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg\" (UID: \"5ce05a81-71f3-4dab-96e1-44c9ea628a7b\") " pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg"
Oct 30 13:34:40.049324 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:40.049247    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg"
Oct 30 13:34:40.707499 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:40.707469    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg"]
Oct 30 13:34:40.709020 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:34:40.708993    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ce05a81_71f3_4dab_96e1_44c9ea628a7b.slice/crio-8ac9dbf8245846554a0c15ab00ba290546298ddc79464e7e3e759d1a9ff693c9 WatchSource:0}: Error finding container 8ac9dbf8245846554a0c15ab00ba290546298ddc79464e7e3e759d1a9ff693c9: Status 404 returned error can't find the container with id 8ac9dbf8245846554a0c15ab00ba290546298ddc79464e7e3e759d1a9ff693c9
Oct 30 13:34:40.711843 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:40.711811    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:34:40.711932 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:40.711879    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:34:40.711932 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:40.711908    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:34:41.473547 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:41.473507    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg" event={"ID":"5ce05a81-71f3-4dab-96e1-44c9ea628a7b","Type":"ContainerStarted","Data":"67063f13ec7a249a230bf997909730d9f8e56e8c0bde3ab950063eef0a4ac3b5"}
Oct 30 13:34:41.474018 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:41.473572    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg" event={"ID":"5ce05a81-71f3-4dab-96e1-44c9ea628a7b","Type":"ContainerStarted","Data":"8ac9dbf8245846554a0c15ab00ba290546298ddc79464e7e3e759d1a9ff693c9"}
Oct 30 13:34:41.508375 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:34:41.508338    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:34:41.545084 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:41.545032    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg" podStartSLOduration=2.545018256 podStartE2EDuration="2.545018256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:34:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:34:41.530048608 +0000 UTC m=+2580.696413268" watchObservedRunningTime="2026-10-30 13:34:41.545018256 +0000 UTC m=+2580.711382903"
Oct 30 13:34:41.957859 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:41.957827    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq"]
Oct 30 13:34:41.961465 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:41.961446    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq"
Oct 30 13:34:41.981634 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:41.981585    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64219df1-aafc-47e9-8452-b26ec5db3150-credential-socket\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq\" (UID: \"64219df1-aafc-47e9-8452-b26ec5db3150\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq"
Oct 30 13:34:41.981778 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:41.981668    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64219df1-aafc-47e9-8452-b26ec5db3150-istio-envoy\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq\" (UID: \"64219df1-aafc-47e9-8452-b26ec5db3150\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq"
Oct 30 13:34:41.981851 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:41.981823    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64219df1-aafc-47e9-8452-b26ec5db3150-istio-data\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq\" (UID: \"64219df1-aafc-47e9-8452-b26ec5db3150\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq"
Oct 30 13:34:41.981913 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:41.981888    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64219df1-aafc-47e9-8452-b26ec5db3150-istio-token\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq\" (UID: \"64219df1-aafc-47e9-8452-b26ec5db3150\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq"
Oct 30 13:34:41.981974 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:41.981957    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64219df1-aafc-47e9-8452-b26ec5db3150-workload-socket\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq\" (UID: \"64219df1-aafc-47e9-8452-b26ec5db3150\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq"
Oct 30 13:34:41.982040 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:41.982003    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64219df1-aafc-47e9-8452-b26ec5db3150-workload-certs\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq\" (UID: \"64219df1-aafc-47e9-8452-b26ec5db3150\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq"
Oct 30 13:34:41.982099 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:41.982054    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq882\" (UniqueName: \"kubernetes.io/projected/64219df1-aafc-47e9-8452-b26ec5db3150-kube-api-access-hq882\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq\" (UID: \"64219df1-aafc-47e9-8452-b26ec5db3150\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq"
Oct 30 13:34:41.982156 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:41.982124    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64219df1-aafc-47e9-8452-b26ec5db3150-istiod-ca-cert\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq\" (UID: \"64219df1-aafc-47e9-8452-b26ec5db3150\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq"
Oct 30 13:34:41.982383 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:41.982360    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64219df1-aafc-47e9-8452-b26ec5db3150-istio-podinfo\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq\" (UID: \"64219df1-aafc-47e9-8452-b26ec5db3150\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq"
Oct 30 13:34:41.987688 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:41.987665    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq"]
Oct 30 13:34:42.049933 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:42.049905    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg"
Oct 30 13:34:42.056040 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:42.056018    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg"
Oct 30 13:34:42.064441 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:42.064420    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-082748-openshift-default-dockercfg-fhnpr\""
Oct 30 13:34:42.084744 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:42.084720    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64219df1-aafc-47e9-8452-b26ec5db3150-workload-certs\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq\" (UID: \"64219df1-aafc-47e9-8452-b26ec5db3150\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq"
Oct 30 13:34:42.084857 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:42.084790    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq882\" (UniqueName: \"kubernetes.io/projected/64219df1-aafc-47e9-8452-b26ec5db3150-kube-api-access-hq882\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq\" (UID: \"64219df1-aafc-47e9-8452-b26ec5db3150\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq"
Oct 30 13:34:42.084901 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:42.084867    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64219df1-aafc-47e9-8452-b26ec5db3150-istiod-ca-cert\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq\" (UID: \"64219df1-aafc-47e9-8452-b26ec5db3150\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq"
Oct 30 13:34:42.085029 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:42.085012    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64219df1-aafc-47e9-8452-b26ec5db3150-istio-podinfo\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq\" (UID: \"64219df1-aafc-47e9-8452-b26ec5db3150\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq"
Oct 30 13:34:42.085094 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:42.085073    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64219df1-aafc-47e9-8452-b26ec5db3150-credential-socket\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq\" (UID: \"64219df1-aafc-47e9-8452-b26ec5db3150\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq"
Oct 30 13:34:42.085162 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:42.085122    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64219df1-aafc-47e9-8452-b26ec5db3150-istio-envoy\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq\" (UID: \"64219df1-aafc-47e9-8452-b26ec5db3150\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq"
Oct 30 13:34:42.085162 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:42.085144    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64219df1-aafc-47e9-8452-b26ec5db3150-workload-certs\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq\" (UID: \"64219df1-aafc-47e9-8452-b26ec5db3150\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq"
Oct 30 13:34:42.085284 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:42.085191    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64219df1-aafc-47e9-8452-b26ec5db3150-istio-data\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq\" (UID: \"64219df1-aafc-47e9-8452-b26ec5db3150\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq"
Oct 30 13:34:42.085284 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:42.085256    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64219df1-aafc-47e9-8452-b26ec5db3150-istio-token\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq\" (UID: \"64219df1-aafc-47e9-8452-b26ec5db3150\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq"
Oct 30 13:34:42.085397 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:42.085383    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64219df1-aafc-47e9-8452-b26ec5db3150-workload-socket\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq\" (UID: \"64219df1-aafc-47e9-8452-b26ec5db3150\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq"
Oct 30 13:34:42.085459 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:42.085410    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64219df1-aafc-47e9-8452-b26ec5db3150-credential-socket\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq\" (UID: \"64219df1-aafc-47e9-8452-b26ec5db3150\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq"
Oct 30 13:34:42.085693 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:42.085663    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64219df1-aafc-47e9-8452-b26ec5db3150-istiod-ca-cert\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq\" (UID: \"64219df1-aafc-47e9-8452-b26ec5db3150\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq"
Oct 30 13:34:42.085854 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:42.085727    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64219df1-aafc-47e9-8452-b26ec5db3150-workload-socket\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq\" (UID: \"64219df1-aafc-47e9-8452-b26ec5db3150\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq"
Oct 30 13:34:42.085908 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:42.085861    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64219df1-aafc-47e9-8452-b26ec5db3150-istio-data\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq\" (UID: \"64219df1-aafc-47e9-8452-b26ec5db3150\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq"
Oct 30 13:34:42.087531 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:42.087509    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64219df1-aafc-47e9-8452-b26ec5db3150-istio-envoy\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq\" (UID: \"64219df1-aafc-47e9-8452-b26ec5db3150\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq"
Oct 30 13:34:42.087644 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:42.087536    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64219df1-aafc-47e9-8452-b26ec5db3150-istio-podinfo\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq\" (UID: \"64219df1-aafc-47e9-8452-b26ec5db3150\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq"
Oct 30 13:34:42.122463 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:42.122430    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64219df1-aafc-47e9-8452-b26ec5db3150-istio-token\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq\" (UID: \"64219df1-aafc-47e9-8452-b26ec5db3150\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq"
Oct 30 13:34:42.123183 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:42.123154    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq882\" (UniqueName: \"kubernetes.io/projected/64219df1-aafc-47e9-8452-b26ec5db3150-kube-api-access-hq882\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq\" (UID: \"64219df1-aafc-47e9-8452-b26ec5db3150\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq"
Oct 30 13:34:42.274895 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:42.274815    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq"
Oct 30 13:34:42.482567 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:42.482532    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg"
Oct 30 13:34:42.483543 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:42.483517    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg"
Oct 30 13:34:42.645998 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:34:42.645948    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64219df1_aafc_47e9_8452_b26ec5db3150.slice/crio-ecf88cbd334e59fa62b4522bdf056df0df60cff2e3c97ba412c8e60ca84d3f71 WatchSource:0}: Error finding container ecf88cbd334e59fa62b4522bdf056df0df60cff2e3c97ba412c8e60ca84d3f71: Status 404 returned error can't find the container with id ecf88cbd334e59fa62b4522bdf056df0df60cff2e3c97ba412c8e60ca84d3f71
Oct 30 13:34:42.650169 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:42.650126    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:34:42.650304 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:42.650216    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq"]
Oct 30 13:34:42.650304 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:42.650251    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:34:42.650439 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:42.650304    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:34:43.492190 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:43.492149    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq" event={"ID":"64219df1-aafc-47e9-8452-b26ec5db3150","Type":"ContainerStarted","Data":"72d1e09260df7e841a4c031f44df1e1ca8037a6606d5d0e26879e1ae07e3c899"}
Oct 30 13:34:43.492629 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:43.492212    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq" event={"ID":"64219df1-aafc-47e9-8452-b26ec5db3150","Type":"ContainerStarted","Data":"ecf88cbd334e59fa62b4522bdf056df0df60cff2e3c97ba412c8e60ca84d3f71"}
Oct 30 13:34:43.543190 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:43.543137    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq" podStartSLOduration=2.543121077 podStartE2EDuration="2.543121077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:34:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:34:43.538506815 +0000 UTC m=+2582.704871465" watchObservedRunningTime="2026-10-30 13:34:43.543121077 +0000 UTC m=+2582.709485724"
Oct 30 13:34:43.608334 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:43.608300    4364 scope.go:122] "RemoveContainer" containerID="0358195fcc4dc2a31dd1255e0f67eb655f04587a506e63525372bb573686b7cb"
Oct 30 13:34:43.620872 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:43.620850    4364 scope.go:122] "RemoveContainer" containerID="995a7d010a3b52ede92973828802157c68009bb81a3bca9b6d507cb6db9e77fc"
Oct 30 13:34:44.275776 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:44.275736    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq"
Oct 30 13:34:44.277654 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:44.277629    4364 patch_prober.go:29] interesting pod/e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" start-of-body=
Oct 30 13:34:44.277772 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:44.277690    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq" podUID="64219df1-aafc-47e9-8452-b26ec5db3150" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused"
Oct 30 13:34:45.275571 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:45.275535    4364 patch_prober.go:29] interesting pod/e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" start-of-body=
Oct 30 13:34:45.276042 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:45.275630    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq" podUID="64219df1-aafc-47e9-8452-b26ec5db3150" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused"
Oct 30 13:34:46.275174 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:46.275138    4364 patch_prober.go:29] interesting pod/e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" start-of-body=
Oct 30 13:34:46.275361 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:46.275190    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq" podUID="64219df1-aafc-47e9-8452-b26ec5db3150" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused"
Oct 30 13:34:47.289721 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:47.289682    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b4fb25-5596cbb8f4-rcktl"]
Oct 30 13:34:47.419627 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:47.298754    4364 patch_prober.go:29] interesting pod/e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" start-of-body=
Oct 30 13:34:47.419627 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:47.388674    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq" podUID="64219df1-aafc-47e9-8452-b26ec5db3150" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused"
Oct 30 13:34:47.502619 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:47.476682    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b4fb25-5596cbb8f4-rcktl"
Oct 30 13:34:47.529621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:47.522027    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-b4fb25\""
Oct 30 13:34:47.540614 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:47.539572    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-b4fb25-dockercfg-stfr8\""
Oct 30 13:34:47.577723 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:47.558300    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b4fb25-5596cbb8f4-rcktl"]
Oct 30 13:34:47.577723 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:47.577098    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0beff44-3603-4a88-bc25-78f6e3fbbddf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b4fb25-5596cbb8f4-rcktl\" (UID: \"c0beff44-3603-4a88-bc25-78f6e3fbbddf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b4fb25-5596cbb8f4-rcktl"
Oct 30 13:34:47.577723 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:47.577278    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkx5l\" (UniqueName: \"kubernetes.io/projected/c0beff44-3603-4a88-bc25-78f6e3fbbddf-kube-api-access-wkx5l\") pod \"payload-processing-e2e-ipp-a-b4fb25-5596cbb8f4-rcktl\" (UID: \"c0beff44-3603-4a88-bc25-78f6e3fbbddf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b4fb25-5596cbb8f4-rcktl"
Oct 30 13:34:47.678276 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:47.678242    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0beff44-3603-4a88-bc25-78f6e3fbbddf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b4fb25-5596cbb8f4-rcktl\" (UID: \"c0beff44-3603-4a88-bc25-78f6e3fbbddf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b4fb25-5596cbb8f4-rcktl"
Oct 30 13:34:47.678448 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:47.678326    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkx5l\" (UniqueName: \"kubernetes.io/projected/c0beff44-3603-4a88-bc25-78f6e3fbbddf-kube-api-access-wkx5l\") pod \"payload-processing-e2e-ipp-a-b4fb25-5596cbb8f4-rcktl\" (UID: \"c0beff44-3603-4a88-bc25-78f6e3fbbddf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b4fb25-5596cbb8f4-rcktl"
Oct 30 13:34:47.702843 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:47.702809    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0beff44-3603-4a88-bc25-78f6e3fbbddf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b4fb25-5596cbb8f4-rcktl\" (UID: \"c0beff44-3603-4a88-bc25-78f6e3fbbddf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b4fb25-5596cbb8f4-rcktl"
Oct 30 13:34:47.706413 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:47.706382    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkx5l\" (UniqueName: \"kubernetes.io/projected/c0beff44-3603-4a88-bc25-78f6e3fbbddf-kube-api-access-wkx5l\") pod \"payload-processing-e2e-ipp-a-b4fb25-5596cbb8f4-rcktl\" (UID: \"c0beff44-3603-4a88-bc25-78f6e3fbbddf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b4fb25-5596cbb8f4-rcktl"
Oct 30 13:34:47.916371 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:47.916342    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b4fb25-5596cbb8f4-rcktl"
Oct 30 13:34:48.222917 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:48.222889    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b4fb25-5596cbb8f4-rcktl"]
Oct 30 13:34:48.224568 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:34:48.224544    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0beff44_3603_4a88_bc25_78f6e3fbbddf.slice/crio-e411bde0eb1ab8bc8a6689ddf3dda13cb9aae5a845486129df7b40e4b2b55d1b WatchSource:0}: Error finding container e411bde0eb1ab8bc8a6689ddf3dda13cb9aae5a845486129df7b40e4b2b55d1b: Status 404 returned error can't find the container with id e411bde0eb1ab8bc8a6689ddf3dda13cb9aae5a845486129df7b40e4b2b55d1b
Oct 30 13:34:48.276837 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:48.276810    4364 patch_prober.go:29] interesting pod/e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" start-of-body=
Oct 30 13:34:48.276961 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:48.276873    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq" podUID="64219df1-aafc-47e9-8452-b26ec5db3150" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused"
Oct 30 13:34:48.515866 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:48.515784    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-428a0e-65684c5fd4-vgkz2"]
Oct 30 13:34:48.519558 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:48.519536    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-428a0e-65684c5fd4-vgkz2"
Oct 30 13:34:48.547569 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:48.547545    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-428a0e\""
Oct 30 13:34:48.547731 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:48.547574    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-428a0e-dockercfg-r866k\""
Oct 30 13:34:48.556092 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:48.556059    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-428a0e-65684c5fd4-vgkz2"]
Oct 30 13:34:48.616703 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:48.616671    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sxs8\" (UniqueName: \"kubernetes.io/projected/668e3034-e72e-4e25-ab29-d76303e341ac-kube-api-access-5sxs8\") pod \"payload-processing-e2e-ipp-a-428a0e-65684c5fd4-vgkz2\" (UID: \"668e3034-e72e-4e25-ab29-d76303e341ac\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-428a0e-65684c5fd4-vgkz2"
Oct 30 13:34:48.616922 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:48.616868    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/668e3034-e72e-4e25-ab29-d76303e341ac-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-428a0e-65684c5fd4-vgkz2\" (UID: \"668e3034-e72e-4e25-ab29-d76303e341ac\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-428a0e-65684c5fd4-vgkz2"
Oct 30 13:34:48.631504 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:48.631463    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b4fb25-5596cbb8f4-rcktl" event={"ID":"c0beff44-3603-4a88-bc25-78f6e3fbbddf","Type":"ContainerStarted","Data":"16ab27ab7fbec08ea81d7c487418e220711f88ce0cb3943fe05033e304461cab"}
Oct 30 13:34:48.631711 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:48.631537    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b4fb25-5596cbb8f4-rcktl" event={"ID":"c0beff44-3603-4a88-bc25-78f6e3fbbddf","Type":"ContainerStarted","Data":"e411bde0eb1ab8bc8a6689ddf3dda13cb9aae5a845486129df7b40e4b2b55d1b"}
Oct 30 13:34:48.632698 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:48.632673    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-b4fb25-5596cbb8f4-rcktl"
Oct 30 13:34:48.687628 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:48.686331    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-b4fb25-5596cbb8f4-rcktl" podStartSLOduration=1.686307443 podStartE2EDuration="1.686307443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:34:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:34:48.675074125 +0000 UTC m=+2587.841438774" watchObservedRunningTime="2026-10-30 13:34:48.686307443 +0000 UTC m=+2587.852672104"
Oct 30 13:34:48.755903 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:48.755851    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sxs8\" (UniqueName: \"kubernetes.io/projected/668e3034-e72e-4e25-ab29-d76303e341ac-kube-api-access-5sxs8\") pod \"payload-processing-e2e-ipp-a-428a0e-65684c5fd4-vgkz2\" (UID: \"668e3034-e72e-4e25-ab29-d76303e341ac\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-428a0e-65684c5fd4-vgkz2"
Oct 30 13:34:48.756433 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:48.756401    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/668e3034-e72e-4e25-ab29-d76303e341ac-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-428a0e-65684c5fd4-vgkz2\" (UID: \"668e3034-e72e-4e25-ab29-d76303e341ac\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-428a0e-65684c5fd4-vgkz2"
Oct 30 13:34:48.800780 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:48.785363    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l"]
Oct 30 13:34:48.854428 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:48.854399    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l"]
Oct 30 13:34:48.854817 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:48.854797    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l"
Oct 30 13:34:48.985622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:48.942570    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sxs8\" (UniqueName: \"kubernetes.io/projected/668e3034-e72e-4e25-ab29-d76303e341ac-kube-api-access-5sxs8\") pod \"payload-processing-e2e-ipp-a-428a0e-65684c5fd4-vgkz2\" (UID: \"668e3034-e72e-4e25-ab29-d76303e341ac\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-428a0e-65684c5fd4-vgkz2"
Oct 30 13:34:48.985622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:48.961341    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/297ed651-f2f1-40f4-bbe9-bf93abf2ac8d-istio-envoy\") pod \"e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l\" (UID: \"297ed651-f2f1-40f4-bbe9-bf93abf2ac8d\") " pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l"
Oct 30 13:34:48.985622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:48.961394    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/297ed651-f2f1-40f4-bbe9-bf93abf2ac8d-workload-certs\") pod \"e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l\" (UID: \"297ed651-f2f1-40f4-bbe9-bf93abf2ac8d\") " pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l"
Oct 30 13:34:48.985622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:48.961486    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/297ed651-f2f1-40f4-bbe9-bf93abf2ac8d-istiod-ca-cert\") pod \"e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l\" (UID: \"297ed651-f2f1-40f4-bbe9-bf93abf2ac8d\") " pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l"
Oct 30 13:34:48.985622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:48.961558    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/297ed651-f2f1-40f4-bbe9-bf93abf2ac8d-istio-podinfo\") pod \"e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l\" (UID: \"297ed651-f2f1-40f4-bbe9-bf93abf2ac8d\") " pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l"
Oct 30 13:34:48.985622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:48.961680    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/297ed651-f2f1-40f4-bbe9-bf93abf2ac8d-credential-socket\") pod \"e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l\" (UID: \"297ed651-f2f1-40f4-bbe9-bf93abf2ac8d\") " pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l"
Oct 30 13:34:48.985622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:48.961732    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/297ed651-f2f1-40f4-bbe9-bf93abf2ac8d-workload-socket\") pod \"e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l\" (UID: \"297ed651-f2f1-40f4-bbe9-bf93abf2ac8d\") " pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l"
Oct 30 13:34:48.985622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:48.961780    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/297ed651-f2f1-40f4-bbe9-bf93abf2ac8d-istio-data\") pod \"e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l\" (UID: \"297ed651-f2f1-40f4-bbe9-bf93abf2ac8d\") " pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l"
Oct 30 13:34:48.985622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:48.961828    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/297ed651-f2f1-40f4-bbe9-bf93abf2ac8d-istio-token\") pod \"e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l\" (UID: \"297ed651-f2f1-40f4-bbe9-bf93abf2ac8d\") " pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l"
Oct 30 13:34:48.985622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:48.961924    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvjlh\" (UniqueName: \"kubernetes.io/projected/297ed651-f2f1-40f4-bbe9-bf93abf2ac8d-kube-api-access-rvjlh\") pod \"e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l\" (UID: \"297ed651-f2f1-40f4-bbe9-bf93abf2ac8d\") " pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l"
Oct 30 13:34:48.999971 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:48.999935    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/668e3034-e72e-4e25-ab29-d76303e341ac-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-428a0e-65684c5fd4-vgkz2\" (UID: \"668e3034-e72e-4e25-ab29-d76303e341ac\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-428a0e-65684c5fd4-vgkz2"
Oct 30 13:34:49.070618 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:49.067253    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/297ed651-f2f1-40f4-bbe9-bf93abf2ac8d-credential-socket\") pod \"e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l\" (UID: \"297ed651-f2f1-40f4-bbe9-bf93abf2ac8d\") " pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l"
Oct 30 13:34:49.070618 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:49.067331    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/297ed651-f2f1-40f4-bbe9-bf93abf2ac8d-workload-socket\") pod \"e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l\" (UID: \"297ed651-f2f1-40f4-bbe9-bf93abf2ac8d\") " pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l"
Oct 30 13:34:49.070618 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:49.067377    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/297ed651-f2f1-40f4-bbe9-bf93abf2ac8d-istio-data\") pod \"e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l\" (UID: \"297ed651-f2f1-40f4-bbe9-bf93abf2ac8d\") " pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l"
Oct 30 13:34:49.070618 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:49.067423    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/297ed651-f2f1-40f4-bbe9-bf93abf2ac8d-istio-token\") pod \"e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l\" (UID: \"297ed651-f2f1-40f4-bbe9-bf93abf2ac8d\") " pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l"
Oct 30 13:34:49.070618 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:49.067518    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvjlh\" (UniqueName: \"kubernetes.io/projected/297ed651-f2f1-40f4-bbe9-bf93abf2ac8d-kube-api-access-rvjlh\") pod \"e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l\" (UID: \"297ed651-f2f1-40f4-bbe9-bf93abf2ac8d\") " pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l"
Oct 30 13:34:49.070618 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:49.067744    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/297ed651-f2f1-40f4-bbe9-bf93abf2ac8d-istio-envoy\") pod \"e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l\" (UID: \"297ed651-f2f1-40f4-bbe9-bf93abf2ac8d\") " pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l"
Oct 30 13:34:49.070618 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:49.067780    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/297ed651-f2f1-40f4-bbe9-bf93abf2ac8d-workload-certs\") pod \"e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l\" (UID: \"297ed651-f2f1-40f4-bbe9-bf93abf2ac8d\") " pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l"
Oct 30 13:34:49.070618 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:49.067872    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/297ed651-f2f1-40f4-bbe9-bf93abf2ac8d-istiod-ca-cert\") pod \"e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l\" (UID: \"297ed651-f2f1-40f4-bbe9-bf93abf2ac8d\") " pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l"
Oct 30 13:34:49.070618 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:49.067940    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/297ed651-f2f1-40f4-bbe9-bf93abf2ac8d-istio-podinfo\") pod \"e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l\" (UID: \"297ed651-f2f1-40f4-bbe9-bf93abf2ac8d\") " pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l"
Oct 30 13:34:49.071213 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:49.070770    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/297ed651-f2f1-40f4-bbe9-bf93abf2ac8d-credential-socket\") pod \"e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l\" (UID: \"297ed651-f2f1-40f4-bbe9-bf93abf2ac8d\") " pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l"
Oct 30 13:34:49.071213 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:49.071042    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/297ed651-f2f1-40f4-bbe9-bf93abf2ac8d-workload-socket\") pod \"e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l\" (UID: \"297ed651-f2f1-40f4-bbe9-bf93abf2ac8d\") " pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l"
Oct 30 13:34:49.071331 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:49.071287    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/297ed651-f2f1-40f4-bbe9-bf93abf2ac8d-istio-data\") pod \"e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l\" (UID: \"297ed651-f2f1-40f4-bbe9-bf93abf2ac8d\") " pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l"
Oct 30 13:34:49.087114 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:49.074123    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/297ed651-f2f1-40f4-bbe9-bf93abf2ac8d-workload-certs\") pod \"e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l\" (UID: \"297ed651-f2f1-40f4-bbe9-bf93abf2ac8d\") " pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l"
Oct 30 13:34:49.087114 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:49.074880    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/297ed651-f2f1-40f4-bbe9-bf93abf2ac8d-istiod-ca-cert\") pod \"e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l\" (UID: \"297ed651-f2f1-40f4-bbe9-bf93abf2ac8d\") " pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l"
Oct 30 13:34:49.123847 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:49.123327    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-fdd4d4-openshift-default-dockercfg-2qk7r\""
Oct 30 13:34:49.123847 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:49.123453    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/297ed651-f2f1-40f4-bbe9-bf93abf2ac8d-istio-podinfo\") pod \"e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l\" (UID: \"297ed651-f2f1-40f4-bbe9-bf93abf2ac8d\") " pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l"
Oct 30 13:34:49.125091 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:49.125061    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/297ed651-f2f1-40f4-bbe9-bf93abf2ac8d-istio-envoy\") pod \"e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l\" (UID: \"297ed651-f2f1-40f4-bbe9-bf93abf2ac8d\") " pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l"
Oct 30 13:34:49.125491 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:49.125470    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/297ed651-f2f1-40f4-bbe9-bf93abf2ac8d-istio-token\") pod \"e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l\" (UID: \"297ed651-f2f1-40f4-bbe9-bf93abf2ac8d\") " pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l"
Oct 30 13:34:49.126348 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:49.126322    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvjlh\" (UniqueName: \"kubernetes.io/projected/297ed651-f2f1-40f4-bbe9-bf93abf2ac8d-kube-api-access-rvjlh\") pod \"e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l\" (UID: \"297ed651-f2f1-40f4-bbe9-bf93abf2ac8d\") " pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l"
Oct 30 13:34:49.133999 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:49.133968    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-428a0e-65684c5fd4-vgkz2"
Oct 30 13:34:49.275865 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:49.275835    4364 patch_prober.go:29] interesting pod/e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" start-of-body=
Oct 30 13:34:49.276027 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:49.275890    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq" podUID="64219df1-aafc-47e9-8452-b26ec5db3150" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused"
Oct 30 13:34:49.405851 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:49.405473    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l"
Oct 30 13:34:49.497062 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:49.497027    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-428a0e-65684c5fd4-vgkz2"]
Oct 30 13:34:49.640437 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:49.640402    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-428a0e-65684c5fd4-vgkz2" event={"ID":"668e3034-e72e-4e25-ab29-d76303e341ac","Type":"ContainerStarted","Data":"85acf5387a434bbbbc9aa90081ae0f03c074268d1a3a3869517b5fd5ece9ad71"}
Oct 30 13:34:49.640925 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:49.640457    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-428a0e-65684c5fd4-vgkz2" event={"ID":"668e3034-e72e-4e25-ab29-d76303e341ac","Type":"ContainerStarted","Data":"c3041c38be269d05f8a3e921473b937235c43804da974bab7f877ad6d0905280"}
Oct 30 13:34:49.640925 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:49.640500    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-428a0e-65684c5fd4-vgkz2"
Oct 30 13:34:49.680528 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:49.680426    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-428a0e-65684c5fd4-vgkz2" podStartSLOduration=1.6804120299999998 podStartE2EDuration="1.68041203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:34:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:34:49.679947681 +0000 UTC m=+2588.846312330" watchObservedRunningTime="2026-10-30 13:34:49.68041203 +0000 UTC m=+2588.846776678"
Oct 30 13:34:49.792254 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:49.792214    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l"]
Oct 30 13:34:49.885856 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:49.885809    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:34:49.886078 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:49.885914    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:34:49.886078 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:49.885959    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:34:50.281944 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:50.281848    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq"
Oct 30 13:34:50.282248 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:50.282222    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq"
Oct 30 13:34:50.282931 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:50.282903    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq"
Oct 30 13:34:50.650868 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:50.650828    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l" event={"ID":"297ed651-f2f1-40f4-bbe9-bf93abf2ac8d","Type":"ContainerStarted","Data":"1070c42cc863836cf2980d60febfcb8d13f8bb15303a6085e9eb1a2bb304041d"}
Oct 30 13:34:50.651279 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:50.650890    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l" event={"ID":"297ed651-f2f1-40f4-bbe9-bf93abf2ac8d","Type":"ContainerStarted","Data":"7d3b61a8829a6b646eb402fb586112904776d680d1cafc8ce1ec4f03bca65770"}
Oct 30 13:34:50.699494 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:50.699445    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l" podStartSLOduration=2.6994312750000002 podStartE2EDuration="2.699431275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:34:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:34:50.696731591 +0000 UTC m=+2589.863096240" watchObservedRunningTime="2026-10-30 13:34:50.699431275 +0000 UTC m=+2589.865795922"
Oct 30 13:34:51.406503 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:51.406462    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l"
Oct 30 13:34:51.408059 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:51.408025    4364 patch_prober.go:29] interesting pod/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body=
Oct 30 13:34:51.408211 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:51.408075    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l" podUID="297ed651-f2f1-40f4-bbe9-bf93abf2ac8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused"
Oct 30 13:34:52.407240 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:52.407205    4364 patch_prober.go:29] interesting pod/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body=
Oct 30 13:34:52.407665 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:52.407258    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l" podUID="297ed651-f2f1-40f4-bbe9-bf93abf2ac8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused"
Oct 30 13:34:53.406881 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:53.406845    4364 patch_prober.go:29] interesting pod/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body=
Oct 30 13:34:53.407044 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:53.406911    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l" podUID="297ed651-f2f1-40f4-bbe9-bf93abf2ac8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused"
Oct 30 13:34:53.604737 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:53.604686    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-4b0ea9/auth-test-model-4b0ea9-kserve-86458657c-m5qrl" podUID="825bb91f-8422-4b4d-8e13-ad421aa3c20f" containerName="main" containerID="cri-o://9a82173e937d061bd8daf22e63f31c803e8cd578b31bdc5ad3cef5323527e857" gracePeriod=45
Oct 30 13:34:53.951066 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:53.951039    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4b0ea9/auth-test-model-4b0ea9-kserve-86458657c-m5qrl"
Oct 30 13:34:53.971155 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:53.971094    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/825bb91f-8422-4b4d-8e13-ad421aa3c20f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/825bb91f-8422-4b4d-8e13-ad421aa3c20f-dshm\") pod \"825bb91f-8422-4b4d-8e13-ad421aa3c20f\" (UID: \"825bb91f-8422-4b4d-8e13-ad421aa3c20f\") "
Oct 30 13:34:53.971278 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:53.971184    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/825bb91f-8422-4b4d-8e13-ad421aa3c20f-home\" (UniqueName: \"kubernetes.io/empty-dir/825bb91f-8422-4b4d-8e13-ad421aa3c20f-home\") pod \"825bb91f-8422-4b4d-8e13-ad421aa3c20f\" (UID: \"825bb91f-8422-4b4d-8e13-ad421aa3c20f\") "
Oct 30 13:34:53.971278 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:53.971210    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/825bb91f-8422-4b4d-8e13-ad421aa3c20f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/825bb91f-8422-4b4d-8e13-ad421aa3c20f-model-cache\") pod \"825bb91f-8422-4b4d-8e13-ad421aa3c20f\" (UID: \"825bb91f-8422-4b4d-8e13-ad421aa3c20f\") "
Oct 30 13:34:53.971278 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:53.971233    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/825bb91f-8422-4b4d-8e13-ad421aa3c20f-kube-api-access-hqndb\" (UniqueName: \"kubernetes.io/projected/825bb91f-8422-4b4d-8e13-ad421aa3c20f-kube-api-access-hqndb\") pod \"825bb91f-8422-4b4d-8e13-ad421aa3c20f\" (UID: \"825bb91f-8422-4b4d-8e13-ad421aa3c20f\") "
Oct 30 13:34:53.971460 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:53.971325    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/825bb91f-8422-4b4d-8e13-ad421aa3c20f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/825bb91f-8422-4b4d-8e13-ad421aa3c20f-tmp-dir\") pod \"825bb91f-8422-4b4d-8e13-ad421aa3c20f\" (UID: \"825bb91f-8422-4b4d-8e13-ad421aa3c20f\") "
Oct 30 13:34:53.971460 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:53.971409    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/825bb91f-8422-4b4d-8e13-ad421aa3c20f-tls-certs\" (UniqueName: \"kubernetes.io/secret/825bb91f-8422-4b4d-8e13-ad421aa3c20f-tls-certs\") pod \"825bb91f-8422-4b4d-8e13-ad421aa3c20f\" (UID: \"825bb91f-8422-4b4d-8e13-ad421aa3c20f\") "
Oct 30 13:34:53.971572 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:53.971470    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/825bb91f-8422-4b4d-8e13-ad421aa3c20f-model-cache" pod "825bb91f-8422-4b4d-8e13-ad421aa3c20f" (UID: "825bb91f-8422-4b4d-8e13-ad421aa3c20f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:34:53.971572 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:53.971482    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/825bb91f-8422-4b4d-8e13-ad421aa3c20f-home" pod "825bb91f-8422-4b4d-8e13-ad421aa3c20f" (UID: "825bb91f-8422-4b4d-8e13-ad421aa3c20f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:34:53.971699 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:53.971666    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/825bb91f-8422-4b4d-8e13-ad421aa3c20f-tmp-dir" pod "825bb91f-8422-4b4d-8e13-ad421aa3c20f" (UID: "825bb91f-8422-4b4d-8e13-ad421aa3c20f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:34:53.972143 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:53.972120    4364 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/825bb91f-8422-4b4d-8e13-ad421aa3c20f-tmp-dir\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:34:53.972256 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:53.972148    4364 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/825bb91f-8422-4b4d-8e13-ad421aa3c20f-home\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:34:53.972256 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:53.972163    4364 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/825bb91f-8422-4b4d-8e13-ad421aa3c20f-model-cache\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:34:53.973432 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:53.973412    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/825bb91f-8422-4b4d-8e13-ad421aa3c20f-dshm" pod "825bb91f-8422-4b4d-8e13-ad421aa3c20f" (UID: "825bb91f-8422-4b4d-8e13-ad421aa3c20f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:34:53.973553 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:53.973536    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/825bb91f-8422-4b4d-8e13-ad421aa3c20f-tls-certs" pod "825bb91f-8422-4b4d-8e13-ad421aa3c20f" (UID: "825bb91f-8422-4b4d-8e13-ad421aa3c20f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:34:53.973658 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:53.973637    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/825bb91f-8422-4b4d-8e13-ad421aa3c20f-kube-api-access-hqndb" pod "825bb91f-8422-4b4d-8e13-ad421aa3c20f" (UID: "825bb91f-8422-4b4d-8e13-ad421aa3c20f"). InnerVolumeSpecName "kube-api-access-hqndb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:34:54.074566 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:54.074535    4364 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/825bb91f-8422-4b4d-8e13-ad421aa3c20f-dshm\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:34:54.074566 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:54.074563    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqndb\" (UniqueName: \"kubernetes.io/projected/825bb91f-8422-4b4d-8e13-ad421aa3c20f-kube-api-access-hqndb\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:34:54.074566 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:54.074573    4364 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/825bb91f-8422-4b4d-8e13-ad421aa3c20f-tls-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:34:54.074871 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:54.074845    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w6-cda0d9/rate-test-model-cda0d9-kserve-9d99f655c-swqhw" podUID="d21066d6-fdb1-4c53-8e3b-b8b254112229" containerName="main" containerID="cri-o://28611909a0fd3f9b745f018b3f4e78717ff98dbebe07111a5e889fd8b06a6aa5" gracePeriod=45
Oct 30 13:34:54.203169 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:54.187944    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w6-cda0d9/rate-test-model-cda0d9-kserve-9d99f655c-swqhw" podUID="d21066d6-fdb1-4c53-8e3b-b8b254112229" containerName="main" probeResult="failure" output="Get \"https://10.133.0.225:8000/ready\": dial tcp 10.133.0.225:8000: connect: connection refused"
Oct 30 13:34:54.408762 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:54.408725    4364 patch_prober.go:29] interesting pod/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body=
Oct 30 13:34:54.409015 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:54.408980    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l" podUID="297ed651-f2f1-40f4-bbe9-bf93abf2ac8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused"
Oct 30 13:34:54.417954 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:54.417931    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-cda0d9/rate-test-model-cda0d9-kserve-9d99f655c-swqhw"
Oct 30 13:34:54.533673 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:54.511248    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d21066d6-fdb1-4c53-8e3b-b8b254112229-home\" (UniqueName: \"kubernetes.io/empty-dir/d21066d6-fdb1-4c53-8e3b-b8b254112229-home\") pod \"d21066d6-fdb1-4c53-8e3b-b8b254112229\" (UID: \"d21066d6-fdb1-4c53-8e3b-b8b254112229\") "
Oct 30 13:34:54.540257 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:54.540205    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d21066d6-fdb1-4c53-8e3b-b8b254112229-tls-certs\" (UniqueName: \"kubernetes.io/secret/d21066d6-fdb1-4c53-8e3b-b8b254112229-tls-certs\") pod \"d21066d6-fdb1-4c53-8e3b-b8b254112229\" (UID: \"d21066d6-fdb1-4c53-8e3b-b8b254112229\") "
Oct 30 13:34:54.540826 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:54.540798    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d21066d6-fdb1-4c53-8e3b-b8b254112229-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d21066d6-fdb1-4c53-8e3b-b8b254112229-tmp-dir\") pod \"d21066d6-fdb1-4c53-8e3b-b8b254112229\" (UID: \"d21066d6-fdb1-4c53-8e3b-b8b254112229\") "
Oct 30 13:34:54.547634 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:54.542754    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d21066d6-fdb1-4c53-8e3b-b8b254112229-home" pod "d21066d6-fdb1-4c53-8e3b-b8b254112229" (UID: "d21066d6-fdb1-4c53-8e3b-b8b254112229"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:34:54.547882 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:54.547859    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d21066d6-fdb1-4c53-8e3b-b8b254112229-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d21066d6-fdb1-4c53-8e3b-b8b254112229-model-cache\") pod \"d21066d6-fdb1-4c53-8e3b-b8b254112229\" (UID: \"d21066d6-fdb1-4c53-8e3b-b8b254112229\") "
Oct 30 13:34:54.556300 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:54.556267    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d21066d6-fdb1-4c53-8e3b-b8b254112229-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d21066d6-fdb1-4c53-8e3b-b8b254112229-dshm\") pod \"d21066d6-fdb1-4c53-8e3b-b8b254112229\" (UID: \"d21066d6-fdb1-4c53-8e3b-b8b254112229\") "
Oct 30 13:34:54.556870 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:54.556840    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d21066d6-fdb1-4c53-8e3b-b8b254112229-kube-api-access-lrjrf\" (UniqueName: \"kubernetes.io/projected/d21066d6-fdb1-4c53-8e3b-b8b254112229-kube-api-access-lrjrf\") pod \"d21066d6-fdb1-4c53-8e3b-b8b254112229\" (UID: \"d21066d6-fdb1-4c53-8e3b-b8b254112229\") "
Oct 30 13:34:54.558147 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:54.558119    4364 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d21066d6-fdb1-4c53-8e3b-b8b254112229-home\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:34:54.574144 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:54.573833    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d21066d6-fdb1-4c53-8e3b-b8b254112229-kube-api-access-lrjrf" pod "d21066d6-fdb1-4c53-8e3b-b8b254112229" (UID: "d21066d6-fdb1-4c53-8e3b-b8b254112229"). InnerVolumeSpecName "kube-api-access-lrjrf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:34:54.574699 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:54.574442    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d21066d6-fdb1-4c53-8e3b-b8b254112229-tmp-dir" pod "d21066d6-fdb1-4c53-8e3b-b8b254112229" (UID: "d21066d6-fdb1-4c53-8e3b-b8b254112229"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:34:54.575131 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:54.574861    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d21066d6-fdb1-4c53-8e3b-b8b254112229-model-cache" pod "d21066d6-fdb1-4c53-8e3b-b8b254112229" (UID: "d21066d6-fdb1-4c53-8e3b-b8b254112229"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:34:54.616648 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:54.616560    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d21066d6-fdb1-4c53-8e3b-b8b254112229-tls-certs" pod "d21066d6-fdb1-4c53-8e3b-b8b254112229" (UID: "d21066d6-fdb1-4c53-8e3b-b8b254112229"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:34:54.617044 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:54.616688    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d21066d6-fdb1-4c53-8e3b-b8b254112229-dshm" pod "d21066d6-fdb1-4c53-8e3b-b8b254112229" (UID: "d21066d6-fdb1-4c53-8e3b-b8b254112229"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:34:54.659928 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:54.659895    4364 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d21066d6-fdb1-4c53-8e3b-b8b254112229-tmp-dir\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:34:54.659928 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:54.659923    4364 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d21066d6-fdb1-4c53-8e3b-b8b254112229-model-cache\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:34:54.659928 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:54.659931    4364 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d21066d6-fdb1-4c53-8e3b-b8b254112229-dshm\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:34:54.660141 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:54.659941    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrjrf\" (UniqueName: \"kubernetes.io/projected/d21066d6-fdb1-4c53-8e3b-b8b254112229-kube-api-access-lrjrf\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:34:54.660141 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:54.659950    4364 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d21066d6-fdb1-4c53-8e3b-b8b254112229-tls-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:34:54.710978 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:54.710948    4364 generic.go:356] "Generic (PLEG): container finished" podID="d21066d6-fdb1-4c53-8e3b-b8b254112229" containerID="28611909a0fd3f9b745f018b3f4e78717ff98dbebe07111a5e889fd8b06a6aa5" exitCode=0
Oct 30 13:34:54.711234 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:54.711214    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-cda0d9/rate-test-model-cda0d9-kserve-9d99f655c-swqhw"
Oct 30 13:34:54.711302 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:54.711177    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-cda0d9/rate-test-model-cda0d9-kserve-9d99f655c-swqhw" event={"ID":"d21066d6-fdb1-4c53-8e3b-b8b254112229","Type":"ContainerDied","Data":"28611909a0fd3f9b745f018b3f4e78717ff98dbebe07111a5e889fd8b06a6aa5"}
Oct 30 13:34:54.711302 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:54.711292    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-cda0d9/rate-test-model-cda0d9-kserve-9d99f655c-swqhw" event={"ID":"d21066d6-fdb1-4c53-8e3b-b8b254112229","Type":"ContainerDied","Data":"f68408d50e68cbc1cf524b00af6533710dcb67d32f8c736e74458128f084762e"}
Oct 30 13:34:54.711372 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:54.711317    4364 scope.go:122] "RemoveContainer" containerID="28611909a0fd3f9b745f018b3f4e78717ff98dbebe07111a5e889fd8b06a6aa5"
Oct 30 13:34:54.713461 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:54.713435    4364 generic.go:356] "Generic (PLEG): container finished" podID="825bb91f-8422-4b4d-8e13-ad421aa3c20f" containerID="9a82173e937d061bd8daf22e63f31c803e8cd578b31bdc5ad3cef5323527e857" exitCode=0
Oct 30 13:34:54.713577 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:54.713474    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4b0ea9/auth-test-model-4b0ea9-kserve-86458657c-m5qrl"
Oct 30 13:34:54.713780 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:54.713663    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4b0ea9/auth-test-model-4b0ea9-kserve-86458657c-m5qrl" event={"ID":"825bb91f-8422-4b4d-8e13-ad421aa3c20f","Type":"ContainerDied","Data":"9a82173e937d061bd8daf22e63f31c803e8cd578b31bdc5ad3cef5323527e857"}
Oct 30 13:34:54.729629 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:54.714194    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4b0ea9/auth-test-model-4b0ea9-kserve-86458657c-m5qrl" event={"ID":"825bb91f-8422-4b4d-8e13-ad421aa3c20f","Type":"ContainerDied","Data":"f8c9619ccfe9e26804870767fe682ed345d229922b4bc6b3f2e21b2ccdbbb6e9"}
Oct 30 13:34:54.750639 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:54.750612    4364 scope.go:122] "RemoveContainer" containerID="28611909a0fd3f9b745f018b3f4e78717ff98dbebe07111a5e889fd8b06a6aa5"
Oct 30 13:34:54.751010 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:34:54.750984    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28611909a0fd3f9b745f018b3f4e78717ff98dbebe07111a5e889fd8b06a6aa5\": container with ID starting with 28611909a0fd3f9b745f018b3f4e78717ff98dbebe07111a5e889fd8b06a6aa5 not found: ID does not exist" containerID="28611909a0fd3f9b745f018b3f4e78717ff98dbebe07111a5e889fd8b06a6aa5"
Oct 30 13:34:54.751060 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:54.751021    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28611909a0fd3f9b745f018b3f4e78717ff98dbebe07111a5e889fd8b06a6aa5"} err="failed to get container status \"28611909a0fd3f9b745f018b3f4e78717ff98dbebe07111a5e889fd8b06a6aa5\": rpc error: code = NotFound desc = could not find container \"28611909a0fd3f9b745f018b3f4e78717ff98dbebe07111a5e889fd8b06a6aa5\": container with ID starting with 28611909a0fd3f9b745f018b3f4e78717ff98dbebe07111a5e889fd8b06a6aa5 not found: ID does not exist"
Oct 30 13:34:54.751060 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:54.751040    4364 scope.go:122] "RemoveContainer" containerID="9a82173e937d061bd8daf22e63f31c803e8cd578b31bdc5ad3cef5323527e857"
Oct 30 13:34:54.762358 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:54.762335    4364 scope.go:122] "RemoveContainer" containerID="9a82173e937d061bd8daf22e63f31c803e8cd578b31bdc5ad3cef5323527e857"
Oct 30 13:34:54.762699 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:34:54.762676    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a82173e937d061bd8daf22e63f31c803e8cd578b31bdc5ad3cef5323527e857\": container with ID starting with 9a82173e937d061bd8daf22e63f31c803e8cd578b31bdc5ad3cef5323527e857 not found: ID does not exist" containerID="9a82173e937d061bd8daf22e63f31c803e8cd578b31bdc5ad3cef5323527e857"
Oct 30 13:34:54.762755 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:54.762708    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a82173e937d061bd8daf22e63f31c803e8cd578b31bdc5ad3cef5323527e857"} err="failed to get container status \"9a82173e937d061bd8daf22e63f31c803e8cd578b31bdc5ad3cef5323527e857\": rpc error: code = NotFound desc = could not find container \"9a82173e937d061bd8daf22e63f31c803e8cd578b31bdc5ad3cef5323527e857\": container with ID starting with 9a82173e937d061bd8daf22e63f31c803e8cd578b31bdc5ad3cef5323527e857 not found: ID does not exist"
Oct 30 13:34:54.813312 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:54.813236    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4b0ea9/auth-test-model-4b0ea9-kserve-86458657c-m5qrl"]
Oct 30 13:34:54.824320 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:54.824127    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4b0ea9/auth-test-model-4b0ea9-kserve-86458657c-m5qrl"]
Oct 30 13:34:54.832360 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:54.832325    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-cda0d9/rate-test-model-cda0d9-kserve-9d99f655c-swqhw"]
Oct 30 13:34:54.836680 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:54.836658    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w6-cda0d9/rate-test-model-cda0d9-kserve-9d99f655c-swqhw"]
Oct 30 13:34:54.971929 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:54.971900    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="825bb91f-8422-4b4d-8e13-ad421aa3c20f" path="/var/lib/kubelet/pods/825bb91f-8422-4b4d-8e13-ad421aa3c20f/volumes"
Oct 30 13:34:54.972435 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:54.972417    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d21066d6-fdb1-4c53-8e3b-b8b254112229" path="/var/lib/kubelet/pods/d21066d6-fdb1-4c53-8e3b-b8b254112229/volumes"
Oct 30 13:34:55.409113 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:55.406929    4364 patch_prober.go:29] interesting pod/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body=
Oct 30 13:34:55.409113 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:55.406989    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l" podUID="297ed651-f2f1-40f4-bbe9-bf93abf2ac8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused"
Oct 30 13:34:56.410151 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:56.410106    4364 patch_prober.go:29] interesting pod/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body=
Oct 30 13:34:56.410837 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:56.410800    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l" podUID="297ed651-f2f1-40f4-bbe9-bf93abf2ac8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused"
Oct 30 13:34:57.412547 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:57.412521    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l"
Oct 30 13:34:57.413075 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:57.413060    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l"
Oct 30 13:34:57.413835 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:34:57.413813    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l"
Oct 30 13:35:00.654705 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:00.654666    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-b4fb25-5596cbb8f4-rcktl"
Oct 30 13:35:00.655136 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:00.654745    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-428a0e-65684c5fd4-vgkz2"
Oct 30 13:35:11.505245 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:35:11.505202    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:35:19.841814 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:19.841781    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:35:22.153902 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:22.153860    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-8d82d5/rate-test-model-8d82d5-kserve-66649ffff7-dndh9" podUID="d8781a4e-4171-4db3-808e-a8f412894661" containerName="main" probeResult="failure" output="Get \"https://10.133.0.228:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:35:22.194893 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:22.194850    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w5-82796d/auth-test-model-82796d-kserve-fd6dd7b57-fc2sg" podUID="56e242de-da4d-4e53-b4b3-0e4ecd40ef8a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.229:8000/ready\": context deadline exceeded"
Oct 30 13:35:22.229744 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:22.229692    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74" podUID="4f2b06ac-9bf8-4b57-bd22-85532576f9c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:35:22.417893 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:22.417765    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz" podUID="bc5caf69-a649-47e5-8910-9b8f331dfd4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 13:35:22.418650 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:22.418587    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh" podUID="d41ac9e5-eeed-4e07-a7fa-05d3260630b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 30 13:35:22.477372 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:22.422695    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6" podUID="b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:35:22.529980 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:22.424011    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz" podUID="a568f1ce-602d-4069-a41c-9fe1a7635c5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:35:22.544207 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:22.426299    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4" podUID="0cbe6af7-d884-4613-90b8-192e5ffdb027" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 13:35:22.598706 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:22.543696    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr" podUID="446c12df-de12-4c7c-8c71-dfed93cc33f1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded"
Oct 30 13:35:23.083070 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:23.082095    4364 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz" podUID="a568f1ce-602d-4069-a41c-9fe1a7635c5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:35:23.164624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:23.157185    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-8d82d5/rate-test-model-8d82d5-kserve-66649ffff7-dndh9" podUID="d8781a4e-4171-4db3-808e-a8f412894661" containerName="main" probeResult="failure" output="Get \"https://10.133.0.228:8000/ready\": context deadline exceeded"
Oct 30 13:35:23.182855 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:23.170987    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc" podUID="8911b790-331d-490c-876b-647e6d2fcd14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded"
Oct 30 13:35:23.200619 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:23.197828    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w5-82796d/auth-test-model-82796d-kserve-fd6dd7b57-fc2sg" podUID="56e242de-da4d-4e53-b4b3-0e4ecd40ef8a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.229:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:35:23.225625 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:23.223707    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-8d82d5/rate-test-model-8d82d5-kserve-66649ffff7-dndh9"
Oct 30 13:35:23.225625 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:23.223796    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w5-82796d/auth-test-model-82796d-kserve-fd6dd7b57-fc2sg"
Oct 30 13:35:23.321473 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:23.313237    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74" podUID="4f2b06ac-9bf8-4b57-bd22-85532576f9c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded"
Oct 30 13:35:23.321473 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:23.321216    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d833bd/distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74"
Oct 30 13:35:23.422793 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:23.422737    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz" podUID="bc5caf69-a649-47e5-8910-9b8f331dfd4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 13:35:23.442168 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:23.430960    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz"
Oct 30 13:35:23.600622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:23.527253    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6" podUID="b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 13:35:23.601016 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:23.600976    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4" podUID="0cbe6af7-d884-4613-90b8-192e5ffdb027" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:35:23.601487 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:23.550306    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh" podUID="d41ac9e5-eeed-4e07-a7fa-05d3260630b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 30 13:35:23.603319 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:23.603296    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4"
Oct 30 13:35:23.603512 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:23.603499    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh"
Oct 30 13:35:23.637657 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:23.549715    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz" podUID="a568f1ce-602d-4069-a41c-9fe1a7635c5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:35:23.638057 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:23.638027    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr" podUID="446c12df-de12-4c7c-8c71-dfed93cc33f1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded"
Oct 30 13:35:23.677750 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:23.655019    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz"
Oct 30 13:35:23.744086 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:23.744052    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr"
Oct 30 13:35:23.744467 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:23.744436    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6"
Oct 30 13:35:24.252736 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:24.252685    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w5-82796d/auth-test-model-82796d-kserve-fd6dd7b57-fc2sg" podUID="56e242de-da4d-4e53-b4b3-0e4ecd40ef8a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.229:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:35:24.254277 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:24.254241    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc" podUID="8911b790-331d-490c-876b-647e6d2fcd14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:35:24.254660 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:24.254637    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-8d82d5/rate-test-model-8d82d5-kserve-66649ffff7-dndh9" podUID="d8781a4e-4171-4db3-808e-a8f412894661" containerName="main" probeResult="failure" output="Get \"https://10.133.0.228:8000/ready\": context deadline exceeded"
Oct 30 13:35:24.255130 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:24.255110    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc"
Oct 30 13:35:24.330272 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:24.330221    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74" podUID="4f2b06ac-9bf8-4b57-bd22-85532576f9c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:35:24.401210 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:24.398285    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh"
Oct 30 13:35:24.565891 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:24.565794    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz" podUID="bc5caf69-a649-47e5-8910-9b8f331dfd4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:35:24.571192 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:24.571156    4364 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4" podUID="0cbe6af7-d884-4613-90b8-192e5ffdb027" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:35:24.574284 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:24.574261    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6"
Oct 30 13:35:24.602943 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:24.602887    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4" podUID="0cbe6af7-d884-4613-90b8-192e5ffdb027" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:35:24.608082 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:24.608050    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d833bd/distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74"
Oct 30 13:35:24.653632 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:24.636827    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-8d82d5/rate-test-model-8d82d5-kserve-66649ffff7-dndh9"
Oct 30 13:35:24.653632 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:24.638922    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz"
Oct 30 13:35:24.653632 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:24.644267    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz"
Oct 30 13:35:24.671619 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:24.657837    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr"
Oct 30 13:35:24.742534 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:24.742498    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w5-82796d/auth-test-model-82796d-kserve-fd6dd7b57-fc2sg"
Oct 30 13:35:24.743194 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:24.743162    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4"
Oct 30 13:35:24.743672 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:24.743652    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc"
Oct 30 13:35:27.054252 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:27.054210    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm"]
Oct 30 13:35:27.055270 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:27.055246    4364 cpu_manager.go:414] "containerMap: removing container" podUID="d21066d6-fdb1-4c53-8e3b-b8b254112229" containerName="main"
Oct 30 13:35:27.055270 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:27.055270    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d21066d6-fdb1-4c53-8e3b-b8b254112229" containerName="main"
Oct 30 13:35:27.055468 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:27.055325    4364 cpu_manager.go:414] "containerMap: removing container" podUID="825bb91f-8422-4b4d-8e13-ad421aa3c20f" containerName="main"
Oct 30 13:35:27.055468 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:27.055334    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="825bb91f-8422-4b4d-8e13-ad421aa3c20f" containerName="main"
Oct 30 13:35:27.055575 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:27.055506    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="d21066d6-fdb1-4c53-8e3b-b8b254112229" containerName="main"
Oct 30 13:35:27.055575 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:27.055528    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="825bb91f-8422-4b4d-8e13-ad421aa3c20f" containerName="main"
Oct 30 13:35:27.061412 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:27.061392    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm"
Oct 30 13:35:27.112992 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:27.108658    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm"]
Oct 30 13:35:27.211296 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:27.211261    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9abdd340-6b70-4f16-a849-c53d2fe03672-workload-socket\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm\" (UID: \"9abdd340-6b70-4f16-a849-c53d2fe03672\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm"
Oct 30 13:35:27.211296 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:27.211307    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9abdd340-6b70-4f16-a849-c53d2fe03672-workload-certs\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm\" (UID: \"9abdd340-6b70-4f16-a849-c53d2fe03672\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm"
Oct 30 13:35:27.211522 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:27.211387    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9abdd340-6b70-4f16-a849-c53d2fe03672-istio-envoy\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm\" (UID: \"9abdd340-6b70-4f16-a849-c53d2fe03672\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm"
Oct 30 13:35:27.211690 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:27.211660    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9abdd340-6b70-4f16-a849-c53d2fe03672-istio-token\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm\" (UID: \"9abdd340-6b70-4f16-a849-c53d2fe03672\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm"
Oct 30 13:35:27.211839 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:27.211719    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9abdd340-6b70-4f16-a849-c53d2fe03672-istio-podinfo\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm\" (UID: \"9abdd340-6b70-4f16-a849-c53d2fe03672\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm"
Oct 30 13:35:27.211891 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:27.211865    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9abdd340-6b70-4f16-a849-c53d2fe03672-istiod-ca-cert\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm\" (UID: \"9abdd340-6b70-4f16-a849-c53d2fe03672\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm"
Oct 30 13:35:27.211949 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:27.211913    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9abdd340-6b70-4f16-a849-c53d2fe03672-istio-data\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm\" (UID: \"9abdd340-6b70-4f16-a849-c53d2fe03672\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm"
Oct 30 13:35:27.212100 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:27.212081    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvk4c\" (UniqueName: \"kubernetes.io/projected/9abdd340-6b70-4f16-a849-c53d2fe03672-kube-api-access-zvk4c\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm\" (UID: \"9abdd340-6b70-4f16-a849-c53d2fe03672\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm"
Oct 30 13:35:27.212137 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:27.212126    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9abdd340-6b70-4f16-a849-c53d2fe03672-credential-socket\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm\" (UID: \"9abdd340-6b70-4f16-a849-c53d2fe03672\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm"
Oct 30 13:35:27.313270 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:27.313185    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvk4c\" (UniqueName: \"kubernetes.io/projected/9abdd340-6b70-4f16-a849-c53d2fe03672-kube-api-access-zvk4c\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm\" (UID: \"9abdd340-6b70-4f16-a849-c53d2fe03672\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm"
Oct 30 13:35:27.313270 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:27.313237    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9abdd340-6b70-4f16-a849-c53d2fe03672-credential-socket\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm\" (UID: \"9abdd340-6b70-4f16-a849-c53d2fe03672\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm"
Oct 30 13:35:27.313507 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:27.313304    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9abdd340-6b70-4f16-a849-c53d2fe03672-workload-socket\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm\" (UID: \"9abdd340-6b70-4f16-a849-c53d2fe03672\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm"
Oct 30 13:35:27.313507 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:27.313336    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9abdd340-6b70-4f16-a849-c53d2fe03672-workload-certs\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm\" (UID: \"9abdd340-6b70-4f16-a849-c53d2fe03672\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm"
Oct 30 13:35:27.313507 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:27.313364    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9abdd340-6b70-4f16-a849-c53d2fe03672-istio-envoy\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm\" (UID: \"9abdd340-6b70-4f16-a849-c53d2fe03672\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm"
Oct 30 13:35:27.313694 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:27.313629    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9abdd340-6b70-4f16-a849-c53d2fe03672-istio-token\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm\" (UID: \"9abdd340-6b70-4f16-a849-c53d2fe03672\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm"
Oct 30 13:35:27.313694 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:27.313682    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9abdd340-6b70-4f16-a849-c53d2fe03672-istio-podinfo\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm\" (UID: \"9abdd340-6b70-4f16-a849-c53d2fe03672\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm"
Oct 30 13:35:27.313806 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:27.313693    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9abdd340-6b70-4f16-a849-c53d2fe03672-credential-socket\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm\" (UID: \"9abdd340-6b70-4f16-a849-c53d2fe03672\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm"
Oct 30 13:35:27.313806 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:27.313747    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9abdd340-6b70-4f16-a849-c53d2fe03672-istiod-ca-cert\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm\" (UID: \"9abdd340-6b70-4f16-a849-c53d2fe03672\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm"
Oct 30 13:35:27.313806 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:27.313752    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9abdd340-6b70-4f16-a849-c53d2fe03672-workload-certs\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm\" (UID: \"9abdd340-6b70-4f16-a849-c53d2fe03672\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm"
Oct 30 13:35:27.313806 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:27.313781    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9abdd340-6b70-4f16-a849-c53d2fe03672-workload-socket\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm\" (UID: \"9abdd340-6b70-4f16-a849-c53d2fe03672\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm"
Oct 30 13:35:27.314042 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:27.313817    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9abdd340-6b70-4f16-a849-c53d2fe03672-istio-data\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm\" (UID: \"9abdd340-6b70-4f16-a849-c53d2fe03672\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm"
Oct 30 13:35:27.314169 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:27.314145    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9abdd340-6b70-4f16-a849-c53d2fe03672-istio-data\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm\" (UID: \"9abdd340-6b70-4f16-a849-c53d2fe03672\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm"
Oct 30 13:35:27.314371 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:27.314349    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9abdd340-6b70-4f16-a849-c53d2fe03672-istiod-ca-cert\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm\" (UID: \"9abdd340-6b70-4f16-a849-c53d2fe03672\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm"
Oct 30 13:35:27.316138 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:27.316114    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9abdd340-6b70-4f16-a849-c53d2fe03672-istio-envoy\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm\" (UID: \"9abdd340-6b70-4f16-a849-c53d2fe03672\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm"
Oct 30 13:35:27.316344 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:27.316325    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9abdd340-6b70-4f16-a849-c53d2fe03672-istio-podinfo\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm\" (UID: \"9abdd340-6b70-4f16-a849-c53d2fe03672\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm"
Oct 30 13:35:27.336877 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:27.336848    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9abdd340-6b70-4f16-a849-c53d2fe03672-istio-token\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm\" (UID: \"9abdd340-6b70-4f16-a849-c53d2fe03672\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm"
Oct 30 13:35:27.337084 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:27.337062    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvk4c\" (UniqueName: \"kubernetes.io/projected/9abdd340-6b70-4f16-a849-c53d2fe03672-kube-api-access-zvk4c\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm\" (UID: \"9abdd340-6b70-4f16-a849-c53d2fe03672\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm"
Oct 30 13:35:27.378098 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:27.378075    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm"
Oct 30 13:35:27.712411 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:27.712382    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm"]
Oct 30 13:35:27.713029 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:35:27.712999    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9abdd340_6b70_4f16_a849_c53d2fe03672.slice/crio-b3c07a6b9604b3ab6407ddce4a412b7d5f07512f502147ea89aca3a00f1f416b WatchSource:0}: Error finding container b3c07a6b9604b3ab6407ddce4a412b7d5f07512f502147ea89aca3a00f1f416b: Status 404 returned error can't find the container with id b3c07a6b9604b3ab6407ddce4a412b7d5f07512f502147ea89aca3a00f1f416b
Oct 30 13:35:27.716051 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:27.716018    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:35:27.716143 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:27.716086    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:35:27.716143 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:27.716118    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:35:27.852912 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:27.852866    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm" event={"ID":"9abdd340-6b70-4f16-a849-c53d2fe03672","Type":"ContainerStarted","Data":"dffa0b0b6a13bfc17aae55501ef04fe767dcb62f318ca0a5def37a0a131ce7eb"}
Oct 30 13:35:27.853099 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:27.852933    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm" event={"ID":"9abdd340-6b70-4f16-a849-c53d2fe03672","Type":"ContainerStarted","Data":"b3c07a6b9604b3ab6407ddce4a412b7d5f07512f502147ea89aca3a00f1f416b"}
Oct 30 13:35:27.905459 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:27.905386    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm" podStartSLOduration=0.905367059 podStartE2EDuration="905.367059ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:35:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:35:27.901981584 +0000 UTC m=+2627.068346232" watchObservedRunningTime="2026-10-30 13:35:27.905367059 +0000 UTC m=+2627.071731712"
Oct 30 13:35:28.404815 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:28.404774    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm"
Oct 30 13:35:28.405989 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:28.405958    4364 patch_prober.go:29] interesting pod/e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body=
Oct 30 13:35:28.406118 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:28.406054    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm" podUID="9abdd340-6b70-4f16-a849-c53d2fe03672" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused"
Oct 30 13:35:29.405238 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:29.404746    4364 patch_prober.go:29] interesting pod/e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body=
Oct 30 13:35:29.405238 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:29.404813    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm" podUID="9abdd340-6b70-4f16-a849-c53d2fe03672" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused"
Oct 30 13:35:30.378592 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:30.378562    4364 patch_prober.go:29] interesting pod/e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body=
Oct 30 13:35:30.378823 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:30.378635    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm" podUID="9abdd340-6b70-4f16-a849-c53d2fe03672" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused"
Oct 30 13:35:31.403622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:31.389234    4364 patch_prober.go:29] interesting pod/e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body=
Oct 30 13:35:31.403622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:31.389294    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm" podUID="9abdd340-6b70-4f16-a849-c53d2fe03672" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused"
Oct 30 13:35:32.392622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:32.382947    4364 patch_prober.go:29] interesting pod/e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body=
Oct 30 13:35:32.392622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:32.383011    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm" podUID="9abdd340-6b70-4f16-a849-c53d2fe03672" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused"
Oct 30 13:35:33.392624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:33.391649    4364 patch_prober.go:29] interesting pod/e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body=
Oct 30 13:35:33.392624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:33.391721    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm" podUID="9abdd340-6b70-4f16-a849-c53d2fe03672" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused"
Oct 30 13:35:34.383594 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:34.383560    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm"
Oct 30 13:35:34.383974 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:34.383953    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm"
Oct 30 13:35:34.384699 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:34.384681    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm"
Oct 30 13:35:41.505874 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:35:41.505836    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:35:58.572857 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:35:58.572776    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b29df4-857fc5bcc9-lw9kb"]
Oct 30 13:36:00.847102 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:00.847061    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-82796d/auth-test-model-82796d-kserve-fd6dd7b57-fc2sg"]
Oct 30 13:36:03.574009 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:03.573953    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b29df4-857fc5bcc9-lw9kb" podUID="49bf7b08-c971-4722-af65-b145ed2b1f21" containerName="payload-pre-processing" containerID="cri-o://0c9916c6065a29944e8de212bf1390499e8b338060c0e5ed28a7d002b3b3270d" gracePeriod=25
Oct 30 13:36:07.426256 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:07.426211    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a8d3b8-689875d59-92xn4"]
Oct 30 13:36:07.432835 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:07.432810    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a8d3b8-689875d59-92xn4"
Oct 30 13:36:07.448247 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:07.448214    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-a8d3b8-dockercfg-hlgm9\""
Oct 30 13:36:07.450422 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:07.450394    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a8d3b8-689875d59-92xn4"]
Oct 30 13:36:07.455952 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:07.455848    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-a8d3b8\""
Oct 30 13:36:07.599048 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:07.597942    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7840586e-1003-414a-8ce9-b988bb4a34ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a8d3b8-689875d59-92xn4\" (UID: \"7840586e-1003-414a-8ce9-b988bb4a34ac\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a8d3b8-689875d59-92xn4"
Oct 30 13:36:07.599048 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:07.598297    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtnhj\" (UniqueName: \"kubernetes.io/projected/7840586e-1003-414a-8ce9-b988bb4a34ac-kube-api-access-qtnhj\") pod \"payload-pre-processing-e2e-ipp-a-a8d3b8-689875d59-92xn4\" (UID: \"7840586e-1003-414a-8ce9-b988bb4a34ac\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a8d3b8-689875d59-92xn4"
Oct 30 13:36:07.763783 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:07.755115    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtnhj\" (UniqueName: \"kubernetes.io/projected/7840586e-1003-414a-8ce9-b988bb4a34ac-kube-api-access-qtnhj\") pod \"payload-pre-processing-e2e-ipp-a-a8d3b8-689875d59-92xn4\" (UID: \"7840586e-1003-414a-8ce9-b988bb4a34ac\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a8d3b8-689875d59-92xn4"
Oct 30 13:36:07.763783 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:07.755269    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7840586e-1003-414a-8ce9-b988bb4a34ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a8d3b8-689875d59-92xn4\" (UID: \"7840586e-1003-414a-8ce9-b988bb4a34ac\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a8d3b8-689875d59-92xn4"
Oct 30 13:36:07.763783 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:07.756317    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7840586e-1003-414a-8ce9-b988bb4a34ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a8d3b8-689875d59-92xn4\" (UID: \"7840586e-1003-414a-8ce9-b988bb4a34ac\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a8d3b8-689875d59-92xn4"
Oct 30 13:36:07.857218 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:07.857181    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtnhj\" (UniqueName: \"kubernetes.io/projected/7840586e-1003-414a-8ce9-b988bb4a34ac-kube-api-access-qtnhj\") pod \"payload-pre-processing-e2e-ipp-a-a8d3b8-689875d59-92xn4\" (UID: \"7840586e-1003-414a-8ce9-b988bb4a34ac\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a8d3b8-689875d59-92xn4"
Oct 30 13:36:08.052155 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:08.052068    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a8d3b8-689875d59-92xn4"
Oct 30 13:36:08.367901 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:08.367861    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a8d3b8-689875d59-92xn4"]
Oct 30 13:36:08.368897 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:36:08.368871    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7840586e_1003_414a_8ce9_b988bb4a34ac.slice/crio-ca7a0a9b9e0b88c3cb7f70397f7b92ba856ae22f566972a39d33e04ed3ede374 WatchSource:0}: Error finding container ca7a0a9b9e0b88c3cb7f70397f7b92ba856ae22f566972a39d33e04ed3ede374: Status 404 returned error can't find the container with id ca7a0a9b9e0b88c3cb7f70397f7b92ba856ae22f566972a39d33e04ed3ede374
Oct 30 13:36:08.877975 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:08.877921    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a8d3b8-689875d59-92xn4" event={"ID":"7840586e-1003-414a-8ce9-b988bb4a34ac","Type":"ContainerStarted","Data":"288c6d42f9f8efc6efcdf101961bbf502b523699c63d39e1ec1a49a4d4654136"}
Oct 30 13:36:08.878466 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:08.877986    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a8d3b8-689875d59-92xn4" event={"ID":"7840586e-1003-414a-8ce9-b988bb4a34ac","Type":"ContainerStarted","Data":"ca7a0a9b9e0b88c3cb7f70397f7b92ba856ae22f566972a39d33e04ed3ede374"}
Oct 30 13:36:08.878466 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:08.878016    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a8d3b8-689875d59-92xn4"
Oct 30 13:36:08.949624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:08.946918    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a8d3b8-689875d59-92xn4" podStartSLOduration=1.94689378 podStartE2EDuration="1.94689378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:36:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:36:08.933947874 +0000 UTC m=+2668.100312526" watchObservedRunningTime="2026-10-30 13:36:08.94689378 +0000 UTC m=+2668.113258455"
Oct 30 13:36:09.864555 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:09.864521    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b29df4-857fc5bcc9-lw9kb" podUID="49bf7b08-c971-4722-af65-b145ed2b1f21" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:36:11.505924 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:36:11.505885    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:36:13.549223 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:13.543693    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:36:14.416303 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:14.416264    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf3c79-7846779cd-pmvpz"]
Oct 30 13:36:14.421366 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:14.421341    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf3c79-7846779cd-pmvpz"
Oct 30 13:36:14.449709 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:14.449593    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-cf3c79-serving-cert\""
Oct 30 13:36:14.456198 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:14.456145    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf3c79-7846779cd-pmvpz"]
Oct 30 13:36:14.511973 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:14.511936    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmlzz\" (UniqueName: \"kubernetes.io/projected/966c9a37-1b20-416b-aa28-c9703fca7fbe-kube-api-access-qmlzz\") pod \"maas-api-e2e-ipp-a-cf3c79-7846779cd-pmvpz\" (UID: \"966c9a37-1b20-416b-aa28-c9703fca7fbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf3c79-7846779cd-pmvpz"
Oct 30 13:36:14.512509 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:14.512261    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/966c9a37-1b20-416b-aa28-c9703fca7fbe-metrics-certs\") pod \"maas-api-e2e-ipp-a-cf3c79-7846779cd-pmvpz\" (UID: \"966c9a37-1b20-416b-aa28-c9703fca7fbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf3c79-7846779cd-pmvpz"
Oct 30 13:36:14.512509 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:14.512332    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/966c9a37-1b20-416b-aa28-c9703fca7fbe-maas-api-tls\") pod \"maas-api-e2e-ipp-a-cf3c79-7846779cd-pmvpz\" (UID: \"966c9a37-1b20-416b-aa28-c9703fca7fbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf3c79-7846779cd-pmvpz"
Oct 30 13:36:14.512509 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:14.512365    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/966c9a37-1b20-416b-aa28-c9703fca7fbe-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-cf3c79-7846779cd-pmvpz\" (UID: \"966c9a37-1b20-416b-aa28-c9703fca7fbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf3c79-7846779cd-pmvpz"
Oct 30 13:36:14.614786 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:14.614520    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/966c9a37-1b20-416b-aa28-c9703fca7fbe-maas-api-tls\") pod \"maas-api-e2e-ipp-a-cf3c79-7846779cd-pmvpz\" (UID: \"966c9a37-1b20-416b-aa28-c9703fca7fbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf3c79-7846779cd-pmvpz"
Oct 30 13:36:14.614786 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:14.614575    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/966c9a37-1b20-416b-aa28-c9703fca7fbe-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-cf3c79-7846779cd-pmvpz\" (UID: \"966c9a37-1b20-416b-aa28-c9703fca7fbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf3c79-7846779cd-pmvpz"
Oct 30 13:36:14.615493 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:14.614821    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmlzz\" (UniqueName: \"kubernetes.io/projected/966c9a37-1b20-416b-aa28-c9703fca7fbe-kube-api-access-qmlzz\") pod \"maas-api-e2e-ipp-a-cf3c79-7846779cd-pmvpz\" (UID: \"966c9a37-1b20-416b-aa28-c9703fca7fbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf3c79-7846779cd-pmvpz"
Oct 30 13:36:14.615493 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:14.615081    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/966c9a37-1b20-416b-aa28-c9703fca7fbe-metrics-certs\") pod \"maas-api-e2e-ipp-a-cf3c79-7846779cd-pmvpz\" (UID: \"966c9a37-1b20-416b-aa28-c9703fca7fbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf3c79-7846779cd-pmvpz"
Oct 30 13:36:14.615636 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:14.615563    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/966c9a37-1b20-416b-aa28-c9703fca7fbe-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-cf3c79-7846779cd-pmvpz\" (UID: \"966c9a37-1b20-416b-aa28-c9703fca7fbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf3c79-7846779cd-pmvpz"
Oct 30 13:36:14.618034 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:14.618012    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/966c9a37-1b20-416b-aa28-c9703fca7fbe-maas-api-tls\") pod \"maas-api-e2e-ipp-a-cf3c79-7846779cd-pmvpz\" (UID: \"966c9a37-1b20-416b-aa28-c9703fca7fbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf3c79-7846779cd-pmvpz"
Oct 30 13:36:14.618155 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:14.618072    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/966c9a37-1b20-416b-aa28-c9703fca7fbe-metrics-certs\") pod \"maas-api-e2e-ipp-a-cf3c79-7846779cd-pmvpz\" (UID: \"966c9a37-1b20-416b-aa28-c9703fca7fbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf3c79-7846779cd-pmvpz"
Oct 30 13:36:14.641442 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:14.641410    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmlzz\" (UniqueName: \"kubernetes.io/projected/966c9a37-1b20-416b-aa28-c9703fca7fbe-kube-api-access-qmlzz\") pod \"maas-api-e2e-ipp-a-cf3c79-7846779cd-pmvpz\" (UID: \"966c9a37-1b20-416b-aa28-c9703fca7fbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf3c79-7846779cd-pmvpz"
Oct 30 13:36:14.742679 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:14.738102    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf3c79-7846779cd-pmvpz"
Oct 30 13:36:15.840684 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:15.804733    4364 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-d833bd/distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74" podUID="4f2b06ac-9bf8-4b57-bd22-85532576f9c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:36:16.131253 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:16.021896    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w5-82796d/auth-test-model-82796d-kserve-fd6dd7b57-fc2sg" podUID="56e242de-da4d-4e53-b4b3-0e4ecd40ef8a" containerName="main" containerID="cri-o://39cbb0c9f1231b21a1ce74e65cf4be03949fa99b9e166da0becd7b85d8168017" gracePeriod=45
Oct 30 13:36:16.185628 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:16.174813    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf3c79-7846779cd-pmvpz"]
Oct 30 13:36:16.255800 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:16.253906    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w5-82796d/auth-test-model-82796d-kserve-fd6dd7b57-fc2sg" podUID="56e242de-da4d-4e53-b4b3-0e4ecd40ef8a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.229:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:36:16.580105 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:16.580059    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz" podUID="bc5caf69-a649-47e5-8910-9b8f331dfd4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:36:16.629180 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:16.628491    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74" podUID="4f2b06ac-9bf8-4b57-bd22-85532576f9c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:36:16.637690 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:16.629436    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4" podUID="0cbe6af7-d884-4613-90b8-192e5ffdb027" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 13:36:16.658298 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:16.656713    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh" podUID="d41ac9e5-eeed-4e07-a7fa-05d3260630b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:36:16.658298 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:16.657090    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz" podUID="a568f1ce-602d-4069-a41c-9fe1a7635c5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 30 13:36:16.663724 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:16.663096    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w5-82796d/auth-test-model-82796d-kserve-fd6dd7b57-fc2sg" podUID="56e242de-da4d-4e53-b4b3-0e4ecd40ef8a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.229:8000/ready\": read tcp 10.133.0.2:37934->10.133.0.229:8000: read: connection reset by peer"
Oct 30 13:36:16.663724 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:16.663195    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w5-82796d/auth-test-model-82796d-kserve-fd6dd7b57-fc2sg"
Oct 30 13:36:16.678182 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:16.678130    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr" podUID="446c12df-de12-4c7c-8c71-dfed93cc33f1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:36:16.772491 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:16.771616    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc" podUID="8911b790-331d-490c-876b-647e6d2fcd14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:36:16.781335 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:16.773510    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6" podUID="b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:36:17.277617 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:17.260631    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w5-82796d/auth-test-model-82796d-kserve-fd6dd7b57-fc2sg" podUID="56e242de-da4d-4e53-b4b3-0e4ecd40ef8a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.229:8000/ready\": dial tcp 10.133.0.229:8000: connect: connection refused"
Oct 30 13:36:17.299146 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:17.297679    4364 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh" podUID="d41ac9e5-eeed-4e07-a7fa-05d3260630b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:36:17.415136 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:17.415091    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf3c79-7846779cd-pmvpz" event={"ID":"966c9a37-1b20-416b-aa28-c9703fca7fbe","Type":"ContainerStarted","Data":"82bb8da453afd6703ebb7d6d065c5fcaf52e14ef102230914da847255497e3d9"}
Oct 30 13:36:18.070960 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:18.070933    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-82796d/auth-test-model-82796d-kserve-fd6dd7b57-fc2sg"
Oct 30 13:36:18.178014 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:18.177936    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56e242de-da4d-4e53-b4b3-0e4ecd40ef8a-home\" (UniqueName: \"kubernetes.io/empty-dir/56e242de-da4d-4e53-b4b3-0e4ecd40ef8a-home\") pod \"56e242de-da4d-4e53-b4b3-0e4ecd40ef8a\" (UID: \"56e242de-da4d-4e53-b4b3-0e4ecd40ef8a\") "
Oct 30 13:36:18.178161 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:18.178068    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56e242de-da4d-4e53-b4b3-0e4ecd40ef8a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/56e242de-da4d-4e53-b4b3-0e4ecd40ef8a-dshm\") pod \"56e242de-da4d-4e53-b4b3-0e4ecd40ef8a\" (UID: \"56e242de-da4d-4e53-b4b3-0e4ecd40ef8a\") "
Oct 30 13:36:18.178161 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:18.178153    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56e242de-da4d-4e53-b4b3-0e4ecd40ef8a-tls-certs\" (UniqueName: \"kubernetes.io/secret/56e242de-da4d-4e53-b4b3-0e4ecd40ef8a-tls-certs\") pod \"56e242de-da4d-4e53-b4b3-0e4ecd40ef8a\" (UID: \"56e242de-da4d-4e53-b4b3-0e4ecd40ef8a\") "
Oct 30 13:36:18.178244 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:18.178174    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56e242de-da4d-4e53-b4b3-0e4ecd40ef8a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56e242de-da4d-4e53-b4b3-0e4ecd40ef8a-model-cache\") pod \"56e242de-da4d-4e53-b4b3-0e4ecd40ef8a\" (UID: \"56e242de-da4d-4e53-b4b3-0e4ecd40ef8a\") "
Oct 30 13:36:18.178283 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:18.178239    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56e242de-da4d-4e53-b4b3-0e4ecd40ef8a-home" pod "56e242de-da4d-4e53-b4b3-0e4ecd40ef8a" (UID: "56e242de-da4d-4e53-b4b3-0e4ecd40ef8a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:36:18.178283 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:18.178251    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56e242de-da4d-4e53-b4b3-0e4ecd40ef8a-kube-api-access-44fvq\" (UniqueName: \"kubernetes.io/projected/56e242de-da4d-4e53-b4b3-0e4ecd40ef8a-kube-api-access-44fvq\") pod \"56e242de-da4d-4e53-b4b3-0e4ecd40ef8a\" (UID: \"56e242de-da4d-4e53-b4b3-0e4ecd40ef8a\") "
Oct 30 13:36:18.178360 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:18.178327    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56e242de-da4d-4e53-b4b3-0e4ecd40ef8a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56e242de-da4d-4e53-b4b3-0e4ecd40ef8a-tmp-dir\") pod \"56e242de-da4d-4e53-b4b3-0e4ecd40ef8a\" (UID: \"56e242de-da4d-4e53-b4b3-0e4ecd40ef8a\") "
Oct 30 13:36:18.178542 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:18.178517    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56e242de-da4d-4e53-b4b3-0e4ecd40ef8a-model-cache" pod "56e242de-da4d-4e53-b4b3-0e4ecd40ef8a" (UID: "56e242de-da4d-4e53-b4b3-0e4ecd40ef8a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:36:18.178726 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:18.178650    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56e242de-da4d-4e53-b4b3-0e4ecd40ef8a-tmp-dir" pod "56e242de-da4d-4e53-b4b3-0e4ecd40ef8a" (UID: "56e242de-da4d-4e53-b4b3-0e4ecd40ef8a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:36:18.179159 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:18.179138    4364 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56e242de-da4d-4e53-b4b3-0e4ecd40ef8a-home\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:36:18.179159 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:18.179159    4364 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56e242de-da4d-4e53-b4b3-0e4ecd40ef8a-model-cache\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:36:18.179326 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:18.179170    4364 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56e242de-da4d-4e53-b4b3-0e4ecd40ef8a-tmp-dir\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:36:18.180312 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:18.180292    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56e242de-da4d-4e53-b4b3-0e4ecd40ef8a-dshm" pod "56e242de-da4d-4e53-b4b3-0e4ecd40ef8a" (UID: "56e242de-da4d-4e53-b4b3-0e4ecd40ef8a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:36:18.180426 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:18.180406    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56e242de-da4d-4e53-b4b3-0e4ecd40ef8a-tls-certs" pod "56e242de-da4d-4e53-b4b3-0e4ecd40ef8a" (UID: "56e242de-da4d-4e53-b4b3-0e4ecd40ef8a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:36:18.180471 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:18.180447    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56e242de-da4d-4e53-b4b3-0e4ecd40ef8a-kube-api-access-44fvq" pod "56e242de-da4d-4e53-b4b3-0e4ecd40ef8a" (UID: "56e242de-da4d-4e53-b4b3-0e4ecd40ef8a"). InnerVolumeSpecName "kube-api-access-44fvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:36:18.280998 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:18.280965    4364 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56e242de-da4d-4e53-b4b3-0e4ecd40ef8a-dshm\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:36:18.280998 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:18.280992    4364 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56e242de-da4d-4e53-b4b3-0e4ecd40ef8a-tls-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:36:18.280998 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:18.281001    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44fvq\" (UniqueName: \"kubernetes.io/projected/56e242de-da4d-4e53-b4b3-0e4ecd40ef8a-kube-api-access-44fvq\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:36:18.425031 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:18.424995    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf3c79-7846779cd-pmvpz" event={"ID":"966c9a37-1b20-416b-aa28-c9703fca7fbe","Type":"ContainerStarted","Data":"3a514d1ecd07faf5726c8706cc5ebec9523625416c362bd58b6b0fc893f761f6"}
Oct 30 13:36:18.425231 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:18.425062    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf3c79-7846779cd-pmvpz"
Oct 30 13:36:18.426550 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:18.426522    4364 generic.go:356] "Generic (PLEG): container finished" podID="56e242de-da4d-4e53-b4b3-0e4ecd40ef8a" containerID="39cbb0c9f1231b21a1ce74e65cf4be03949fa99b9e166da0becd7b85d8168017" exitCode=0
Oct 30 13:36:18.426733 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:18.426566    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-82796d/auth-test-model-82796d-kserve-fd6dd7b57-fc2sg"
Oct 30 13:36:18.426733 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:18.426642    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-82796d/auth-test-model-82796d-kserve-fd6dd7b57-fc2sg" event={"ID":"56e242de-da4d-4e53-b4b3-0e4ecd40ef8a","Type":"ContainerDied","Data":"39cbb0c9f1231b21a1ce74e65cf4be03949fa99b9e166da0becd7b85d8168017"}
Oct 30 13:36:18.426733 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:18.426689    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-82796d/auth-test-model-82796d-kserve-fd6dd7b57-fc2sg" event={"ID":"56e242de-da4d-4e53-b4b3-0e4ecd40ef8a","Type":"ContainerDied","Data":"cb681c0c75c29bd708d3897a265aeeafe72daa5d36dbb2e054dc215142da3d67"}
Oct 30 13:36:18.426733 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:18.426708    4364 scope.go:122] "RemoveContainer" containerID="39cbb0c9f1231b21a1ce74e65cf4be03949fa99b9e166da0becd7b85d8168017"
Oct 30 13:36:18.439477 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:18.439459    4364 scope.go:122] "RemoveContainer" containerID="39cbb0c9f1231b21a1ce74e65cf4be03949fa99b9e166da0becd7b85d8168017"
Oct 30 13:36:18.439852 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:36:18.439820    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39cbb0c9f1231b21a1ce74e65cf4be03949fa99b9e166da0becd7b85d8168017\": container with ID starting with 39cbb0c9f1231b21a1ce74e65cf4be03949fa99b9e166da0becd7b85d8168017 not found: ID does not exist" containerID="39cbb0c9f1231b21a1ce74e65cf4be03949fa99b9e166da0becd7b85d8168017"
Oct 30 13:36:18.439949 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:18.439857    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39cbb0c9f1231b21a1ce74e65cf4be03949fa99b9e166da0becd7b85d8168017"} err="failed to get container status \"39cbb0c9f1231b21a1ce74e65cf4be03949fa99b9e166da0becd7b85d8168017\": rpc error: code = NotFound desc = could not find container \"39cbb0c9f1231b21a1ce74e65cf4be03949fa99b9e166da0becd7b85d8168017\": container with ID starting with 39cbb0c9f1231b21a1ce74e65cf4be03949fa99b9e166da0becd7b85d8168017 not found: ID does not exist"
Oct 30 13:36:18.472929 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:18.472887    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf3c79-7846779cd-pmvpz" podStartSLOduration=3.950642567 podStartE2EDuration="4.47287454s" totalImagesPullingTime="522.231973ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:36:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:36:18.467912098 +0000 UTC m=+2677.634276746" watchObservedRunningTime="2026-10-30 13:36:18.47287454 +0000 UTC m=+2677.639239188"
Oct 30 13:36:18.508165 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:18.508141    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-82796d/auth-test-model-82796d-kserve-fd6dd7b57-fc2sg"]
Oct 30 13:36:18.512884 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:18.512864    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w5-82796d/auth-test-model-82796d-kserve-fd6dd7b57-fc2sg"]
Oct 30 13:36:18.973124 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:18.973093    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56e242de-da4d-4e53-b4b3-0e4ecd40ef8a" path="/var/lib/kubelet/pods/56e242de-da4d-4e53-b4b3-0e4ecd40ef8a/volumes"
Oct 30 13:36:19.865214 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:19.865175    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b29df4-857fc5bcc9-lw9kb" podUID="49bf7b08-c971-4722-af65-b145ed2b1f21" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:36:19.888810 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:19.888783    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a8d3b8-689875d59-92xn4"
Oct 30 13:36:21.172080 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:21.172043    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-8d82d5/rate-test-model-8d82d5-kserve-66649ffff7-dndh9"]
Oct 30 13:36:22.237628 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:22.237565    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6"]
Oct 30 13:36:22.252423 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:22.252385    4364 cpu_manager.go:414] "containerMap: removing container" podUID="56e242de-da4d-4e53-b4b3-0e4ecd40ef8a" containerName="main"
Oct 30 13:36:22.252423 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:22.252422    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56e242de-da4d-4e53-b4b3-0e4ecd40ef8a" containerName="main"
Oct 30 13:36:22.252941 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:22.252919    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="56e242de-da4d-4e53-b4b3-0e4ecd40ef8a" containerName="main"
Oct 30 13:36:22.260884 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:22.260854    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6"
Oct 30 13:36:22.283661 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:22.280959    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6"]
Oct 30 13:36:22.283661 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:22.281197    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-78dc89-openshift-default-dockercfg-jhffp\""
Oct 30 13:36:22.369584 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:22.365733    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/864a2c8d-97ba-4141-957c-0810628ba6ec-istio-envoy\") pod \"e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6\" (UID: \"864a2c8d-97ba-4141-957c-0810628ba6ec\") " pod="openshift-ingress/e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6"
Oct 30 13:36:22.372064 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:22.372023    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/864a2c8d-97ba-4141-957c-0810628ba6ec-workload-certs\") pod \"e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6\" (UID: \"864a2c8d-97ba-4141-957c-0810628ba6ec\") " pod="openshift-ingress/e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6"
Oct 30 13:36:22.383848 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:22.372388    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/864a2c8d-97ba-4141-957c-0810628ba6ec-istio-podinfo\") pod \"e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6\" (UID: \"864a2c8d-97ba-4141-957c-0810628ba6ec\") " pod="openshift-ingress/e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6"
Oct 30 13:36:22.384372 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:22.384218    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdscw\" (UniqueName: \"kubernetes.io/projected/864a2c8d-97ba-4141-957c-0810628ba6ec-kube-api-access-xdscw\") pod \"e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6\" (UID: \"864a2c8d-97ba-4141-957c-0810628ba6ec\") " pod="openshift-ingress/e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6"
Oct 30 13:36:22.384787 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:22.384749    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/864a2c8d-97ba-4141-957c-0810628ba6ec-credential-socket\") pod \"e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6\" (UID: \"864a2c8d-97ba-4141-957c-0810628ba6ec\") " pod="openshift-ingress/e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6"
Oct 30 13:36:22.385108 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:22.385019    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/864a2c8d-97ba-4141-957c-0810628ba6ec-istio-data\") pod \"e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6\" (UID: \"864a2c8d-97ba-4141-957c-0810628ba6ec\") " pod="openshift-ingress/e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6"
Oct 30 13:36:22.390752 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:22.390695    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/864a2c8d-97ba-4141-957c-0810628ba6ec-istiod-ca-cert\") pod \"e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6\" (UID: \"864a2c8d-97ba-4141-957c-0810628ba6ec\") " pod="openshift-ingress/e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6"
Oct 30 13:36:22.391199 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:22.391170    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/864a2c8d-97ba-4141-957c-0810628ba6ec-workload-socket\") pod \"e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6\" (UID: \"864a2c8d-97ba-4141-957c-0810628ba6ec\") " pod="openshift-ingress/e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6"
Oct 30 13:36:22.391379 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:22.391361    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/864a2c8d-97ba-4141-957c-0810628ba6ec-istio-token\") pod \"e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6\" (UID: \"864a2c8d-97ba-4141-957c-0810628ba6ec\") " pod="openshift-ingress/e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6"
Oct 30 13:36:22.496151 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:22.496058    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/864a2c8d-97ba-4141-957c-0810628ba6ec-workload-socket\") pod \"e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6\" (UID: \"864a2c8d-97ba-4141-957c-0810628ba6ec\") " pod="openshift-ingress/e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6"
Oct 30 13:36:22.496421 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:22.496398    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/864a2c8d-97ba-4141-957c-0810628ba6ec-istio-token\") pod \"e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6\" (UID: \"864a2c8d-97ba-4141-957c-0810628ba6ec\") " pod="openshift-ingress/e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6"
Oct 30 13:36:22.496554 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:22.496540    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/864a2c8d-97ba-4141-957c-0810628ba6ec-istio-envoy\") pod \"e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6\" (UID: \"864a2c8d-97ba-4141-957c-0810628ba6ec\") " pod="openshift-ingress/e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6"
Oct 30 13:36:22.496777 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:22.496756    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/864a2c8d-97ba-4141-957c-0810628ba6ec-workload-certs\") pod \"e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6\" (UID: \"864a2c8d-97ba-4141-957c-0810628ba6ec\") " pod="openshift-ingress/e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6"
Oct 30 13:36:22.502630 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:22.496982    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/864a2c8d-97ba-4141-957c-0810628ba6ec-istio-podinfo\") pod \"e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6\" (UID: \"864a2c8d-97ba-4141-957c-0810628ba6ec\") " pod="openshift-ingress/e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6"
Oct 30 13:36:22.502863 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:22.502837    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdscw\" (UniqueName: \"kubernetes.io/projected/864a2c8d-97ba-4141-957c-0810628ba6ec-kube-api-access-xdscw\") pod \"e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6\" (UID: \"864a2c8d-97ba-4141-957c-0810628ba6ec\") " pod="openshift-ingress/e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6"
Oct 30 13:36:22.503056 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:22.503034    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/864a2c8d-97ba-4141-957c-0810628ba6ec-credential-socket\") pod \"e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6\" (UID: \"864a2c8d-97ba-4141-957c-0810628ba6ec\") " pod="openshift-ingress/e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6"
Oct 30 13:36:22.503285 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:22.503261    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/864a2c8d-97ba-4141-957c-0810628ba6ec-istio-data\") pod \"e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6\" (UID: \"864a2c8d-97ba-4141-957c-0810628ba6ec\") " pod="openshift-ingress/e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6"
Oct 30 13:36:22.503430 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:22.503413    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/864a2c8d-97ba-4141-957c-0810628ba6ec-istiod-ca-cert\") pod \"e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6\" (UID: \"864a2c8d-97ba-4141-957c-0810628ba6ec\") " pod="openshift-ingress/e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6"
Oct 30 13:36:22.516617 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:22.504656    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/864a2c8d-97ba-4141-957c-0810628ba6ec-istiod-ca-cert\") pod \"e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6\" (UID: \"864a2c8d-97ba-4141-957c-0810628ba6ec\") " pod="openshift-ingress/e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6"
Oct 30 13:36:22.517868 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:22.517837    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/864a2c8d-97ba-4141-957c-0810628ba6ec-workload-certs\") pod \"e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6\" (UID: \"864a2c8d-97ba-4141-957c-0810628ba6ec\") " pod="openshift-ingress/e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6"
Oct 30 13:36:22.534641 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:22.518265    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/864a2c8d-97ba-4141-957c-0810628ba6ec-workload-socket\") pod \"e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6\" (UID: \"864a2c8d-97ba-4141-957c-0810628ba6ec\") " pod="openshift-ingress/e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6"
Oct 30 13:36:22.534858 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:22.519922    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/864a2c8d-97ba-4141-957c-0810628ba6ec-credential-socket\") pod \"e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6\" (UID: \"864a2c8d-97ba-4141-957c-0810628ba6ec\") " pod="openshift-ingress/e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6"
Oct 30 13:36:22.534966 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:22.520174    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/864a2c8d-97ba-4141-957c-0810628ba6ec-istio-data\") pod \"e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6\" (UID: \"864a2c8d-97ba-4141-957c-0810628ba6ec\") " pod="openshift-ingress/e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6"
Oct 30 13:36:22.535060 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:22.534278    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/864a2c8d-97ba-4141-957c-0810628ba6ec-istio-podinfo\") pod \"e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6\" (UID: \"864a2c8d-97ba-4141-957c-0810628ba6ec\") " pod="openshift-ingress/e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6"
Oct 30 13:36:22.535152 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:22.534549    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/864a2c8d-97ba-4141-957c-0810628ba6ec-istio-envoy\") pod \"e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6\" (UID: \"864a2c8d-97ba-4141-957c-0810628ba6ec\") " pod="openshift-ingress/e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6"
Oct 30 13:36:22.577520 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:22.577392    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdscw\" (UniqueName: \"kubernetes.io/projected/864a2c8d-97ba-4141-957c-0810628ba6ec-kube-api-access-xdscw\") pod \"e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6\" (UID: \"864a2c8d-97ba-4141-957c-0810628ba6ec\") " pod="openshift-ingress/e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6"
Oct 30 13:36:22.615367 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:22.615331    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/864a2c8d-97ba-4141-957c-0810628ba6ec-istio-token\") pod \"e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6\" (UID: \"864a2c8d-97ba-4141-957c-0810628ba6ec\") " pod="openshift-ingress/e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6"
Oct 30 13:36:22.885869 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:22.885840    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6"
Oct 30 13:36:23.333640 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:23.333610    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6"]
Oct 30 13:36:23.334977 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:36:23.334949    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod864a2c8d_97ba_4141_957c_0810628ba6ec.slice/crio-13dd9e002f40ad251dc8ce918b872fb578a03ef0879a1394f6e4c1c337de20ca WatchSource:0}: Error finding container 13dd9e002f40ad251dc8ce918b872fb578a03ef0879a1394f6e4c1c337de20ca: Status 404 returned error can't find the container with id 13dd9e002f40ad251dc8ce918b872fb578a03ef0879a1394f6e4c1c337de20ca
Oct 30 13:36:23.337956 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:23.337926    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:36:23.338038 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:23.337996    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:36:23.338038 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:23.338024    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:36:23.628198 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:23.628153    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6" event={"ID":"864a2c8d-97ba-4141-957c-0810628ba6ec","Type":"ContainerStarted","Data":"385f1fb76fa6f0e60e13865dfc171e43478952e68c76add876ed08e5bbdfe068"}
Oct 30 13:36:23.628342 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:23.628218    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6" event={"ID":"864a2c8d-97ba-4141-957c-0810628ba6ec","Type":"ContainerStarted","Data":"13dd9e002f40ad251dc8ce918b872fb578a03ef0879a1394f6e4c1c337de20ca"}
Oct 30 13:36:23.675450 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:23.675400    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6" podStartSLOduration=1.6753854609999999 podStartE2EDuration="1.675385461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:36:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:36:23.672416473 +0000 UTC m=+2682.838781120" watchObservedRunningTime="2026-10-30 13:36:23.675385461 +0000 UTC m=+2682.841750108"
Oct 30 13:36:23.887069 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:23.886827    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6"
Oct 30 13:36:23.978200 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:23.978168    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b29df4-857fc5bcc9-lw9kb"
Oct 30 13:36:24.084695 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:24.084662    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49bf7b08-c971-4722-af65-b145ed2b1f21-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49bf7b08-c971-4722-af65-b145ed2b1f21-plugins-config-volume\") pod \"49bf7b08-c971-4722-af65-b145ed2b1f21\" (UID: \"49bf7b08-c971-4722-af65-b145ed2b1f21\") "
Oct 30 13:36:24.084886 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:24.084771    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49bf7b08-c971-4722-af65-b145ed2b1f21-kube-api-access-4wl4q\" (UniqueName: \"kubernetes.io/projected/49bf7b08-c971-4722-af65-b145ed2b1f21-kube-api-access-4wl4q\") pod \"49bf7b08-c971-4722-af65-b145ed2b1f21\" (UID: \"49bf7b08-c971-4722-af65-b145ed2b1f21\") "
Oct 30 13:36:24.085097 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:24.085073    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49bf7b08-c971-4722-af65-b145ed2b1f21-plugins-config-volume" pod "49bf7b08-c971-4722-af65-b145ed2b1f21" (UID: "49bf7b08-c971-4722-af65-b145ed2b1f21"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:36:24.085487 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:24.085465    4364 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49bf7b08-c971-4722-af65-b145ed2b1f21-plugins-config-volume\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:36:24.086954 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:24.086924    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49bf7b08-c971-4722-af65-b145ed2b1f21-kube-api-access-4wl4q" pod "49bf7b08-c971-4722-af65-b145ed2b1f21" (UID: "49bf7b08-c971-4722-af65-b145ed2b1f21"). InnerVolumeSpecName "kube-api-access-4wl4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:36:24.186770 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:24.186701    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wl4q\" (UniqueName: \"kubernetes.io/projected/49bf7b08-c971-4722-af65-b145ed2b1f21-kube-api-access-4wl4q\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:36:24.442058 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:24.441986    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf3c79-7846779cd-pmvpz"
Oct 30 13:36:24.638273 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:24.638238    4364 generic.go:356] "Generic (PLEG): container finished" podID="49bf7b08-c971-4722-af65-b145ed2b1f21" containerID="0c9916c6065a29944e8de212bf1390499e8b338060c0e5ed28a7d002b3b3270d" exitCode=0
Oct 30 13:36:24.638448 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:24.638293    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b29df4-857fc5bcc9-lw9kb"
Oct 30 13:36:24.638448 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:24.638319    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b29df4-857fc5bcc9-lw9kb" event={"ID":"49bf7b08-c971-4722-af65-b145ed2b1f21","Type":"ContainerDied","Data":"0c9916c6065a29944e8de212bf1390499e8b338060c0e5ed28a7d002b3b3270d"}
Oct 30 13:36:24.638448 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:24.638384    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b29df4-857fc5bcc9-lw9kb" event={"ID":"49bf7b08-c971-4722-af65-b145ed2b1f21","Type":"ContainerDied","Data":"2a75e15f6b5c18b5ab49d7fe7f5bc8831cd8bdbdfa0431baac0e70ab59f0b36c"}
Oct 30 13:36:24.638448 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:24.638425    4364 scope.go:122] "RemoveContainer" containerID="0c9916c6065a29944e8de212bf1390499e8b338060c0e5ed28a7d002b3b3270d"
Oct 30 13:36:24.652556 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:24.652542    4364 scope.go:122] "RemoveContainer" containerID="0c9916c6065a29944e8de212bf1390499e8b338060c0e5ed28a7d002b3b3270d"
Oct 30 13:36:24.652859 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:36:24.652839    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c9916c6065a29944e8de212bf1390499e8b338060c0e5ed28a7d002b3b3270d\": container with ID starting with 0c9916c6065a29944e8de212bf1390499e8b338060c0e5ed28a7d002b3b3270d not found: ID does not exist" containerID="0c9916c6065a29944e8de212bf1390499e8b338060c0e5ed28a7d002b3b3270d"
Oct 30 13:36:24.652920 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:24.652864    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c9916c6065a29944e8de212bf1390499e8b338060c0e5ed28a7d002b3b3270d"} err="failed to get container status \"0c9916c6065a29944e8de212bf1390499e8b338060c0e5ed28a7d002b3b3270d\": rpc error: code = NotFound desc = could not find container \"0c9916c6065a29944e8de212bf1390499e8b338060c0e5ed28a7d002b3b3270d\": container with ID starting with 0c9916c6065a29944e8de212bf1390499e8b338060c0e5ed28a7d002b3b3270d not found: ID does not exist"
Oct 30 13:36:24.701202 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:24.701137    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b29df4-857fc5bcc9-lw9kb"]
Oct 30 13:36:24.705900 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:24.705879    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b29df4-857fc5bcc9-lw9kb"]
Oct 30 13:36:24.936810 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:24.935228    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6"
Oct 30 13:36:24.983240 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:24.983153    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49bf7b08-c971-4722-af65-b145ed2b1f21" path="/var/lib/kubelet/pods/49bf7b08-c971-4722-af65-b145ed2b1f21/volumes"
Oct 30 13:36:25.650167 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:25.650139    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6"
Oct 30 13:36:25.650988 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:25.650969    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6"
Oct 30 13:36:28.638396 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:28.638360    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-78dc89-7fbb976874-xzkzv"]
Oct 30 13:36:28.639241 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:28.639222    4364 cpu_manager.go:414] "containerMap: removing container" podUID="49bf7b08-c971-4722-af65-b145ed2b1f21" containerName="payload-pre-processing"
Oct 30 13:36:28.639241 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:28.639241    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49bf7b08-c971-4722-af65-b145ed2b1f21" containerName="payload-pre-processing"
Oct 30 13:36:28.639427 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:28.639415    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="49bf7b08-c971-4722-af65-b145ed2b1f21" containerName="payload-pre-processing"
Oct 30 13:36:28.644905 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:28.644886    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-78dc89-7fbb976874-xzkzv"
Oct 30 13:36:28.667455 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:28.667416    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-78dc89\""
Oct 30 13:36:28.672447 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:28.672418    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-78dc89-7fbb976874-xzkzv"]
Oct 30 13:36:28.678195 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:28.678172    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-78dc89-6bd5777667-5tn7b"]
Oct 30 13:36:28.683626 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:28.683576    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78dc89-6bd5777667-5tn7b"
Oct 30 13:36:28.736620 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:28.736279    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-78dc89-6bd5777667-5tn7b"]
Oct 30 13:36:28.786635 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:28.780929    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-78dc89-dockercfg-b2smn\""
Oct 30 13:36:28.843686 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:28.840279    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvqg8\" (UniqueName: \"kubernetes.io/projected/cb725a71-8f08-4c47-a8a6-79242757e6b2-kube-api-access-mvqg8\") pod \"payload-pre-processing-e2e-ipp-a-78dc89-6bd5777667-5tn7b\" (UID: \"cb725a71-8f08-4c47-a8a6-79242757e6b2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78dc89-6bd5777667-5tn7b"
Oct 30 13:36:28.843686 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:28.840402    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4a7e4ba-b7e7-4f0f-aae8-460156351389-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-78dc89-7fbb976874-xzkzv\" (UID: \"b4a7e4ba-b7e7-4f0f-aae8-460156351389\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-78dc89-7fbb976874-xzkzv"
Oct 30 13:36:28.843686 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:28.840473    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zntpr\" (UniqueName: \"kubernetes.io/projected/b4a7e4ba-b7e7-4f0f-aae8-460156351389-kube-api-access-zntpr\") pod \"payload-processing-e2e-ipp-a-78dc89-7fbb976874-xzkzv\" (UID: \"b4a7e4ba-b7e7-4f0f-aae8-460156351389\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-78dc89-7fbb976874-xzkzv"
Oct 30 13:36:28.843686 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:28.840660    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb725a71-8f08-4c47-a8a6-79242757e6b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-78dc89-6bd5777667-5tn7b\" (UID: \"cb725a71-8f08-4c47-a8a6-79242757e6b2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78dc89-6bd5777667-5tn7b"
Oct 30 13:36:29.064669 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:29.063798    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvqg8\" (UniqueName: \"kubernetes.io/projected/cb725a71-8f08-4c47-a8a6-79242757e6b2-kube-api-access-mvqg8\") pod \"payload-pre-processing-e2e-ipp-a-78dc89-6bd5777667-5tn7b\" (UID: \"cb725a71-8f08-4c47-a8a6-79242757e6b2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78dc89-6bd5777667-5tn7b"
Oct 30 13:36:29.064669 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:29.063956    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4a7e4ba-b7e7-4f0f-aae8-460156351389-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-78dc89-7fbb976874-xzkzv\" (UID: \"b4a7e4ba-b7e7-4f0f-aae8-460156351389\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-78dc89-7fbb976874-xzkzv"
Oct 30 13:36:29.064669 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:29.064026    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zntpr\" (UniqueName: \"kubernetes.io/projected/b4a7e4ba-b7e7-4f0f-aae8-460156351389-kube-api-access-zntpr\") pod \"payload-processing-e2e-ipp-a-78dc89-7fbb976874-xzkzv\" (UID: \"b4a7e4ba-b7e7-4f0f-aae8-460156351389\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-78dc89-7fbb976874-xzkzv"
Oct 30 13:36:29.064669 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:29.064196    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb725a71-8f08-4c47-a8a6-79242757e6b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-78dc89-6bd5777667-5tn7b\" (UID: \"cb725a71-8f08-4c47-a8a6-79242757e6b2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78dc89-6bd5777667-5tn7b"
Oct 30 13:36:29.069665 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:29.065772    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb725a71-8f08-4c47-a8a6-79242757e6b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-78dc89-6bd5777667-5tn7b\" (UID: \"cb725a71-8f08-4c47-a8a6-79242757e6b2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78dc89-6bd5777667-5tn7b"
Oct 30 13:36:29.069665 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:29.066999    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4a7e4ba-b7e7-4f0f-aae8-460156351389-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-78dc89-7fbb976874-xzkzv\" (UID: \"b4a7e4ba-b7e7-4f0f-aae8-460156351389\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-78dc89-7fbb976874-xzkzv"
Oct 30 13:36:29.262121 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:29.255798    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvqg8\" (UniqueName: \"kubernetes.io/projected/cb725a71-8f08-4c47-a8a6-79242757e6b2-kube-api-access-mvqg8\") pod \"payload-pre-processing-e2e-ipp-a-78dc89-6bd5777667-5tn7b\" (UID: \"cb725a71-8f08-4c47-a8a6-79242757e6b2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78dc89-6bd5777667-5tn7b"
Oct 30 13:36:29.262121 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:29.259833    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zntpr\" (UniqueName: \"kubernetes.io/projected/b4a7e4ba-b7e7-4f0f-aae8-460156351389-kube-api-access-zntpr\") pod \"payload-processing-e2e-ipp-a-78dc89-7fbb976874-xzkzv\" (UID: \"b4a7e4ba-b7e7-4f0f-aae8-460156351389\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-78dc89-7fbb976874-xzkzv"
Oct 30 13:36:29.262121 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:29.261203    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-78dc89-7fbb976874-xzkzv"
Oct 30 13:36:29.301505 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:29.301474    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78dc89-6bd5777667-5tn7b"
Oct 30 13:36:29.610202 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:36:29.610164    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4a7e4ba_b7e7_4f0f_aae8_460156351389.slice/crio-a80f38cdd3de534a07ddb4d0e92aa970dd418526b2b5cc415b8f15f16572cd2a WatchSource:0}: Error finding container a80f38cdd3de534a07ddb4d0e92aa970dd418526b2b5cc415b8f15f16572cd2a: Status 404 returned error can't find the container with id a80f38cdd3de534a07ddb4d0e92aa970dd418526b2b5cc415b8f15f16572cd2a
Oct 30 13:36:29.611471 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:29.611346    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-78dc89-7fbb976874-xzkzv"]
Oct 30 13:36:29.629468 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:29.629438    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-78dc89-6bd5777667-5tn7b"]
Oct 30 13:36:29.630755 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:36:29.630728    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb725a71_8f08_4c47_a8a6_79242757e6b2.slice/crio-7a1e13a9bfa81897c3c3c16dba874463897e74bca7d8f0056f0b922517ad9510 WatchSource:0}: Error finding container 7a1e13a9bfa81897c3c3c16dba874463897e74bca7d8f0056f0b922517ad9510: Status 404 returned error can't find the container with id 7a1e13a9bfa81897c3c3c16dba874463897e74bca7d8f0056f0b922517ad9510
Oct 30 13:36:29.736648 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:29.736593    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78dc89-6bd5777667-5tn7b" event={"ID":"cb725a71-8f08-4c47-a8a6-79242757e6b2","Type":"ContainerStarted","Data":"f9aaae724c4f9dee9cce90e7ab20abd99bcf64f90d336e5eecd95253f3a1a777"}
Oct 30 13:36:29.737045 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:29.736683    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78dc89-6bd5777667-5tn7b" event={"ID":"cb725a71-8f08-4c47-a8a6-79242757e6b2","Type":"ContainerStarted","Data":"7a1e13a9bfa81897c3c3c16dba874463897e74bca7d8f0056f0b922517ad9510"}
Oct 30 13:36:29.737045 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:29.736866    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78dc89-6bd5777667-5tn7b"
Oct 30 13:36:29.738445 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:29.738422    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-78dc89-7fbb976874-xzkzv" event={"ID":"b4a7e4ba-b7e7-4f0f-aae8-460156351389","Type":"ContainerStarted","Data":"1e86e262567304c06b16e7a0fa1f3ce2cae9a940e3226d01876629f404c00c5f"}
Oct 30 13:36:29.738565 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:29.738462    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-78dc89-7fbb976874-xzkzv" event={"ID":"b4a7e4ba-b7e7-4f0f-aae8-460156351389","Type":"ContainerStarted","Data":"a80f38cdd3de534a07ddb4d0e92aa970dd418526b2b5cc415b8f15f16572cd2a"}
Oct 30 13:36:29.738565 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:29.738482    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-78dc89-7fbb976874-xzkzv"
Oct 30 13:36:29.777970 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:29.777921    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78dc89-6bd5777667-5tn7b" podStartSLOduration=1.7779050619999999 podStartE2EDuration="1.777905062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:36:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:36:29.776443812 +0000 UTC m=+2688.942808460" watchObservedRunningTime="2026-10-30 13:36:29.777905062 +0000 UTC m=+2688.944269714"
Oct 30 13:36:29.823743 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:29.823693    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-78dc89-7fbb976874-xzkzv" podStartSLOduration=1.823678337 podStartE2EDuration="1.823678337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:36:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:36:29.81768313 +0000 UTC m=+2688.984047781" watchObservedRunningTime="2026-10-30 13:36:29.823678337 +0000 UTC m=+2688.990042985"
Oct 30 13:36:31.015377 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:31.015342    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-428a0e-65684c5fd4-vgkz2"]
Oct 30 13:36:32.129986 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:32.129948    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b4fb25-5596cbb8f4-rcktl"]
Oct 30 13:36:36.016424 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:36.016375    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-428a0e-65684c5fd4-vgkz2" podUID="668e3034-e72e-4e25-ab29-d76303e341ac" containerName="payload-processing" containerID="cri-o://85acf5387a434bbbbc9aa90081ae0f03c074268d1a3a3869517b5fd5ece9ad71" gracePeriod=25
Oct 30 13:36:36.193922 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:36.193876    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-8d82d5/rate-test-model-8d82d5-kserve-66649ffff7-dndh9" podUID="d8781a4e-4171-4db3-808e-a8f412894661" containerName="main" containerID="cri-o://fa9376c2c207f9d058327167d9032e0fbc89d3cedc2bba9d3f938697976d6fd4" gracePeriod=45
Oct 30 13:36:36.338297 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:36.338262    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-428a0e-65684c5fd4-vgkz2"
Oct 30 13:36:36.479849 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:36.479823    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-8d82d5/rate-test-model-8d82d5-kserve-66649ffff7-dndh9"
Oct 30 13:36:36.493314 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:36.493290    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/668e3034-e72e-4e25-ab29-d76303e341ac-kube-api-access-5sxs8\" (UniqueName: \"kubernetes.io/projected/668e3034-e72e-4e25-ab29-d76303e341ac-kube-api-access-5sxs8\") pod \"668e3034-e72e-4e25-ab29-d76303e341ac\" (UID: \"668e3034-e72e-4e25-ab29-d76303e341ac\") "
Oct 30 13:36:36.493526 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:36.493512    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/668e3034-e72e-4e25-ab29-d76303e341ac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/668e3034-e72e-4e25-ab29-d76303e341ac-plugins-config-volume\") pod \"668e3034-e72e-4e25-ab29-d76303e341ac\" (UID: \"668e3034-e72e-4e25-ab29-d76303e341ac\") "
Oct 30 13:36:36.493867 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:36.493841    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/668e3034-e72e-4e25-ab29-d76303e341ac-plugins-config-volume" pod "668e3034-e72e-4e25-ab29-d76303e341ac" (UID: "668e3034-e72e-4e25-ab29-d76303e341ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:36:36.494180 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:36.494151    4364 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/668e3034-e72e-4e25-ab29-d76303e341ac-plugins-config-volume\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:36:36.495415 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:36.495392    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/668e3034-e72e-4e25-ab29-d76303e341ac-kube-api-access-5sxs8" pod "668e3034-e72e-4e25-ab29-d76303e341ac" (UID: "668e3034-e72e-4e25-ab29-d76303e341ac"). InnerVolumeSpecName "kube-api-access-5sxs8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:36:36.595441 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:36.595353    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8781a4e-4171-4db3-808e-a8f412894661-kube-api-access-ws6l2\" (UniqueName: \"kubernetes.io/projected/d8781a4e-4171-4db3-808e-a8f412894661-kube-api-access-ws6l2\") pod \"d8781a4e-4171-4db3-808e-a8f412894661\" (UID: \"d8781a4e-4171-4db3-808e-a8f412894661\") "
Oct 30 13:36:36.595633 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:36.595590    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8781a4e-4171-4db3-808e-a8f412894661-tls-certs\" (UniqueName: \"kubernetes.io/secret/d8781a4e-4171-4db3-808e-a8f412894661-tls-certs\") pod \"d8781a4e-4171-4db3-808e-a8f412894661\" (UID: \"d8781a4e-4171-4db3-808e-a8f412894661\") "
Oct 30 13:36:36.595722 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:36.595689    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8781a4e-4171-4db3-808e-a8f412894661-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8781a4e-4171-4db3-808e-a8f412894661-tmp-dir\") pod \"d8781a4e-4171-4db3-808e-a8f412894661\" (UID: \"d8781a4e-4171-4db3-808e-a8f412894661\") "
Oct 30 13:36:36.595803 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:36.595730    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8781a4e-4171-4db3-808e-a8f412894661-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8781a4e-4171-4db3-808e-a8f412894661-dshm\") pod \"d8781a4e-4171-4db3-808e-a8f412894661\" (UID: \"d8781a4e-4171-4db3-808e-a8f412894661\") "
Oct 30 13:36:36.595803 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:36.595783    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8781a4e-4171-4db3-808e-a8f412894661-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8781a4e-4171-4db3-808e-a8f412894661-model-cache\") pod \"d8781a4e-4171-4db3-808e-a8f412894661\" (UID: \"d8781a4e-4171-4db3-808e-a8f412894661\") "
Oct 30 13:36:36.595950 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:36.595821    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8781a4e-4171-4db3-808e-a8f412894661-home\" (UniqueName: \"kubernetes.io/empty-dir/d8781a4e-4171-4db3-808e-a8f412894661-home\") pod \"d8781a4e-4171-4db3-808e-a8f412894661\" (UID: \"d8781a4e-4171-4db3-808e-a8f412894661\") "
Oct 30 13:36:36.596010 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:36.595977    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8781a4e-4171-4db3-808e-a8f412894661-tmp-dir" pod "d8781a4e-4171-4db3-808e-a8f412894661" (UID: "d8781a4e-4171-4db3-808e-a8f412894661"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:36:36.596199 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:36.596170    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8781a4e-4171-4db3-808e-a8f412894661-model-cache" pod "d8781a4e-4171-4db3-808e-a8f412894661" (UID: "d8781a4e-4171-4db3-808e-a8f412894661"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:36:36.596315 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:36.596286    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8781a4e-4171-4db3-808e-a8f412894661-home" pod "d8781a4e-4171-4db3-808e-a8f412894661" (UID: "d8781a4e-4171-4db3-808e-a8f412894661"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:36:36.596880 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:36.596854    4364 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8781a4e-4171-4db3-808e-a8f412894661-tmp-dir\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:36:36.596983 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:36.596887    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sxs8\" (UniqueName: \"kubernetes.io/projected/668e3034-e72e-4e25-ab29-d76303e341ac-kube-api-access-5sxs8\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:36:36.596983 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:36.596904    4364 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8781a4e-4171-4db3-808e-a8f412894661-model-cache\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:36:36.596983 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:36.596917    4364 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8781a4e-4171-4db3-808e-a8f412894661-home\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:36:36.597683 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:36.597661    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8781a4e-4171-4db3-808e-a8f412894661-kube-api-access-ws6l2" pod "d8781a4e-4171-4db3-808e-a8f412894661" (UID: "d8781a4e-4171-4db3-808e-a8f412894661"). InnerVolumeSpecName "kube-api-access-ws6l2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:36:36.597869 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:36.597853    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8781a4e-4171-4db3-808e-a8f412894661-dshm" pod "d8781a4e-4171-4db3-808e-a8f412894661" (UID: "d8781a4e-4171-4db3-808e-a8f412894661"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:36:36.597925 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:36.597867    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8781a4e-4171-4db3-808e-a8f412894661-tls-certs" pod "d8781a4e-4171-4db3-808e-a8f412894661" (UID: "d8781a4e-4171-4db3-808e-a8f412894661"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:36:36.698725 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:36.698683    4364 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8781a4e-4171-4db3-808e-a8f412894661-tls-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:36:36.698725 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:36.698716    4364 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8781a4e-4171-4db3-808e-a8f412894661-dshm\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:36:36.698725 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:36.698729    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ws6l2\" (UniqueName: \"kubernetes.io/projected/d8781a4e-4171-4db3-808e-a8f412894661-kube-api-access-ws6l2\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:36:36.984441 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:36.984408    4364 generic.go:356] "Generic (PLEG): container finished" podID="d8781a4e-4171-4db3-808e-a8f412894661" containerID="fa9376c2c207f9d058327167d9032e0fbc89d3cedc2bba9d3f938697976d6fd4" exitCode=0
Oct 30 13:36:36.984657 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:36.984530    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-8d82d5/rate-test-model-8d82d5-kserve-66649ffff7-dndh9"
Oct 30 13:36:36.986357 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:36.986333    4364 generic.go:356] "Generic (PLEG): container finished" podID="668e3034-e72e-4e25-ab29-d76303e341ac" containerID="85acf5387a434bbbbc9aa90081ae0f03c074268d1a3a3869517b5fd5ece9ad71" exitCode=0
Oct 30 13:36:36.986520 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:36.986509    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-428a0e-65684c5fd4-vgkz2"
Oct 30 13:36:36.993252 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:36.993222    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-8d82d5/rate-test-model-8d82d5-kserve-66649ffff7-dndh9" event={"ID":"d8781a4e-4171-4db3-808e-a8f412894661","Type":"ContainerDied","Data":"fa9376c2c207f9d058327167d9032e0fbc89d3cedc2bba9d3f938697976d6fd4"}
Oct 30 13:36:36.993401 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:36.993266    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-8d82d5/rate-test-model-8d82d5-kserve-66649ffff7-dndh9" event={"ID":"d8781a4e-4171-4db3-808e-a8f412894661","Type":"ContainerDied","Data":"f35a970d39d82bc67cc54d320d589dd2f852c127983d82580ad636670b368ebe"}
Oct 30 13:36:36.993401 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:36.993282    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-428a0e-65684c5fd4-vgkz2" event={"ID":"668e3034-e72e-4e25-ab29-d76303e341ac","Type":"ContainerDied","Data":"85acf5387a434bbbbc9aa90081ae0f03c074268d1a3a3869517b5fd5ece9ad71"}
Oct 30 13:36:36.993401 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:36.993307    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-428a0e-65684c5fd4-vgkz2" event={"ID":"668e3034-e72e-4e25-ab29-d76303e341ac","Type":"ContainerDied","Data":"c3041c38be269d05f8a3e921473b937235c43804da974bab7f877ad6d0905280"}
Oct 30 13:36:36.993401 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:36.993333    4364 scope.go:122] "RemoveContainer" containerID="fa9376c2c207f9d058327167d9032e0fbc89d3cedc2bba9d3f938697976d6fd4"
Oct 30 13:36:37.008294 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:37.008271    4364 scope.go:122] "RemoveContainer" containerID="fa9376c2c207f9d058327167d9032e0fbc89d3cedc2bba9d3f938697976d6fd4"
Oct 30 13:36:37.008651 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:36:37.008618    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa9376c2c207f9d058327167d9032e0fbc89d3cedc2bba9d3f938697976d6fd4\": container with ID starting with fa9376c2c207f9d058327167d9032e0fbc89d3cedc2bba9d3f938697976d6fd4 not found: ID does not exist" containerID="fa9376c2c207f9d058327167d9032e0fbc89d3cedc2bba9d3f938697976d6fd4"
Oct 30 13:36:37.008754 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:37.008657    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa9376c2c207f9d058327167d9032e0fbc89d3cedc2bba9d3f938697976d6fd4"} err="failed to get container status \"fa9376c2c207f9d058327167d9032e0fbc89d3cedc2bba9d3f938697976d6fd4\": rpc error: code = NotFound desc = could not find container \"fa9376c2c207f9d058327167d9032e0fbc89d3cedc2bba9d3f938697976d6fd4\": container with ID starting with fa9376c2c207f9d058327167d9032e0fbc89d3cedc2bba9d3f938697976d6fd4 not found: ID does not exist"
Oct 30 13:36:37.008754 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:37.008676    4364 scope.go:122] "RemoveContainer" containerID="85acf5387a434bbbbc9aa90081ae0f03c074268d1a3a3869517b5fd5ece9ad71"
Oct 30 13:36:37.019507 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:37.019362    4364 scope.go:122] "RemoveContainer" containerID="85acf5387a434bbbbc9aa90081ae0f03c074268d1a3a3869517b5fd5ece9ad71"
Oct 30 13:36:37.019842 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:36:37.019725    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85acf5387a434bbbbc9aa90081ae0f03c074268d1a3a3869517b5fd5ece9ad71\": container with ID starting with 85acf5387a434bbbbc9aa90081ae0f03c074268d1a3a3869517b5fd5ece9ad71 not found: ID does not exist" containerID="85acf5387a434bbbbc9aa90081ae0f03c074268d1a3a3869517b5fd5ece9ad71"
Oct 30 13:36:37.019842 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:37.019763    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85acf5387a434bbbbc9aa90081ae0f03c074268d1a3a3869517b5fd5ece9ad71"} err="failed to get container status \"85acf5387a434bbbbc9aa90081ae0f03c074268d1a3a3869517b5fd5ece9ad71\": rpc error: code = NotFound desc = could not find container \"85acf5387a434bbbbc9aa90081ae0f03c074268d1a3a3869517b5fd5ece9ad71\": container with ID starting with 85acf5387a434bbbbc9aa90081ae0f03c074268d1a3a3869517b5fd5ece9ad71 not found: ID does not exist"
Oct 30 13:36:37.056538 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:37.056505    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-428a0e-65684c5fd4-vgkz2"]
Oct 30 13:36:37.069724 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:37.069687    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-428a0e-65684c5fd4-vgkz2"]
Oct 30 13:36:37.098152 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:37.098120    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-8d82d5/rate-test-model-8d82d5-kserve-66649ffff7-dndh9"]
Oct 30 13:36:37.105850 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:37.105824    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-8d82d5/rate-test-model-8d82d5-kserve-66649ffff7-dndh9"]
Oct 30 13:36:37.131176 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:37.131139    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-b4fb25-5596cbb8f4-rcktl" podUID="c0beff44-3603-4a88-bc25-78f6e3fbbddf" containerName="payload-processing" containerID="cri-o://16ab27ab7fbec08ea81d7c487418e220711f88ce0cb3943fe05033e304461cab" gracePeriod=25
Oct 30 13:36:37.440463 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:37.440431    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b4fb25-5596cbb8f4-rcktl"
Oct 30 13:36:37.519187 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:37.519096    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0beff44-3603-4a88-bc25-78f6e3fbbddf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0beff44-3603-4a88-bc25-78f6e3fbbddf-plugins-config-volume\") pod \"c0beff44-3603-4a88-bc25-78f6e3fbbddf\" (UID: \"c0beff44-3603-4a88-bc25-78f6e3fbbddf\") "
Oct 30 13:36:37.519367 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:37.519249    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0beff44-3603-4a88-bc25-78f6e3fbbddf-kube-api-access-wkx5l\" (UniqueName: \"kubernetes.io/projected/c0beff44-3603-4a88-bc25-78f6e3fbbddf-kube-api-access-wkx5l\") pod \"c0beff44-3603-4a88-bc25-78f6e3fbbddf\" (UID: \"c0beff44-3603-4a88-bc25-78f6e3fbbddf\") "
Oct 30 13:36:37.520498 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:37.520460    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0beff44-3603-4a88-bc25-78f6e3fbbddf-plugins-config-volume" pod "c0beff44-3603-4a88-bc25-78f6e3fbbddf" (UID: "c0beff44-3603-4a88-bc25-78f6e3fbbddf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:36:37.522823 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:37.522795    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0beff44-3603-4a88-bc25-78f6e3fbbddf-kube-api-access-wkx5l" pod "c0beff44-3603-4a88-bc25-78f6e3fbbddf" (UID: "c0beff44-3603-4a88-bc25-78f6e3fbbddf"). InnerVolumeSpecName "kube-api-access-wkx5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:36:37.523542 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:37.523513    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkx5l\" (UniqueName: \"kubernetes.io/projected/c0beff44-3603-4a88-bc25-78f6e3fbbddf-kube-api-access-wkx5l\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:36:37.523625 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:37.523544    4364 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0beff44-3603-4a88-bc25-78f6e3fbbddf-plugins-config-volume\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:36:37.997347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:37.997311    4364 generic.go:356] "Generic (PLEG): container finished" podID="c0beff44-3603-4a88-bc25-78f6e3fbbddf" containerID="16ab27ab7fbec08ea81d7c487418e220711f88ce0cb3943fe05033e304461cab" exitCode=0
Oct 30 13:36:37.997522 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:37.997372    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b4fb25-5596cbb8f4-rcktl"
Oct 30 13:36:37.997522 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:37.997395    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b4fb25-5596cbb8f4-rcktl" event={"ID":"c0beff44-3603-4a88-bc25-78f6e3fbbddf","Type":"ContainerDied","Data":"16ab27ab7fbec08ea81d7c487418e220711f88ce0cb3943fe05033e304461cab"}
Oct 30 13:36:37.997522 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:37.997449    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b4fb25-5596cbb8f4-rcktl" event={"ID":"c0beff44-3603-4a88-bc25-78f6e3fbbddf","Type":"ContainerDied","Data":"e411bde0eb1ab8bc8a6689ddf3dda13cb9aae5a845486129df7b40e4b2b55d1b"}
Oct 30 13:36:37.997522 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:37.997472    4364 scope.go:122] "RemoveContainer" containerID="16ab27ab7fbec08ea81d7c487418e220711f88ce0cb3943fe05033e304461cab"
Oct 30 13:36:38.011377 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:38.011352    4364 scope.go:122] "RemoveContainer" containerID="16ab27ab7fbec08ea81d7c487418e220711f88ce0cb3943fe05033e304461cab"
Oct 30 13:36:38.011730 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:36:38.011707    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16ab27ab7fbec08ea81d7c487418e220711f88ce0cb3943fe05033e304461cab\": container with ID starting with 16ab27ab7fbec08ea81d7c487418e220711f88ce0cb3943fe05033e304461cab not found: ID does not exist" containerID="16ab27ab7fbec08ea81d7c487418e220711f88ce0cb3943fe05033e304461cab"
Oct 30 13:36:38.011810 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:38.011735    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16ab27ab7fbec08ea81d7c487418e220711f88ce0cb3943fe05033e304461cab"} err="failed to get container status \"16ab27ab7fbec08ea81d7c487418e220711f88ce0cb3943fe05033e304461cab\": rpc error: code = NotFound desc = could not find container \"16ab27ab7fbec08ea81d7c487418e220711f88ce0cb3943fe05033e304461cab\": container with ID starting with 16ab27ab7fbec08ea81d7c487418e220711f88ce0cb3943fe05033e304461cab not found: ID does not exist"
Oct 30 13:36:38.055087 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:38.055049    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b4fb25-5596cbb8f4-rcktl"]
Oct 30 13:36:38.066218 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:38.066191    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b4fb25-5596cbb8f4-rcktl"]
Oct 30 13:36:38.972681 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:38.972646    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="668e3034-e72e-4e25-ab29-d76303e341ac" path="/var/lib/kubelet/pods/668e3034-e72e-4e25-ab29-d76303e341ac/volumes"
Oct 30 13:36:38.973187 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:38.973172    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0beff44-3603-4a88-bc25-78f6e3fbbddf" path="/var/lib/kubelet/pods/c0beff44-3603-4a88-bc25-78f6e3fbbddf/volumes"
Oct 30 13:36:38.973719 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:38.973703    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8781a4e-4171-4db3-808e-a8f412894661" path="/var/lib/kubelet/pods/d8781a4e-4171-4db3-808e-a8f412894661/volumes"
Oct 30 13:36:40.017582 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:40.017511    4364 generic.go:356] "Generic (PLEG): container finished" podID="5ce05a81-71f3-4dab-96e1-44c9ea628a7b" containerID="67063f13ec7a249a230bf997909730d9f8e56e8c0bde3ab950063eef0a4ac3b5" exitCode=0
Oct 30 13:36:40.018008 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:40.017586    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg" event={"ID":"5ce05a81-71f3-4dab-96e1-44c9ea628a7b","Type":"ContainerDied","Data":"67063f13ec7a249a230bf997909730d9f8e56e8c0bde3ab950063eef0a4ac3b5"}
Oct 30 13:36:40.018008 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:40.017997    4364 scope.go:122] "RemoveContainer" containerID="67063f13ec7a249a230bf997909730d9f8e56e8c0bde3ab950063eef0a4ac3b5"
Oct 30 13:36:40.019773 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:40.019748    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:36:40.019855 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:40.019814    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:36:40.019855 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:40.019847    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:36:40.788624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:40.787873    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78dc89-6bd5777667-5tn7b"
Oct 30 13:36:40.820625 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:40.819792    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-78dc89-7fbb976874-xzkzv"
Oct 30 13:36:41.778410 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:41.778362    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6" podUID="b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 13:36:41.908624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:41.897642    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc" podUID="8911b790-331d-490c-876b-647e6d2fcd14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:36:42.172433 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:36:42.172385    4364 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.21s"
Oct 30 13:36:42.260163 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:36:42.260119    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:36:42.261467 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:42.261431    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg" event={"ID":"5ce05a81-71f3-4dab-96e1-44c9ea628a7b","Type":"ContainerStarted","Data":"b3eb44114eb6eccd60b6eba100561e0a8be2fce1da4d1d51712da581e596d26b"}
Oct 30 13:36:42.486717 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:42.484895    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg"
Oct 30 13:36:43.050224 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:43.050193    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg"
Oct 30 13:36:43.051495 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:43.051469    4364 patch_prober.go:29] interesting pod/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body=
Oct 30 13:36:43.051662 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:43.051523    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg" podUID="5ce05a81-71f3-4dab-96e1-44c9ea628a7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused"
Oct 30 13:36:44.050419 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:44.050342    4364 patch_prober.go:29] interesting pod/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body=
Oct 30 13:36:44.050419 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:44.050404    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg" podUID="5ce05a81-71f3-4dab-96e1-44c9ea628a7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused"
Oct 30 13:36:45.050395 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:45.050356    4364 patch_prober.go:29] interesting pod/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body=
Oct 30 13:36:45.050594 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:45.050424    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg" podUID="5ce05a81-71f3-4dab-96e1-44c9ea628a7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused"
Oct 30 13:36:46.057660 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:46.055468    4364 patch_prober.go:29] interesting pod/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body=
Oct 30 13:36:46.057660 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:46.055544    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg" podUID="5ce05a81-71f3-4dab-96e1-44c9ea628a7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused"
Oct 30 13:36:47.049926 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:47.049890    4364 patch_prober.go:29] interesting pod/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body=
Oct 30 13:36:47.050127 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:47.049962    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg" podUID="5ce05a81-71f3-4dab-96e1-44c9ea628a7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused"
Oct 30 13:36:48.054664 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:48.054630    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg"
Oct 30 13:36:48.055913 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:48.055889    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg"
Oct 30 13:36:53.799823 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:53.799784    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85"]
Oct 30 13:36:53.800334 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:53.800073    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85" podUID="01c37c15-88bc-4359-9651-fdaae5a98c40" containerName="istio-proxy" containerID="cri-o://abd8e875ac42ebe0d62371418107fac03e62f2eb648bd937a344e9931f745a24" gracePeriod=30
Oct 30 13:36:56.458392 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:56.458357    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a8d3b8-7f9bfff7bc-hszb2"]
Oct 30 13:36:56.459374 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:56.459350    4364 cpu_manager.go:414] "containerMap: removing container" podUID="d8781a4e-4171-4db3-808e-a8f412894661" containerName="main"
Oct 30 13:36:56.459374 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:56.459373    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8781a4e-4171-4db3-808e-a8f412894661" containerName="main"
Oct 30 13:36:56.459549 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:56.459390    4364 cpu_manager.go:414] "containerMap: removing container" podUID="668e3034-e72e-4e25-ab29-d76303e341ac" containerName="payload-processing"
Oct 30 13:36:56.459549 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:56.459398    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="668e3034-e72e-4e25-ab29-d76303e341ac" containerName="payload-processing"
Oct 30 13:36:56.459549 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:56.459432    4364 cpu_manager.go:414] "containerMap: removing container" podUID="c0beff44-3603-4a88-bc25-78f6e3fbbddf" containerName="payload-processing"
Oct 30 13:36:56.459549 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:56.459440    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0beff44-3603-4a88-bc25-78f6e3fbbddf" containerName="payload-processing"
Oct 30 13:36:56.459810 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:56.459700    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0beff44-3603-4a88-bc25-78f6e3fbbddf" containerName="payload-processing"
Oct 30 13:36:56.459810 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:56.459734    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8781a4e-4171-4db3-808e-a8f412894661" containerName="main"
Oct 30 13:36:56.459810 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:56.459748    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="668e3034-e72e-4e25-ab29-d76303e341ac" containerName="payload-processing"
Oct 30 13:36:56.463526 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:56.463508    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a8d3b8-7f9bfff7bc-hszb2"
Oct 30 13:36:56.482838 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:56.482811    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a8d3b8-7f9bfff7bc-hszb2"]
Oct 30 13:36:56.498011 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:56.497982    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-a8d3b8-serving-cert\""
Oct 30 13:36:56.535116 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:56.535074    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/213f7500-6291-40df-8a5e-c7008a6ccc47-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a8d3b8-7f9bfff7bc-hszb2\" (UID: \"213f7500-6291-40df-8a5e-c7008a6ccc47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a8d3b8-7f9bfff7bc-hszb2"
Oct 30 13:36:56.535295 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:56.535275    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/213f7500-6291-40df-8a5e-c7008a6ccc47-metrics-certs\") pod \"maas-api-e2e-ipp-a-a8d3b8-7f9bfff7bc-hszb2\" (UID: \"213f7500-6291-40df-8a5e-c7008a6ccc47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a8d3b8-7f9bfff7bc-hszb2"
Oct 30 13:36:56.535453 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:56.535435    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/213f7500-6291-40df-8a5e-c7008a6ccc47-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a8d3b8-7f9bfff7bc-hszb2\" (UID: \"213f7500-6291-40df-8a5e-c7008a6ccc47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a8d3b8-7f9bfff7bc-hszb2"
Oct 30 13:36:56.535531 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:56.535472    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwsqq\" (UniqueName: \"kubernetes.io/projected/213f7500-6291-40df-8a5e-c7008a6ccc47-kube-api-access-mwsqq\") pod \"maas-api-e2e-ipp-a-a8d3b8-7f9bfff7bc-hszb2\" (UID: \"213f7500-6291-40df-8a5e-c7008a6ccc47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a8d3b8-7f9bfff7bc-hszb2"
Oct 30 13:36:56.637139 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:56.637105    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/213f7500-6291-40df-8a5e-c7008a6ccc47-metrics-certs\") pod \"maas-api-e2e-ipp-a-a8d3b8-7f9bfff7bc-hszb2\" (UID: \"213f7500-6291-40df-8a5e-c7008a6ccc47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a8d3b8-7f9bfff7bc-hszb2"
Oct 30 13:36:56.637323 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:56.637204    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/213f7500-6291-40df-8a5e-c7008a6ccc47-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a8d3b8-7f9bfff7bc-hszb2\" (UID: \"213f7500-6291-40df-8a5e-c7008a6ccc47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a8d3b8-7f9bfff7bc-hszb2"
Oct 30 13:36:56.637323 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:56.637231    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwsqq\" (UniqueName: \"kubernetes.io/projected/213f7500-6291-40df-8a5e-c7008a6ccc47-kube-api-access-mwsqq\") pod \"maas-api-e2e-ipp-a-a8d3b8-7f9bfff7bc-hszb2\" (UID: \"213f7500-6291-40df-8a5e-c7008a6ccc47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a8d3b8-7f9bfff7bc-hszb2"
Oct 30 13:36:56.637455 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:56.637396    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/213f7500-6291-40df-8a5e-c7008a6ccc47-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a8d3b8-7f9bfff7bc-hszb2\" (UID: \"213f7500-6291-40df-8a5e-c7008a6ccc47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a8d3b8-7f9bfff7bc-hszb2"
Oct 30 13:36:56.638041 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:56.638015    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/213f7500-6291-40df-8a5e-c7008a6ccc47-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a8d3b8-7f9bfff7bc-hszb2\" (UID: \"213f7500-6291-40df-8a5e-c7008a6ccc47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a8d3b8-7f9bfff7bc-hszb2"
Oct 30 13:36:56.639826 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:56.639797    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/213f7500-6291-40df-8a5e-c7008a6ccc47-metrics-certs\") pod \"maas-api-e2e-ipp-a-a8d3b8-7f9bfff7bc-hszb2\" (UID: \"213f7500-6291-40df-8a5e-c7008a6ccc47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a8d3b8-7f9bfff7bc-hszb2"
Oct 30 13:36:56.640013 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:56.639990    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/213f7500-6291-40df-8a5e-c7008a6ccc47-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a8d3b8-7f9bfff7bc-hszb2\" (UID: \"213f7500-6291-40df-8a5e-c7008a6ccc47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a8d3b8-7f9bfff7bc-hszb2"
Oct 30 13:36:56.662969 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:56.662942    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwsqq\" (UniqueName: \"kubernetes.io/projected/213f7500-6291-40df-8a5e-c7008a6ccc47-kube-api-access-mwsqq\") pod \"maas-api-e2e-ipp-a-a8d3b8-7f9bfff7bc-hszb2\" (UID: \"213f7500-6291-40df-8a5e-c7008a6ccc47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a8d3b8-7f9bfff7bc-hszb2"
Oct 30 13:36:56.780694 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:56.780587    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a8d3b8-7f9bfff7bc-hszb2"
Oct 30 13:36:57.166981 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:57.166954    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a8d3b8-7f9bfff7bc-hszb2"]
Oct 30 13:36:57.167500 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:36:57.167473    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod213f7500_6291_40df_8a5e_c7008a6ccc47.slice/crio-738e71bc74d9e1c5dece10dc325dc371aeb5052afea71b5e7ab616966d914cec WatchSource:0}: Error finding container 738e71bc74d9e1c5dece10dc325dc371aeb5052afea71b5e7ab616966d914cec: Status 404 returned error can't find the container with id 738e71bc74d9e1c5dece10dc325dc371aeb5052afea71b5e7ab616966d914cec
Oct 30 13:36:57.432451 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:57.432365    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a8d3b8-7f9bfff7bc-hszb2" event={"ID":"213f7500-6291-40df-8a5e-c7008a6ccc47","Type":"ContainerStarted","Data":"738e71bc74d9e1c5dece10dc325dc371aeb5052afea71b5e7ab616966d914cec"}
Oct 30 13:36:58.397991 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:58.397954    4364 patch_prober.go:29] interesting pod/e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:36:58.398410 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:58.398016    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85" podUID="01c37c15-88bc-4359-9651-fdaae5a98c40" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:36:58.453666 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:58.453630    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a8d3b8-7f9bfff7bc-hszb2" event={"ID":"213f7500-6291-40df-8a5e-c7008a6ccc47","Type":"ContainerStarted","Data":"e3f050944e199cae5734e83970ef006667b3d2e534b15e22321b96acdcd2689a"}
Oct 30 13:36:58.453875 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:58.453846    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a8d3b8-7f9bfff7bc-hszb2"
Oct 30 13:36:58.500128 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:58.500078    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a8d3b8-7f9bfff7bc-hszb2" podStartSLOduration=2.01439056 podStartE2EDuration="2.50006467s" totalImagesPullingTime="485.67411ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:36:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:36:58.494816369 +0000 UTC m=+2717.661181017" watchObservedRunningTime="2026-10-30 13:36:58.50006467 +0000 UTC m=+2717.666429355"
Oct 30 13:36:59.090485 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:59.090463    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85"
Oct 30 13:36:59.132832 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:59.132806    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01c37c15-88bc-4359-9651-fdaae5a98c40-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01c37c15-88bc-4359-9651-fdaae5a98c40-istiod-ca-cert\") pod \"01c37c15-88bc-4359-9651-fdaae5a98c40\" (UID: \"01c37c15-88bc-4359-9651-fdaae5a98c40\") "
Oct 30 13:36:59.133015 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:59.132876    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01c37c15-88bc-4359-9651-fdaae5a98c40-istio-token\" (UniqueName: \"kubernetes.io/projected/01c37c15-88bc-4359-9651-fdaae5a98c40-istio-token\") pod \"01c37c15-88bc-4359-9651-fdaae5a98c40\" (UID: \"01c37c15-88bc-4359-9651-fdaae5a98c40\") "
Oct 30 13:36:59.133015 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:59.132928    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/01c37c15-88bc-4359-9651-fdaae5a98c40-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01c37c15-88bc-4359-9651-fdaae5a98c40-istio-podinfo\") pod \"01c37c15-88bc-4359-9651-fdaae5a98c40\" (UID: \"01c37c15-88bc-4359-9651-fdaae5a98c40\") "
Oct 30 13:36:59.133015 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:59.132945    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01c37c15-88bc-4359-9651-fdaae5a98c40-kube-api-access-kp7db\" (UniqueName: \"kubernetes.io/projected/01c37c15-88bc-4359-9651-fdaae5a98c40-kube-api-access-kp7db\") pod \"01c37c15-88bc-4359-9651-fdaae5a98c40\" (UID: \"01c37c15-88bc-4359-9651-fdaae5a98c40\") "
Oct 30 13:36:59.133194 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:59.133067    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01c37c15-88bc-4359-9651-fdaae5a98c40-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01c37c15-88bc-4359-9651-fdaae5a98c40-istio-envoy\") pod \"01c37c15-88bc-4359-9651-fdaae5a98c40\" (UID: \"01c37c15-88bc-4359-9651-fdaae5a98c40\") "
Oct 30 13:36:59.133264 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:59.133209    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01c37c15-88bc-4359-9651-fdaae5a98c40-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01c37c15-88bc-4359-9651-fdaae5a98c40-credential-socket\") pod \"01c37c15-88bc-4359-9651-fdaae5a98c40\" (UID: \"01c37c15-88bc-4359-9651-fdaae5a98c40\") "
Oct 30 13:36:59.133264 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:59.133226    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01c37c15-88bc-4359-9651-fdaae5a98c40-istiod-ca-cert" pod "01c37c15-88bc-4359-9651-fdaae5a98c40" (UID: "01c37c15-88bc-4359-9651-fdaae5a98c40"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:36:59.133382 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:59.133302    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01c37c15-88bc-4359-9651-fdaae5a98c40-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01c37c15-88bc-4359-9651-fdaae5a98c40-workload-certs\") pod \"01c37c15-88bc-4359-9651-fdaae5a98c40\" (UID: \"01c37c15-88bc-4359-9651-fdaae5a98c40\") "
Oct 30 13:36:59.133442 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:59.133407    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01c37c15-88bc-4359-9651-fdaae5a98c40-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01c37c15-88bc-4359-9651-fdaae5a98c40-istio-data\") pod \"01c37c15-88bc-4359-9651-fdaae5a98c40\" (UID: \"01c37c15-88bc-4359-9651-fdaae5a98c40\") "
Oct 30 13:36:59.133504 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:59.133445    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01c37c15-88bc-4359-9651-fdaae5a98c40-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01c37c15-88bc-4359-9651-fdaae5a98c40-workload-socket\") pod \"01c37c15-88bc-4359-9651-fdaae5a98c40\" (UID: \"01c37c15-88bc-4359-9651-fdaae5a98c40\") "
Oct 30 13:36:59.133504 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:59.133486    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01c37c15-88bc-4359-9651-fdaae5a98c40-credential-socket" pod "01c37c15-88bc-4359-9651-fdaae5a98c40" (UID: "01c37c15-88bc-4359-9651-fdaae5a98c40"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:36:59.133709 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:59.133683    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01c37c15-88bc-4359-9651-fdaae5a98c40-workload-certs" pod "01c37c15-88bc-4359-9651-fdaae5a98c40" (UID: "01c37c15-88bc-4359-9651-fdaae5a98c40"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:36:59.133709 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:59.133696    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01c37c15-88bc-4359-9651-fdaae5a98c40-istio-data" pod "01c37c15-88bc-4359-9651-fdaae5a98c40" (UID: "01c37c15-88bc-4359-9651-fdaae5a98c40"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:36:59.133992 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:59.133956    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01c37c15-88bc-4359-9651-fdaae5a98c40-workload-socket" pod "01c37c15-88bc-4359-9651-fdaae5a98c40" (UID: "01c37c15-88bc-4359-9651-fdaae5a98c40"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:36:59.134258 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:59.134229    4364 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01c37c15-88bc-4359-9651-fdaae5a98c40-credential-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:36:59.134356 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:59.134264    4364 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01c37c15-88bc-4359-9651-fdaae5a98c40-workload-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:36:59.134356 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:59.134281    4364 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01c37c15-88bc-4359-9651-fdaae5a98c40-istio-data\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:36:59.134356 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:59.134295    4364 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01c37c15-88bc-4359-9651-fdaae5a98c40-workload-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:36:59.134356 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:59.134308    4364 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01c37c15-88bc-4359-9651-fdaae5a98c40-istiod-ca-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:36:59.135434 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:59.135406    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/01c37c15-88bc-4359-9651-fdaae5a98c40-istio-podinfo" pod "01c37c15-88bc-4359-9651-fdaae5a98c40" (UID: "01c37c15-88bc-4359-9651-fdaae5a98c40"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:36:59.135552 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:59.135480    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01c37c15-88bc-4359-9651-fdaae5a98c40-kube-api-access-kp7db" pod "01c37c15-88bc-4359-9651-fdaae5a98c40" (UID: "01c37c15-88bc-4359-9651-fdaae5a98c40"). InnerVolumeSpecName "kube-api-access-kp7db". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:36:59.135643 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:59.135592    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01c37c15-88bc-4359-9651-fdaae5a98c40-istio-token" pod "01c37c15-88bc-4359-9651-fdaae5a98c40" (UID: "01c37c15-88bc-4359-9651-fdaae5a98c40"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:36:59.135691 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:59.135651    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01c37c15-88bc-4359-9651-fdaae5a98c40-istio-envoy" pod "01c37c15-88bc-4359-9651-fdaae5a98c40" (UID: "01c37c15-88bc-4359-9651-fdaae5a98c40"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:36:59.236590 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:59.236556    4364 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01c37c15-88bc-4359-9651-fdaae5a98c40-istio-token\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:36:59.236590 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:59.236590    4364 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01c37c15-88bc-4359-9651-fdaae5a98c40-istio-podinfo\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:36:59.236852 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:59.236611    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kp7db\" (UniqueName: \"kubernetes.io/projected/01c37c15-88bc-4359-9651-fdaae5a98c40-kube-api-access-kp7db\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:36:59.236852 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:59.236623    4364 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01c37c15-88bc-4359-9651-fdaae5a98c40-istio-envoy\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:36:59.318448 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:59.318365    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4"]
Oct 30 13:36:59.467180 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:59.467145    4364 generic.go:356] "Generic (PLEG): container finished" podID="01c37c15-88bc-4359-9651-fdaae5a98c40" containerID="abd8e875ac42ebe0d62371418107fac03e62f2eb648bd937a344e9931f745a24" exitCode=0
Oct 30 13:36:59.467630 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:59.467198    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85"
Oct 30 13:36:59.467630 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:59.467231    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85" event={"ID":"01c37c15-88bc-4359-9651-fdaae5a98c40","Type":"ContainerDied","Data":"abd8e875ac42ebe0d62371418107fac03e62f2eb648bd937a344e9931f745a24"}
Oct 30 13:36:59.467630 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:59.467289    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85" event={"ID":"01c37c15-88bc-4359-9651-fdaae5a98c40","Type":"ContainerDied","Data":"a4fbf50ddd2967977deef7975c07e7bb5cad3ad2cd7aeec5ea0e2415a030eb21"}
Oct 30 13:36:59.467630 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:59.467322    4364 scope.go:122] "RemoveContainer" containerID="abd8e875ac42ebe0d62371418107fac03e62f2eb648bd937a344e9931f745a24"
Oct 30 13:36:59.479912 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:59.479865    4364 scope.go:122] "RemoveContainer" containerID="6e3b5d4a99c8fa36fefdd8cc9cb28310c2a270188417fa6fd0de5a998c86c16e"
Oct 30 13:36:59.489788 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:59.489770    4364 scope.go:122] "RemoveContainer" containerID="abd8e875ac42ebe0d62371418107fac03e62f2eb648bd937a344e9931f745a24"
Oct 30 13:36:59.490060 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:36:59.490042    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abd8e875ac42ebe0d62371418107fac03e62f2eb648bd937a344e9931f745a24\": container with ID starting with abd8e875ac42ebe0d62371418107fac03e62f2eb648bd937a344e9931f745a24 not found: ID does not exist" containerID="abd8e875ac42ebe0d62371418107fac03e62f2eb648bd937a344e9931f745a24"
Oct 30 13:36:59.490101 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:59.490070    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abd8e875ac42ebe0d62371418107fac03e62f2eb648bd937a344e9931f745a24"} err="failed to get container status \"abd8e875ac42ebe0d62371418107fac03e62f2eb648bd937a344e9931f745a24\": rpc error: code = NotFound desc = could not find container \"abd8e875ac42ebe0d62371418107fac03e62f2eb648bd937a344e9931f745a24\": container with ID starting with abd8e875ac42ebe0d62371418107fac03e62f2eb648bd937a344e9931f745a24 not found: ID does not exist"
Oct 30 13:36:59.490101 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:59.490087    4364 scope.go:122] "RemoveContainer" containerID="6e3b5d4a99c8fa36fefdd8cc9cb28310c2a270188417fa6fd0de5a998c86c16e"
Oct 30 13:36:59.490333 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:36:59.490317    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e3b5d4a99c8fa36fefdd8cc9cb28310c2a270188417fa6fd0de5a998c86c16e\": container with ID starting with 6e3b5d4a99c8fa36fefdd8cc9cb28310c2a270188417fa6fd0de5a998c86c16e not found: ID does not exist" containerID="6e3b5d4a99c8fa36fefdd8cc9cb28310c2a270188417fa6fd0de5a998c86c16e"
Oct 30 13:36:59.490378 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:59.490339    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e3b5d4a99c8fa36fefdd8cc9cb28310c2a270188417fa6fd0de5a998c86c16e"} err="failed to get container status \"6e3b5d4a99c8fa36fefdd8cc9cb28310c2a270188417fa6fd0de5a998c86c16e\": rpc error: code = NotFound desc = could not find container \"6e3b5d4a99c8fa36fefdd8cc9cb28310c2a270188417fa6fd0de5a998c86c16e\": container with ID starting with 6e3b5d4a99c8fa36fefdd8cc9cb28310c2a270188417fa6fd0de5a998c86c16e not found: ID does not exist"
Oct 30 13:36:59.523717 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:59.523692    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85"]
Oct 30 13:36:59.528807 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:36:59.528786    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-b29df4-openshift-default-77cf4d4ff-xtx85"]
Oct 30 13:37:00.972149 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:00.972109    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01c37c15-88bc-4359-9651-fdaae5a98c40" path="/var/lib/kubelet/pods/01c37c15-88bc-4359-9651-fdaae5a98c40/volumes"
Oct 30 13:37:04.472700 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:04.472668    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a8d3b8-7f9bfff7bc-hszb2"
Oct 30 13:37:04.980555 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:04.980522    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-d7e62b-6fb765d78-mqjvs"]
Oct 30 13:37:05.012191 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:05.012164    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-d7e62b-6869d5f5b7-ztlqq"]
Oct 30 13:37:09.295339 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:09.295303    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:37:09.980831 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:09.980788    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-d7e62b-6fb765d78-mqjvs" podUID="48e64d9a-c945-4316-b973-6829e30b051b" containerName="payload-processing" containerID="cri-o://6eb00dee5eb32002ae86054ce73c90c8f9396a3696721df345a9bef23945fe8c" gracePeriod=25
Oct 30 13:37:10.013338 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:10.013300    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d7e62b-6869d5f5b7-ztlqq" podUID="c9195c96-60b4-49ff-a742-34b7ce75c7c6" containerName="payload-pre-processing" containerID="cri-o://52e08edc6f1f993e1fdc01e00a46db6a51137044a501ff637f35432ac33c5b1b" gracePeriod=25
Oct 30 13:37:11.506883 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:37:11.506840    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:37:14.341117 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:14.341058    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4" podUID="0cbe6af7-d884-4613-90b8-192e5ffdb027" containerName="main" containerID="cri-o://1e9f15efe880081793ca793be333bbf83b44adcd393205596ea5e290842a2fa4" gracePeriod=45
Oct 30 13:37:14.667643 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:14.667473    4364 generic.go:356] "Generic (PLEG): container finished" podID="0cbe6af7-d884-4613-90b8-192e5ffdb027" containerID="1e9f15efe880081793ca793be333bbf83b44adcd393205596ea5e290842a2fa4" exitCode=0
Oct 30 13:37:14.668671 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:14.668007    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4" event={"ID":"0cbe6af7-d884-4613-90b8-192e5ffdb027","Type":"ContainerDied","Data":"1e9f15efe880081793ca793be333bbf83b44adcd393205596ea5e290842a2fa4"}
Oct 30 13:37:14.717870 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:14.717837    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4"
Oct 30 13:37:14.832927 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:14.832895    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cbe6af7-d884-4613-90b8-192e5ffdb027-kube-api-access-4f4lk\" (UniqueName: \"kubernetes.io/projected/0cbe6af7-d884-4613-90b8-192e5ffdb027-kube-api-access-4f4lk\") pod \"0cbe6af7-d884-4613-90b8-192e5ffdb027\" (UID: \"0cbe6af7-d884-4613-90b8-192e5ffdb027\") "
Oct 30 13:37:14.833129 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:14.833071    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cbe6af7-d884-4613-90b8-192e5ffdb027-home\" (UniqueName: \"kubernetes.io/empty-dir/0cbe6af7-d884-4613-90b8-192e5ffdb027-home\") pod \"0cbe6af7-d884-4613-90b8-192e5ffdb027\" (UID: \"0cbe6af7-d884-4613-90b8-192e5ffdb027\") "
Oct 30 13:37:14.833295 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:14.833276    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cbe6af7-d884-4613-90b8-192e5ffdb027-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cbe6af7-d884-4613-90b8-192e5ffdb027-tmp-dir\") pod \"0cbe6af7-d884-4613-90b8-192e5ffdb027\" (UID: \"0cbe6af7-d884-4613-90b8-192e5ffdb027\") "
Oct 30 13:37:14.833370 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:14.833313    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cbe6af7-d884-4613-90b8-192e5ffdb027-tls-certs\" (UniqueName: \"kubernetes.io/secret/0cbe6af7-d884-4613-90b8-192e5ffdb027-tls-certs\") pod \"0cbe6af7-d884-4613-90b8-192e5ffdb027\" (UID: \"0cbe6af7-d884-4613-90b8-192e5ffdb027\") "
Oct 30 13:37:14.833370 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:14.833332    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cbe6af7-d884-4613-90b8-192e5ffdb027-home" pod "0cbe6af7-d884-4613-90b8-192e5ffdb027" (UID: "0cbe6af7-d884-4613-90b8-192e5ffdb027"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:14.833481 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:14.833446    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cbe6af7-d884-4613-90b8-192e5ffdb027-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cbe6af7-d884-4613-90b8-192e5ffdb027-model-cache\") pod \"0cbe6af7-d884-4613-90b8-192e5ffdb027\" (UID: \"0cbe6af7-d884-4613-90b8-192e5ffdb027\") "
Oct 30 13:37:14.833544 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:14.833525    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cbe6af7-d884-4613-90b8-192e5ffdb027-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cbe6af7-d884-4613-90b8-192e5ffdb027-dshm\") pod \"0cbe6af7-d884-4613-90b8-192e5ffdb027\" (UID: \"0cbe6af7-d884-4613-90b8-192e5ffdb027\") "
Oct 30 13:37:14.833544 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:14.833530    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cbe6af7-d884-4613-90b8-192e5ffdb027-tmp-dir" pod "0cbe6af7-d884-4613-90b8-192e5ffdb027" (UID: "0cbe6af7-d884-4613-90b8-192e5ffdb027"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:14.834332 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:14.834299    4364 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cbe6af7-d884-4613-90b8-192e5ffdb027-home\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:37:14.834332 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:14.834326    4364 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cbe6af7-d884-4613-90b8-192e5ffdb027-tmp-dir\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:37:14.834884 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:14.834857    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cbe6af7-d884-4613-90b8-192e5ffdb027-model-cache" pod "0cbe6af7-d884-4613-90b8-192e5ffdb027" (UID: "0cbe6af7-d884-4613-90b8-192e5ffdb027"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:14.836647 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:14.835693    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cbe6af7-d884-4613-90b8-192e5ffdb027-kube-api-access-4f4lk" pod "0cbe6af7-d884-4613-90b8-192e5ffdb027" (UID: "0cbe6af7-d884-4613-90b8-192e5ffdb027"). InnerVolumeSpecName "kube-api-access-4f4lk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:37:14.836647 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:14.835993    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cbe6af7-d884-4613-90b8-192e5ffdb027-tls-certs" pod "0cbe6af7-d884-4613-90b8-192e5ffdb027" (UID: "0cbe6af7-d884-4613-90b8-192e5ffdb027"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:37:14.836647 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:14.836383    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cbe6af7-d884-4613-90b8-192e5ffdb027-dshm" pod "0cbe6af7-d884-4613-90b8-192e5ffdb027" (UID: "0cbe6af7-d884-4613-90b8-192e5ffdb027"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:14.952691 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:14.951804    4364 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cbe6af7-d884-4613-90b8-192e5ffdb027-model-cache\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:37:14.952691 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:14.951846    4364 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cbe6af7-d884-4613-90b8-192e5ffdb027-dshm\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:37:14.952691 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:14.951860    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4f4lk\" (UniqueName: \"kubernetes.io/projected/0cbe6af7-d884-4613-90b8-192e5ffdb027-kube-api-access-4f4lk\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:37:14.952691 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:14.951875    4364 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cbe6af7-d884-4613-90b8-192e5ffdb027-tls-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:37:15.635237 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:15.635192    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4" podUID="0cbe6af7-d884-4613-90b8-192e5ffdb027" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:37:15.676976 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:15.676944    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4"
Oct 30 13:37:15.677171 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:15.676982    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4" event={"ID":"0cbe6af7-d884-4613-90b8-192e5ffdb027","Type":"ContainerDied","Data":"afe7bb8a4b232148ec5e82d2f6eee1fc9b894750d5282ea32ee6da7bb1559a9c"}
Oct 30 13:37:15.677171 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:15.677046    4364 scope.go:122] "RemoveContainer" containerID="1e9f15efe880081793ca793be333bbf83b44adcd393205596ea5e290842a2fa4"
Oct 30 13:37:15.730575 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:15.730546    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4"]
Oct 30 13:37:15.742003 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:15.741978    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-d7e62b/distinct-w4-d7e62b-kserve-577bbb658c-2gch4"]
Oct 30 13:37:16.972577 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:16.972543    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cbe6af7-d884-4613-90b8-192e5ffdb027" path="/var/lib/kubelet/pods/0cbe6af7-d884-4613-90b8-192e5ffdb027/volumes"
Oct 30 13:37:17.797056 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:17.797025    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-78dc89-797f8b64d4-rw7rg"]
Oct 30 13:37:17.797980 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:17.797960    4364 cpu_manager.go:414] "containerMap: removing container" podUID="01c37c15-88bc-4359-9651-fdaae5a98c40" containerName="istio-proxy"
Oct 30 13:37:17.797980 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:17.797979    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01c37c15-88bc-4359-9651-fdaae5a98c40" containerName="istio-proxy"
Oct 30 13:37:17.798146 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:17.797995    4364 cpu_manager.go:414] "containerMap: removing container" podUID="0cbe6af7-d884-4613-90b8-192e5ffdb027" containerName="main"
Oct 30 13:37:17.798146 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:17.798001    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cbe6af7-d884-4613-90b8-192e5ffdb027" containerName="main"
Oct 30 13:37:17.798146 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:17.798022    4364 cpu_manager.go:414] "containerMap: removing container" podUID="01c37c15-88bc-4359-9651-fdaae5a98c40" containerName="istio-proxy"
Oct 30 13:37:17.798146 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:17.798027    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01c37c15-88bc-4359-9651-fdaae5a98c40" containerName="istio-proxy"
Oct 30 13:37:17.798146 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:17.798129    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="01c37c15-88bc-4359-9651-fdaae5a98c40" containerName="istio-proxy"
Oct 30 13:37:17.798146 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:17.798145    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cbe6af7-d884-4613-90b8-192e5ffdb027" containerName="main"
Oct 30 13:37:17.801549 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:17.801530    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78dc89-797f8b64d4-rw7rg"
Oct 30 13:37:17.825300 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:17.825273    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-78dc89-serving-cert\""
Oct 30 13:37:17.862477 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:17.862439    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-78dc89-797f8b64d4-rw7rg"]
Oct 30 13:37:17.896225 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:17.896190    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94f6ed0a-3007-4d3a-a569-920348b7da49-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-78dc89-797f8b64d4-rw7rg\" (UID: \"94f6ed0a-3007-4d3a-a569-920348b7da49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78dc89-797f8b64d4-rw7rg"
Oct 30 13:37:17.896433 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:17.896270    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94f6ed0a-3007-4d3a-a569-920348b7da49-maas-api-tls\") pod \"maas-api-e2e-ipp-a-78dc89-797f8b64d4-rw7rg\" (UID: \"94f6ed0a-3007-4d3a-a569-920348b7da49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78dc89-797f8b64d4-rw7rg"
Oct 30 13:37:17.896433 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:17.896382    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7smr6\" (UniqueName: \"kubernetes.io/projected/94f6ed0a-3007-4d3a-a569-920348b7da49-kube-api-access-7smr6\") pod \"maas-api-e2e-ipp-a-78dc89-797f8b64d4-rw7rg\" (UID: \"94f6ed0a-3007-4d3a-a569-920348b7da49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78dc89-797f8b64d4-rw7rg"
Oct 30 13:37:17.896632 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:17.896578    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94f6ed0a-3007-4d3a-a569-920348b7da49-metrics-certs\") pod \"maas-api-e2e-ipp-a-78dc89-797f8b64d4-rw7rg\" (UID: \"94f6ed0a-3007-4d3a-a569-920348b7da49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78dc89-797f8b64d4-rw7rg"
Oct 30 13:37:17.998199 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:17.998164    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94f6ed0a-3007-4d3a-a569-920348b7da49-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-78dc89-797f8b64d4-rw7rg\" (UID: \"94f6ed0a-3007-4d3a-a569-920348b7da49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78dc89-797f8b64d4-rw7rg"
Oct 30 13:37:17.998623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:17.998224    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94f6ed0a-3007-4d3a-a569-920348b7da49-maas-api-tls\") pod \"maas-api-e2e-ipp-a-78dc89-797f8b64d4-rw7rg\" (UID: \"94f6ed0a-3007-4d3a-a569-920348b7da49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78dc89-797f8b64d4-rw7rg"
Oct 30 13:37:17.998623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:17.998271    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7smr6\" (UniqueName: \"kubernetes.io/projected/94f6ed0a-3007-4d3a-a569-920348b7da49-kube-api-access-7smr6\") pod \"maas-api-e2e-ipp-a-78dc89-797f8b64d4-rw7rg\" (UID: \"94f6ed0a-3007-4d3a-a569-920348b7da49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78dc89-797f8b64d4-rw7rg"
Oct 30 13:37:17.998623 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:37:17.998374    4364 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-78dc89-serving-cert: secret "maas-api-e2e-ipp-a-78dc89-serving-cert" not found
Oct 30 13:37:17.998623 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:37:17.998447    4364 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94f6ed0a-3007-4d3a-a569-920348b7da49-maas-api-tls podName:94f6ed0a-3007-4d3a-a569-920348b7da49 nodeName:}" failed. No retries permitted until 2026-10-30 13:37:18.498425769 +0000 UTC m=+2737.664790412 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/94f6ed0a-3007-4d3a-a569-920348b7da49-maas-api-tls") pod "maas-api-e2e-ipp-a-78dc89-797f8b64d4-rw7rg" (UID: "94f6ed0a-3007-4d3a-a569-920348b7da49") : secret "maas-api-e2e-ipp-a-78dc89-serving-cert" not found
Oct 30 13:37:17.998623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:17.998384    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94f6ed0a-3007-4d3a-a569-920348b7da49-metrics-certs\") pod \"maas-api-e2e-ipp-a-78dc89-797f8b64d4-rw7rg\" (UID: \"94f6ed0a-3007-4d3a-a569-920348b7da49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78dc89-797f8b64d4-rw7rg"
Oct 30 13:37:17.998956 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:17.998921    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94f6ed0a-3007-4d3a-a569-920348b7da49-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-78dc89-797f8b64d4-rw7rg\" (UID: \"94f6ed0a-3007-4d3a-a569-920348b7da49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78dc89-797f8b64d4-rw7rg"
Oct 30 13:37:18.000945 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:18.000926    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94f6ed0a-3007-4d3a-a569-920348b7da49-metrics-certs\") pod \"maas-api-e2e-ipp-a-78dc89-797f8b64d4-rw7rg\" (UID: \"94f6ed0a-3007-4d3a-a569-920348b7da49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78dc89-797f8b64d4-rw7rg"
Oct 30 13:37:18.022818 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:18.022793    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7smr6\" (UniqueName: \"kubernetes.io/projected/94f6ed0a-3007-4d3a-a569-920348b7da49-kube-api-access-7smr6\") pod \"maas-api-e2e-ipp-a-78dc89-797f8b64d4-rw7rg\" (UID: \"94f6ed0a-3007-4d3a-a569-920348b7da49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78dc89-797f8b64d4-rw7rg"
Oct 30 13:37:18.340156 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:18.340120    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-d7e62b-6fb765d78-mqjvs" podUID="48e64d9a-c945-4316-b973-6829e30b051b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:37:18.340334 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:18.340122    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d7e62b-6869d5f5b7-ztlqq" podUID="c9195c96-60b4-49ff-a742-34b7ce75c7c6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:37:18.509732 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:18.509701    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94f6ed0a-3007-4d3a-a569-920348b7da49-maas-api-tls\") pod \"maas-api-e2e-ipp-a-78dc89-797f8b64d4-rw7rg\" (UID: \"94f6ed0a-3007-4d3a-a569-920348b7da49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78dc89-797f8b64d4-rw7rg"
Oct 30 13:37:18.512250 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:18.512218    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94f6ed0a-3007-4d3a-a569-920348b7da49-maas-api-tls\") pod \"maas-api-e2e-ipp-a-78dc89-797f8b64d4-rw7rg\" (UID: \"94f6ed0a-3007-4d3a-a569-920348b7da49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78dc89-797f8b64d4-rw7rg"
Oct 30 13:37:18.716377 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:18.716348    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78dc89-797f8b64d4-rw7rg"
Oct 30 13:37:19.056292 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:19.056262    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-78dc89-797f8b64d4-rw7rg"]
Oct 30 13:37:19.057344 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:37:19.057317    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94f6ed0a_3007_4d3a_a569_920348b7da49.slice/crio-7e10ff88512ce729f2347d0b962cc9362049a5f07e1edead50c4ad3d192020b3 WatchSource:0}: Error finding container 7e10ff88512ce729f2347d0b962cc9362049a5f07e1edead50c4ad3d192020b3: Status 404 returned error can't find the container with id 7e10ff88512ce729f2347d0b962cc9362049a5f07e1edead50c4ad3d192020b3
Oct 30 13:37:19.717630 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:19.717510    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78dc89-797f8b64d4-rw7rg" event={"ID":"94f6ed0a-3007-4d3a-a569-920348b7da49","Type":"ContainerStarted","Data":"e3a020c85c35e381f39b31651d4cec000281abb28698bc1c59c5c6fd925f6372"}
Oct 30 13:37:19.717630 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:19.717573    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78dc89-797f8b64d4-rw7rg" event={"ID":"94f6ed0a-3007-4d3a-a569-920348b7da49","Type":"ContainerStarted","Data":"7e10ff88512ce729f2347d0b962cc9362049a5f07e1edead50c4ad3d192020b3"}
Oct 30 13:37:19.717630 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:19.717630    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78dc89-797f8b64d4-rw7rg"
Oct 30 13:37:19.763589 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:19.763542    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78dc89-797f8b64d4-rw7rg" podStartSLOduration=2.364893104 podStartE2EDuration="2.76352952s" totalImagesPullingTime="398.636416ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:37:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:37:19.760992269 +0000 UTC m=+2738.927356918" watchObservedRunningTime="2026-10-30 13:37:19.76352952 +0000 UTC m=+2738.929894167"
Oct 30 13:37:25.732348 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:25.732274    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78dc89-797f8b64d4-rw7rg"
Oct 30 13:37:28.339555 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:28.339516    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d7e62b-6869d5f5b7-ztlqq" podUID="c9195c96-60b4-49ff-a742-34b7ce75c7c6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:37:28.339555 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:28.339543    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-d7e62b-6fb765d78-mqjvs" podUID="48e64d9a-c945-4316-b973-6829e30b051b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:37:30.299567 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:30.299540    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-d7e62b-6fb765d78-mqjvs"
Oct 30 13:37:30.351390 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:30.351363    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d7e62b-6869d5f5b7-ztlqq"
Oct 30 13:37:30.472749 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:30.472714    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48e64d9a-c945-4316-b973-6829e30b051b-kube-api-access-zkrxt\" (UniqueName: \"kubernetes.io/projected/48e64d9a-c945-4316-b973-6829e30b051b-kube-api-access-zkrxt\") pod \"48e64d9a-c945-4316-b973-6829e30b051b\" (UID: \"48e64d9a-c945-4316-b973-6829e30b051b\") "
Oct 30 13:37:30.472948 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:30.472775    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48e64d9a-c945-4316-b973-6829e30b051b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48e64d9a-c945-4316-b973-6829e30b051b-plugins-config-volume\") pod \"48e64d9a-c945-4316-b973-6829e30b051b\" (UID: \"48e64d9a-c945-4316-b973-6829e30b051b\") "
Oct 30 13:37:30.472948 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:30.472802    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9195c96-60b4-49ff-a742-34b7ce75c7c6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9195c96-60b4-49ff-a742-34b7ce75c7c6-plugins-config-volume\") pod \"c9195c96-60b4-49ff-a742-34b7ce75c7c6\" (UID: \"c9195c96-60b4-49ff-a742-34b7ce75c7c6\") "
Oct 30 13:37:30.472948 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:30.472849    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9195c96-60b4-49ff-a742-34b7ce75c7c6-kube-api-access-4lqrj\" (UniqueName: \"kubernetes.io/projected/c9195c96-60b4-49ff-a742-34b7ce75c7c6-kube-api-access-4lqrj\") pod \"c9195c96-60b4-49ff-a742-34b7ce75c7c6\" (UID: \"c9195c96-60b4-49ff-a742-34b7ce75c7c6\") "
Oct 30 13:37:30.473212 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:30.473187    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48e64d9a-c945-4316-b973-6829e30b051b-plugins-config-volume" pod "48e64d9a-c945-4316-b973-6829e30b051b" (UID: "48e64d9a-c945-4316-b973-6829e30b051b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:37:30.473297 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:30.473223    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9195c96-60b4-49ff-a742-34b7ce75c7c6-plugins-config-volume" pod "c9195c96-60b4-49ff-a742-34b7ce75c7c6" (UID: "c9195c96-60b4-49ff-a742-34b7ce75c7c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:37:30.473704 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:30.473687    4364 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48e64d9a-c945-4316-b973-6829e30b051b-plugins-config-volume\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:37:30.473788 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:30.473705    4364 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9195c96-60b4-49ff-a742-34b7ce75c7c6-plugins-config-volume\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:37:30.475025 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:30.475001    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48e64d9a-c945-4316-b973-6829e30b051b-kube-api-access-zkrxt" pod "48e64d9a-c945-4316-b973-6829e30b051b" (UID: "48e64d9a-c945-4316-b973-6829e30b051b"). InnerVolumeSpecName "kube-api-access-zkrxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:37:30.475118 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:30.475079    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9195c96-60b4-49ff-a742-34b7ce75c7c6-kube-api-access-4lqrj" pod "c9195c96-60b4-49ff-a742-34b7ce75c7c6" (UID: "c9195c96-60b4-49ff-a742-34b7ce75c7c6"). InnerVolumeSpecName "kube-api-access-4lqrj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:37:30.575844 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:30.575769    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkrxt\" (UniqueName: \"kubernetes.io/projected/48e64d9a-c945-4316-b973-6829e30b051b-kube-api-access-zkrxt\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:37:30.575844 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:30.575795    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lqrj\" (UniqueName: \"kubernetes.io/projected/c9195c96-60b4-49ff-a742-34b7ce75c7c6-kube-api-access-4lqrj\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:37:30.819863 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:30.819829    4364 generic.go:356] "Generic (PLEG): container finished" podID="48e64d9a-c945-4316-b973-6829e30b051b" containerID="6eb00dee5eb32002ae86054ce73c90c8f9396a3696721df345a9bef23945fe8c" exitCode=0
Oct 30 13:37:30.820035 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:30.819869    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-d7e62b-6fb765d78-mqjvs"
Oct 30 13:37:30.820035 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:30.819908    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-d7e62b-6fb765d78-mqjvs" event={"ID":"48e64d9a-c945-4316-b973-6829e30b051b","Type":"ContainerDied","Data":"6eb00dee5eb32002ae86054ce73c90c8f9396a3696721df345a9bef23945fe8c"}
Oct 30 13:37:30.820035 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:30.819965    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-d7e62b-6fb765d78-mqjvs" event={"ID":"48e64d9a-c945-4316-b973-6829e30b051b","Type":"ContainerDied","Data":"66c76fccd1c125a7a72cebf8394a6d64841f26e4740d27f7137a0b45dfb7c284"}
Oct 30 13:37:30.820035 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:30.819988    4364 scope.go:122] "RemoveContainer" containerID="6eb00dee5eb32002ae86054ce73c90c8f9396a3696721df345a9bef23945fe8c"
Oct 30 13:37:30.821518 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:30.821497    4364 generic.go:356] "Generic (PLEG): container finished" podID="c9195c96-60b4-49ff-a742-34b7ce75c7c6" containerID="52e08edc6f1f993e1fdc01e00a46db6a51137044a501ff637f35432ac33c5b1b" exitCode=0
Oct 30 13:37:30.821738 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:30.821540    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d7e62b-6869d5f5b7-ztlqq"
Oct 30 13:37:30.821858 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:30.821530    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d7e62b-6869d5f5b7-ztlqq" event={"ID":"c9195c96-60b4-49ff-a742-34b7ce75c7c6","Type":"ContainerDied","Data":"52e08edc6f1f993e1fdc01e00a46db6a51137044a501ff637f35432ac33c5b1b"}
Oct 30 13:37:30.821912 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:30.821850    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d7e62b-6869d5f5b7-ztlqq" event={"ID":"c9195c96-60b4-49ff-a742-34b7ce75c7c6","Type":"ContainerDied","Data":"90215cfafa5af1c2f59e20baec7db46fbfce7301eb10e4c5665c18ae2f8c8147"}
Oct 30 13:37:30.833200 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:30.833180    4364 scope.go:122] "RemoveContainer" containerID="6eb00dee5eb32002ae86054ce73c90c8f9396a3696721df345a9bef23945fe8c"
Oct 30 13:37:30.833475 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:37:30.833453    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6eb00dee5eb32002ae86054ce73c90c8f9396a3696721df345a9bef23945fe8c\": container with ID starting with 6eb00dee5eb32002ae86054ce73c90c8f9396a3696721df345a9bef23945fe8c not found: ID does not exist" containerID="6eb00dee5eb32002ae86054ce73c90c8f9396a3696721df345a9bef23945fe8c"
Oct 30 13:37:30.833526 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:30.833481    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6eb00dee5eb32002ae86054ce73c90c8f9396a3696721df345a9bef23945fe8c"} err="failed to get container status \"6eb00dee5eb32002ae86054ce73c90c8f9396a3696721df345a9bef23945fe8c\": rpc error: code = NotFound desc = could not find container \"6eb00dee5eb32002ae86054ce73c90c8f9396a3696721df345a9bef23945fe8c\": container with ID starting with 6eb00dee5eb32002ae86054ce73c90c8f9396a3696721df345a9bef23945fe8c not found: ID does not exist"
Oct 30 13:37:30.833526 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:30.833498    4364 scope.go:122] "RemoveContainer" containerID="52e08edc6f1f993e1fdc01e00a46db6a51137044a501ff637f35432ac33c5b1b"
Oct 30 13:37:30.843540 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:30.843519    4364 scope.go:122] "RemoveContainer" containerID="52e08edc6f1f993e1fdc01e00a46db6a51137044a501ff637f35432ac33c5b1b"
Oct 30 13:37:30.843861 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:37:30.843837    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52e08edc6f1f993e1fdc01e00a46db6a51137044a501ff637f35432ac33c5b1b\": container with ID starting with 52e08edc6f1f993e1fdc01e00a46db6a51137044a501ff637f35432ac33c5b1b not found: ID does not exist" containerID="52e08edc6f1f993e1fdc01e00a46db6a51137044a501ff637f35432ac33c5b1b"
Oct 30 13:37:30.843952 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:30.843867    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52e08edc6f1f993e1fdc01e00a46db6a51137044a501ff637f35432ac33c5b1b"} err="failed to get container status \"52e08edc6f1f993e1fdc01e00a46db6a51137044a501ff637f35432ac33c5b1b\": rpc error: code = NotFound desc = could not find container \"52e08edc6f1f993e1fdc01e00a46db6a51137044a501ff637f35432ac33c5b1b\": container with ID starting with 52e08edc6f1f993e1fdc01e00a46db6a51137044a501ff637f35432ac33c5b1b not found: ID does not exist"
Oct 30 13:37:30.873757 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:30.873736    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-d7e62b-6869d5f5b7-ztlqq"]
Oct 30 13:37:30.885162 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:30.885140    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-d7e62b-6869d5f5b7-ztlqq"]
Oct 30 13:37:30.914729 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:30.914703    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-d7e62b-6fb765d78-mqjvs"]
Oct 30 13:37:30.919318 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:30.919298    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-d7e62b-6fb765d78-mqjvs"]
Oct 30 13:37:30.973690 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:30.973662    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48e64d9a-c945-4316-b973-6829e30b051b" path="/var/lib/kubelet/pods/48e64d9a-c945-4316-b973-6829e30b051b/volumes"
Oct 30 13:37:30.974141 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:30.974128    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9195c96-60b4-49ff-a742-34b7ce75c7c6" path="/var/lib/kubelet/pods/c9195c96-60b4-49ff-a742-34b7ce75c7c6/volumes"
Oct 30 13:37:41.507689 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:37:41.507651    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:37:45.828099 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:45.828059    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c54f0f-6dc7885d9f-h9ghd"]
Oct 30 13:37:45.829119 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:45.829096    4364 cpu_manager.go:414] "containerMap: removing container" podUID="c9195c96-60b4-49ff-a742-34b7ce75c7c6" containerName="payload-pre-processing"
Oct 30 13:37:45.829119 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:45.829116    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9195c96-60b4-49ff-a742-34b7ce75c7c6" containerName="payload-pre-processing"
Oct 30 13:37:45.829281 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:45.829180    4364 cpu_manager.go:414] "containerMap: removing container" podUID="48e64d9a-c945-4316-b973-6829e30b051b" containerName="payload-processing"
Oct 30 13:37:45.829281 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:45.829187    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48e64d9a-c945-4316-b973-6829e30b051b" containerName="payload-processing"
Oct 30 13:37:45.829363 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:45.829294    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="48e64d9a-c945-4316-b973-6829e30b051b" containerName="payload-processing"
Oct 30 13:37:45.829363 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:45.829308    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9195c96-60b4-49ff-a742-34b7ce75c7c6" containerName="payload-pre-processing"
Oct 30 13:37:45.829363 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:45.829316    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="01c37c15-88bc-4359-9651-fdaae5a98c40" containerName="istio-proxy"
Oct 30 13:37:45.834350 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:45.834329    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c54f0f-6dc7885d9f-h9ghd"
Oct 30 13:37:45.855883 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:45.855856    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c54f0f-6dc7885d9f-h9ghd"]
Oct 30 13:37:45.862269 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:45.862240    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-c54f0f\""
Oct 30 13:37:45.862424 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:45.862399    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-c54f0f-dockercfg-twfnb\""
Oct 30 13:37:45.971629 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:45.957668    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tjxq\" (UniqueName: \"kubernetes.io/projected/88af8672-a246-457e-9cfa-534c8d1d604d-kube-api-access-8tjxq\") pod \"payload-pre-processing-e2e-ipp-b-c54f0f-6dc7885d9f-h9ghd\" (UID: \"88af8672-a246-457e-9cfa-534c8d1d604d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c54f0f-6dc7885d9f-h9ghd"
Oct 30 13:37:45.971629 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:45.957972    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88af8672-a246-457e-9cfa-534c8d1d604d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c54f0f-6dc7885d9f-h9ghd\" (UID: \"88af8672-a246-457e-9cfa-534c8d1d604d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c54f0f-6dc7885d9f-h9ghd"
Oct 30 13:37:46.137676 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:46.069931    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88af8672-a246-457e-9cfa-534c8d1d604d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c54f0f-6dc7885d9f-h9ghd\" (UID: \"88af8672-a246-457e-9cfa-534c8d1d604d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c54f0f-6dc7885d9f-h9ghd"
Oct 30 13:37:46.186521 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:46.186483    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tjxq\" (UniqueName: \"kubernetes.io/projected/88af8672-a246-457e-9cfa-534c8d1d604d-kube-api-access-8tjxq\") pod \"payload-pre-processing-e2e-ipp-b-c54f0f-6dc7885d9f-h9ghd\" (UID: \"88af8672-a246-457e-9cfa-534c8d1d604d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c54f0f-6dc7885d9f-h9ghd"
Oct 30 13:37:46.186823 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:46.139012    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88af8672-a246-457e-9cfa-534c8d1d604d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c54f0f-6dc7885d9f-h9ghd\" (UID: \"88af8672-a246-457e-9cfa-534c8d1d604d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c54f0f-6dc7885d9f-h9ghd"
Oct 30 13:37:46.193911 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:46.193879    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c54f0f-6f599994bb-ntwlx"]
Oct 30 13:37:46.210943 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:46.210894    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c54f0f-6f599994bb-ntwlx"]
Oct 30 13:37:46.211280 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:46.211210    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c54f0f-6f599994bb-ntwlx"
Oct 30 13:37:46.236740 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:46.236376    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tjxq\" (UniqueName: \"kubernetes.io/projected/88af8672-a246-457e-9cfa-534c8d1d604d-kube-api-access-8tjxq\") pod \"payload-pre-processing-e2e-ipp-b-c54f0f-6dc7885d9f-h9ghd\" (UID: \"88af8672-a246-457e-9cfa-534c8d1d604d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c54f0f-6dc7885d9f-h9ghd"
Oct 30 13:37:46.288251 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:46.288221    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/094f69e3-b678-4281-9d9b-c0a4b25d0895-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c54f0f-6f599994bb-ntwlx\" (UID: \"094f69e3-b678-4281-9d9b-c0a4b25d0895\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c54f0f-6f599994bb-ntwlx"
Oct 30 13:37:46.288449 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:46.288293    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr2wp\" (UniqueName: \"kubernetes.io/projected/094f69e3-b678-4281-9d9b-c0a4b25d0895-kube-api-access-lr2wp\") pod \"payload-processing-e2e-ipp-b-c54f0f-6f599994bb-ntwlx\" (UID: \"094f69e3-b678-4281-9d9b-c0a4b25d0895\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c54f0f-6f599994bb-ntwlx"
Oct 30 13:37:46.390431 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:46.390349    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/094f69e3-b678-4281-9d9b-c0a4b25d0895-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c54f0f-6f599994bb-ntwlx\" (UID: \"094f69e3-b678-4281-9d9b-c0a4b25d0895\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c54f0f-6f599994bb-ntwlx"
Oct 30 13:37:46.390431 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:46.390413    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr2wp\" (UniqueName: \"kubernetes.io/projected/094f69e3-b678-4281-9d9b-c0a4b25d0895-kube-api-access-lr2wp\") pod \"payload-processing-e2e-ipp-b-c54f0f-6f599994bb-ntwlx\" (UID: \"094f69e3-b678-4281-9d9b-c0a4b25d0895\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c54f0f-6f599994bb-ntwlx"
Oct 30 13:37:46.391128 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:46.391105    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/094f69e3-b678-4281-9d9b-c0a4b25d0895-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c54f0f-6f599994bb-ntwlx\" (UID: \"094f69e3-b678-4281-9d9b-c0a4b25d0895\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c54f0f-6f599994bb-ntwlx"
Oct 30 13:37:46.413842 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:46.413810    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr2wp\" (UniqueName: \"kubernetes.io/projected/094f69e3-b678-4281-9d9b-c0a4b25d0895-kube-api-access-lr2wp\") pod \"payload-processing-e2e-ipp-b-c54f0f-6f599994bb-ntwlx\" (UID: \"094f69e3-b678-4281-9d9b-c0a4b25d0895\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c54f0f-6f599994bb-ntwlx"
Oct 30 13:37:46.449177 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:46.449136    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c54f0f-6dc7885d9f-h9ghd"
Oct 30 13:37:46.558246 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:46.558198    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c54f0f-6f599994bb-ntwlx"
Oct 30 13:37:46.784032 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:46.784005    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c54f0f-6dc7885d9f-h9ghd"]
Oct 30 13:37:46.785183 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:37:46.785154    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88af8672_a246_457e_9cfa_534c8d1d604d.slice/crio-baaa43a30e120d3d307443df64f4c160fbe1f302c9d68e804d705f8e38f8f1d3 WatchSource:0}: Error finding container baaa43a30e120d3d307443df64f4c160fbe1f302c9d68e804d705f8e38f8f1d3: Status 404 returned error can't find the container with id baaa43a30e120d3d307443df64f4c160fbe1f302c9d68e804d705f8e38f8f1d3
Oct 30 13:37:46.874943 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:46.874918    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c54f0f-6f599994bb-ntwlx"]
Oct 30 13:37:46.876136 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:37:46.876111    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod094f69e3_b678_4281_9d9b_c0a4b25d0895.slice/crio-7005b971ab9f796b5cb1d90e9bf76260163d49b79add8afaca2a971c7e4f1ef5 WatchSource:0}: Error finding container 7005b971ab9f796b5cb1d90e9bf76260163d49b79add8afaca2a971c7e4f1ef5: Status 404 returned error can't find the container with id 7005b971ab9f796b5cb1d90e9bf76260163d49b79add8afaca2a971c7e4f1ef5
Oct 30 13:37:47.116977 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:47.116949    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c54f0f-6dc7885d9f-h9ghd" event={"ID":"88af8672-a246-457e-9cfa-534c8d1d604d","Type":"ContainerStarted","Data":"30344301d8d5cdb8c4ed00fbef02bdc48933aa0e2a787c2a5d84ab56a66eb7c4"}
Oct 30 13:37:47.117159 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:47.116997    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c54f0f-6dc7885d9f-h9ghd" event={"ID":"88af8672-a246-457e-9cfa-534c8d1d604d","Type":"ContainerStarted","Data":"baaa43a30e120d3d307443df64f4c160fbe1f302c9d68e804d705f8e38f8f1d3"}
Oct 30 13:37:47.117159 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:47.117041    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c54f0f-6dc7885d9f-h9ghd"
Oct 30 13:37:47.118655 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:47.118625    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c54f0f-6f599994bb-ntwlx" event={"ID":"094f69e3-b678-4281-9d9b-c0a4b25d0895","Type":"ContainerStarted","Data":"862e3ddccd982605540d1143036989b15d27bc21934c7242d3358f498eeac9f3"}
Oct 30 13:37:47.118790 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:47.118673    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c54f0f-6f599994bb-ntwlx" event={"ID":"094f69e3-b678-4281-9d9b-c0a4b25d0895","Type":"ContainerStarted","Data":"7005b971ab9f796b5cb1d90e9bf76260163d49b79add8afaca2a971c7e4f1ef5"}
Oct 30 13:37:47.118790 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:47.118716    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c54f0f-6f599994bb-ntwlx"
Oct 30 13:37:47.159958 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:47.159913    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c54f0f-6dc7885d9f-h9ghd" podStartSLOduration=2.159899816 podStartE2EDuration="2.159899816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:37:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:37:47.156327842 +0000 UTC m=+2766.322692490" watchObservedRunningTime="2026-10-30 13:37:47.159899816 +0000 UTC m=+2766.326264464"
Oct 30 13:37:47.200663 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:47.200618    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-c54f0f-6f599994bb-ntwlx" podStartSLOduration=2.200592217 podStartE2EDuration="2.200592217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:37:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:37:47.195842563 +0000 UTC m=+2766.362207210" watchObservedRunningTime="2026-10-30 13:37:47.200592217 +0000 UTC m=+2766.366956864"
Oct 30 13:37:47.472819 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:47.472737    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-fca422-69f9fc7b66-ktsvw"]
Oct 30 13:37:47.478247 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:47.478208    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-fca422-69f9fc7b66-ktsvw"
Oct 30 13:37:47.494555 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:47.494528    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-fca422\""
Oct 30 13:37:47.495940 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:47.495915    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-fca422-dockercfg-dzsws\""
Oct 30 13:37:47.500353 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:47.500326    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-fca422-69f9fc7b66-ktsvw"]
Oct 30 13:37:47.701619 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:47.699491    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl2kn\" (UniqueName: \"kubernetes.io/projected/4e1fd541-b715-4ba4-a160-bd86c2076d92-kube-api-access-kl2kn\") pod \"payload-processing-e2e-ipp-b-fca422-69f9fc7b66-ktsvw\" (UID: \"4e1fd541-b715-4ba4-a160-bd86c2076d92\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-fca422-69f9fc7b66-ktsvw"
Oct 30 13:37:47.701619 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:47.699930    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e1fd541-b715-4ba4-a160-bd86c2076d92-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-fca422-69f9fc7b66-ktsvw\" (UID: \"4e1fd541-b715-4ba4-a160-bd86c2076d92\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-fca422-69f9fc7b66-ktsvw"
Oct 30 13:37:47.802036 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:47.801924    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl2kn\" (UniqueName: \"kubernetes.io/projected/4e1fd541-b715-4ba4-a160-bd86c2076d92-kube-api-access-kl2kn\") pod \"payload-processing-e2e-ipp-b-fca422-69f9fc7b66-ktsvw\" (UID: \"4e1fd541-b715-4ba4-a160-bd86c2076d92\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-fca422-69f9fc7b66-ktsvw"
Oct 30 13:37:47.802648 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:47.802617    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e1fd541-b715-4ba4-a160-bd86c2076d92-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-fca422-69f9fc7b66-ktsvw\" (UID: \"4e1fd541-b715-4ba4-a160-bd86c2076d92\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-fca422-69f9fc7b66-ktsvw"
Oct 30 13:37:47.803885 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:47.803860    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e1fd541-b715-4ba4-a160-bd86c2076d92-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-fca422-69f9fc7b66-ktsvw\" (UID: \"4e1fd541-b715-4ba4-a160-bd86c2076d92\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-fca422-69f9fc7b66-ktsvw"
Oct 30 13:37:47.845356 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:47.845315    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl2kn\" (UniqueName: \"kubernetes.io/projected/4e1fd541-b715-4ba4-a160-bd86c2076d92-kube-api-access-kl2kn\") pod \"payload-processing-e2e-ipp-b-fca422-69f9fc7b66-ktsvw\" (UID: \"4e1fd541-b715-4ba4-a160-bd86c2076d92\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-fca422-69f9fc7b66-ktsvw"
Oct 30 13:37:48.097799 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:48.097719    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-fca422-69f9fc7b66-ktsvw"
Oct 30 13:37:48.410344 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:48.410314    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-fca422-69f9fc7b66-ktsvw"]
Oct 30 13:37:48.411813 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:37:48.411785    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e1fd541_b715_4ba4_a160_bd86c2076d92.slice/crio-46e6a7af5c5c6321ca0118f8ca3ba5c434715b48a0cc8d3d441e3c643a7bdd7b WatchSource:0}: Error finding container 46e6a7af5c5c6321ca0118f8ca3ba5c434715b48a0cc8d3d441e3c643a7bdd7b: Status 404 returned error can't find the container with id 46e6a7af5c5c6321ca0118f8ca3ba5c434715b48a0cc8d3d441e3c643a7bdd7b
Oct 30 13:37:49.138267 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:49.138221    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-fca422-69f9fc7b66-ktsvw" event={"ID":"4e1fd541-b715-4ba4-a160-bd86c2076d92","Type":"ContainerStarted","Data":"92ef052f857a14f10f76eb71294f0b152b2f5a332d88c8a7210c9fe130190f81"}
Oct 30 13:37:49.138788 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:49.138287    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-fca422-69f9fc7b66-ktsvw" event={"ID":"4e1fd541-b715-4ba4-a160-bd86c2076d92","Type":"ContainerStarted","Data":"46e6a7af5c5c6321ca0118f8ca3ba5c434715b48a0cc8d3d441e3c643a7bdd7b"}
Oct 30 13:37:49.138788 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:49.138312    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-fca422-69f9fc7b66-ktsvw"
Oct 30 13:37:49.184145 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:49.184096    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-fca422-69f9fc7b66-ktsvw" podStartSLOduration=2.184083284 podStartE2EDuration="2.184083284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:37:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:37:49.178586978 +0000 UTC m=+2768.344951625" watchObservedRunningTime="2026-10-30 13:37:49.184083284 +0000 UTC m=+2768.350447932"
Oct 30 13:37:58.131455 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:58.131423    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c54f0f-6f599994bb-ntwlx"
Oct 30 13:37:58.131958 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:58.131493    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c54f0f-6dc7885d9f-h9ghd"
Oct 30 13:37:58.212224 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:37:58.212186    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a8d3b8-689875d59-92xn4"]
Oct 30 13:38:00.235873 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:00.227620    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-fca422-69f9fc7b66-ktsvw"
Oct 30 13:38:03.219349 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:03.219231    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a8d3b8-689875d59-92xn4" podUID="7840586e-1003-414a-8ce9-b988bb4a34ac" containerName="payload-pre-processing" containerID="cri-o://288c6d42f9f8efc6efcdf101961bbf502b523699c63d39e1ec1a49a4d4654136" gracePeriod=25
Oct 30 13:38:03.367109 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:03.367080    4364 generic.go:356] "Generic (PLEG): container finished" podID="7840586e-1003-414a-8ce9-b988bb4a34ac" containerID="288c6d42f9f8efc6efcdf101961bbf502b523699c63d39e1ec1a49a4d4654136" exitCode=0
Oct 30 13:38:03.367248 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:03.367155    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a8d3b8-689875d59-92xn4" event={"ID":"7840586e-1003-414a-8ce9-b988bb4a34ac","Type":"ContainerDied","Data":"288c6d42f9f8efc6efcdf101961bbf502b523699c63d39e1ec1a49a4d4654136"}
Oct 30 13:38:03.543965 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:03.543939    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a8d3b8-689875d59-92xn4"
Oct 30 13:38:03.618587 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:03.618543    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7840586e-1003-414a-8ce9-b988bb4a34ac-kube-api-access-qtnhj\" (UniqueName: \"kubernetes.io/projected/7840586e-1003-414a-8ce9-b988bb4a34ac-kube-api-access-qtnhj\") pod \"7840586e-1003-414a-8ce9-b988bb4a34ac\" (UID: \"7840586e-1003-414a-8ce9-b988bb4a34ac\") "
Oct 30 13:38:03.618805 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:03.618653    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7840586e-1003-414a-8ce9-b988bb4a34ac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7840586e-1003-414a-8ce9-b988bb4a34ac-plugins-config-volume\") pod \"7840586e-1003-414a-8ce9-b988bb4a34ac\" (UID: \"7840586e-1003-414a-8ce9-b988bb4a34ac\") "
Oct 30 13:38:03.619416 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:03.619387    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7840586e-1003-414a-8ce9-b988bb4a34ac-plugins-config-volume" pod "7840586e-1003-414a-8ce9-b988bb4a34ac" (UID: "7840586e-1003-414a-8ce9-b988bb4a34ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:38:03.619667 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:03.619648    4364 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7840586e-1003-414a-8ce9-b988bb4a34ac-plugins-config-volume\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:38:03.621092 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:03.621068    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7840586e-1003-414a-8ce9-b988bb4a34ac-kube-api-access-qtnhj" pod "7840586e-1003-414a-8ce9-b988bb4a34ac" (UID: "7840586e-1003-414a-8ce9-b988bb4a34ac"). InnerVolumeSpecName "kube-api-access-qtnhj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:38:03.721169 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:03.721140    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtnhj\" (UniqueName: \"kubernetes.io/projected/7840586e-1003-414a-8ce9-b988bb4a34ac-kube-api-access-qtnhj\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:38:04.376682 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:04.376652    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a8d3b8-689875d59-92xn4"
Oct 30 13:38:04.377112 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:04.376658    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a8d3b8-689875d59-92xn4" event={"ID":"7840586e-1003-414a-8ce9-b988bb4a34ac","Type":"ContainerDied","Data":"ca7a0a9b9e0b88c3cb7f70397f7b92ba856ae22f566972a39d33e04ed3ede374"}
Oct 30 13:38:04.377112 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:04.376794    4364 scope.go:122] "RemoveContainer" containerID="288c6d42f9f8efc6efcdf101961bbf502b523699c63d39e1ec1a49a4d4654136"
Oct 30 13:38:04.431071 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:04.431037    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a8d3b8-689875d59-92xn4"]
Oct 30 13:38:04.442855 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:04.442826    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a8d3b8-689875d59-92xn4"]
Oct 30 13:38:04.972942 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:04.972903    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7840586e-1003-414a-8ce9-b988bb4a34ac" path="/var/lib/kubelet/pods/7840586e-1003-414a-8ce9-b988bb4a34ac/volumes"
Oct 30 13:38:11.505552 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:38:11.505506    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:38:11.593716 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:11.593682    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:38:18.628893 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:18.628386    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-78dc89-7fbb976874-xzkzv"]
Oct 30 13:38:18.639474 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:18.639446    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-78dc89-6bd5777667-5tn7b"]
Oct 30 13:38:19.181625 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:19.180459    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-mqpfw_310804ff-17ef-4a28-959c-47995bde35ca/payload-processing/0.log"
Oct 30 13:38:19.376624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:19.373885    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-mqpfw_310804ff-17ef-4a28-959c-47995bde35ca/payload-processing/0.log"
Oct 30 13:38:23.124188 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:23.124141    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s"]
Oct 30 13:38:23.125730 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:23.125702    4364 cpu_manager.go:414] "containerMap: removing container" podUID="7840586e-1003-414a-8ce9-b988bb4a34ac" containerName="payload-pre-processing"
Oct 30 13:38:23.125730 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:23.125729    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7840586e-1003-414a-8ce9-b988bb4a34ac" containerName="payload-pre-processing"
Oct 30 13:38:23.125976 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:23.125955    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="7840586e-1003-414a-8ce9-b988bb4a34ac" containerName="payload-pre-processing"
Oct 30 13:38:23.132577 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:23.132544    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s"
Oct 30 13:38:23.155920 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:23.155882    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-30c004-openshift-default-dockercfg-7lqcr\""
Oct 30 13:38:23.160748 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:23.160716    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s"]
Oct 30 13:38:23.166350 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:23.166315    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9a3c04-3da8-4efe-8daf-b4821c563a24-workload-socket\") pod \"e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s\" (UID: \"0a9a3c04-3da8-4efe-8daf-b4821c563a24\") " pod="openshift-ingress/e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s"
Oct 30 13:38:23.166525 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:23.166506    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9a3c04-3da8-4efe-8daf-b4821c563a24-credential-socket\") pod \"e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s\" (UID: \"0a9a3c04-3da8-4efe-8daf-b4821c563a24\") " pod="openshift-ingress/e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s"
Oct 30 13:38:23.166649 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:23.166630    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a9a3c04-3da8-4efe-8daf-b4821c563a24-istio-envoy\") pod \"e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s\" (UID: \"0a9a3c04-3da8-4efe-8daf-b4821c563a24\") " pod="openshift-ingress/e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s"
Oct 30 13:38:23.167380 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:23.166860    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a9a3c04-3da8-4efe-8daf-b4821c563a24-workload-certs\") pod \"e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s\" (UID: \"0a9a3c04-3da8-4efe-8daf-b4821c563a24\") " pod="openshift-ingress/e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s"
Oct 30 13:38:23.167380 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:23.167012    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kjzn\" (UniqueName: \"kubernetes.io/projected/0a9a3c04-3da8-4efe-8daf-b4821c563a24-kube-api-access-2kjzn\") pod \"e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s\" (UID: \"0a9a3c04-3da8-4efe-8daf-b4821c563a24\") " pod="openshift-ingress/e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s"
Oct 30 13:38:23.167380 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:23.167049    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a9a3c04-3da8-4efe-8daf-b4821c563a24-istio-token\") pod \"e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s\" (UID: \"0a9a3c04-3da8-4efe-8daf-b4821c563a24\") " pod="openshift-ingress/e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s"
Oct 30 13:38:23.167380 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:23.167138    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a9a3c04-3da8-4efe-8daf-b4821c563a24-istio-data\") pod \"e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s\" (UID: \"0a9a3c04-3da8-4efe-8daf-b4821c563a24\") " pod="openshift-ingress/e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s"
Oct 30 13:38:23.167380 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:23.167171    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a9a3c04-3da8-4efe-8daf-b4821c563a24-istio-podinfo\") pod \"e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s\" (UID: \"0a9a3c04-3da8-4efe-8daf-b4821c563a24\") " pod="openshift-ingress/e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s"
Oct 30 13:38:23.167380 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:23.167217    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a9a3c04-3da8-4efe-8daf-b4821c563a24-istiod-ca-cert\") pod \"e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s\" (UID: \"0a9a3c04-3da8-4efe-8daf-b4821c563a24\") " pod="openshift-ingress/e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s"
Oct 30 13:38:23.268809 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:23.268778    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kjzn\" (UniqueName: \"kubernetes.io/projected/0a9a3c04-3da8-4efe-8daf-b4821c563a24-kube-api-access-2kjzn\") pod \"e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s\" (UID: \"0a9a3c04-3da8-4efe-8daf-b4821c563a24\") " pod="openshift-ingress/e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s"
Oct 30 13:38:23.269003 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:23.268820    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a9a3c04-3da8-4efe-8daf-b4821c563a24-istio-token\") pod \"e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s\" (UID: \"0a9a3c04-3da8-4efe-8daf-b4821c563a24\") " pod="openshift-ingress/e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s"
Oct 30 13:38:23.269003 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:23.268888    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a9a3c04-3da8-4efe-8daf-b4821c563a24-istio-data\") pod \"e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s\" (UID: \"0a9a3c04-3da8-4efe-8daf-b4821c563a24\") " pod="openshift-ingress/e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s"
Oct 30 13:38:23.269003 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:23.268919    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a9a3c04-3da8-4efe-8daf-b4821c563a24-istio-podinfo\") pod \"e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s\" (UID: \"0a9a3c04-3da8-4efe-8daf-b4821c563a24\") " pod="openshift-ingress/e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s"
Oct 30 13:38:23.269003 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:23.268957    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a9a3c04-3da8-4efe-8daf-b4821c563a24-istiod-ca-cert\") pod \"e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s\" (UID: \"0a9a3c04-3da8-4efe-8daf-b4821c563a24\") " pod="openshift-ingress/e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s"
Oct 30 13:38:23.269003 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:23.268997    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9a3c04-3da8-4efe-8daf-b4821c563a24-workload-socket\") pod \"e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s\" (UID: \"0a9a3c04-3da8-4efe-8daf-b4821c563a24\") " pod="openshift-ingress/e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s"
Oct 30 13:38:23.269247 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:23.269039    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9a3c04-3da8-4efe-8daf-b4821c563a24-credential-socket\") pod \"e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s\" (UID: \"0a9a3c04-3da8-4efe-8daf-b4821c563a24\") " pod="openshift-ingress/e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s"
Oct 30 13:38:23.269247 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:23.269101    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a9a3c04-3da8-4efe-8daf-b4821c563a24-istio-envoy\") pod \"e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s\" (UID: \"0a9a3c04-3da8-4efe-8daf-b4821c563a24\") " pod="openshift-ingress/e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s"
Oct 30 13:38:23.269440 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:23.269407    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a9a3c04-3da8-4efe-8daf-b4821c563a24-istio-data\") pod \"e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s\" (UID: \"0a9a3c04-3da8-4efe-8daf-b4821c563a24\") " pod="openshift-ingress/e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s"
Oct 30 13:38:23.269557 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:23.269418    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a9a3c04-3da8-4efe-8daf-b4821c563a24-workload-certs\") pod \"e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s\" (UID: \"0a9a3c04-3da8-4efe-8daf-b4821c563a24\") " pod="openshift-ingress/e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s"
Oct 30 13:38:23.269557 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:23.269486    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9a3c04-3da8-4efe-8daf-b4821c563a24-workload-socket\") pod \"e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s\" (UID: \"0a9a3c04-3da8-4efe-8daf-b4821c563a24\") " pod="openshift-ingress/e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s"
Oct 30 13:38:23.269716 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:23.269574    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9a3c04-3da8-4efe-8daf-b4821c563a24-credential-socket\") pod \"e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s\" (UID: \"0a9a3c04-3da8-4efe-8daf-b4821c563a24\") " pod="openshift-ingress/e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s"
Oct 30 13:38:23.269782 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:23.269755    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a9a3c04-3da8-4efe-8daf-b4821c563a24-workload-certs\") pod \"e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s\" (UID: \"0a9a3c04-3da8-4efe-8daf-b4821c563a24\") " pod="openshift-ingress/e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s"
Oct 30 13:38:23.269843 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:23.269808    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a9a3c04-3da8-4efe-8daf-b4821c563a24-istiod-ca-cert\") pod \"e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s\" (UID: \"0a9a3c04-3da8-4efe-8daf-b4821c563a24\") " pod="openshift-ingress/e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s"
Oct 30 13:38:23.271827 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:23.271802    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a9a3c04-3da8-4efe-8daf-b4821c563a24-istio-podinfo\") pod \"e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s\" (UID: \"0a9a3c04-3da8-4efe-8daf-b4821c563a24\") " pod="openshift-ingress/e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s"
Oct 30 13:38:23.271982 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:23.271965    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a9a3c04-3da8-4efe-8daf-b4821c563a24-istio-envoy\") pod \"e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s\" (UID: \"0a9a3c04-3da8-4efe-8daf-b4821c563a24\") " pod="openshift-ingress/e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s"
Oct 30 13:38:23.294687 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:23.294626    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a9a3c04-3da8-4efe-8daf-b4821c563a24-istio-token\") pod \"e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s\" (UID: \"0a9a3c04-3da8-4efe-8daf-b4821c563a24\") " pod="openshift-ingress/e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s"
Oct 30 13:38:23.295926 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:23.295876    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kjzn\" (UniqueName: \"kubernetes.io/projected/0a9a3c04-3da8-4efe-8daf-b4821c563a24-kube-api-access-2kjzn\") pod \"e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s\" (UID: \"0a9a3c04-3da8-4efe-8daf-b4821c563a24\") " pod="openshift-ingress/e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s"
Oct 30 13:38:23.473895 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:23.473581    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s"
Oct 30 13:38:23.632915 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:23.632684    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-78dc89-7fbb976874-xzkzv" podUID="b4a7e4ba-b7e7-4f0f-aae8-460156351389" containerName="payload-processing" containerID="cri-o://1e86e262567304c06b16e7a0fa1f3ce2cae9a940e3226d01876629f404c00c5f" gracePeriod=25
Oct 30 13:38:23.641708 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:23.640994    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78dc89-6bd5777667-5tn7b" podUID="cb725a71-8f08-4c47-a8a6-79242757e6b2" containerName="payload-pre-processing" containerID="cri-o://f9aaae724c4f9dee9cce90e7ab20abd99bcf64f90d336e5eecd95253f3a1a777" gracePeriod=25
Oct 30 13:38:24.011468 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:24.011418    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s"]
Oct 30 13:38:24.016809 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:24.016736    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-78dc89-7fbb976874-xzkzv"
Oct 30 13:38:24.018129 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:38:24.018101    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a9a3c04_3da8_4efe_8daf_b4821c563a24.slice/crio-d8ae01058b6db37922479aa9c5b2d466ed2b13b55a1b9f15fc5141fe693c32ff WatchSource:0}: Error finding container d8ae01058b6db37922479aa9c5b2d466ed2b13b55a1b9f15fc5141fe693c32ff: Status 404 returned error can't find the container with id d8ae01058b6db37922479aa9c5b2d466ed2b13b55a1b9f15fc5141fe693c32ff
Oct 30 13:38:24.021069 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:24.021036    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:38:24.021178 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:24.021114    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:38:24.021178 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:24.021159    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:38:24.077453 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:24.077425    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78dc89-6bd5777667-5tn7b"
Oct 30 13:38:24.106886 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:24.106856    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4a7e4ba-b7e7-4f0f-aae8-460156351389-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4a7e4ba-b7e7-4f0f-aae8-460156351389-plugins-config-volume\") pod \"b4a7e4ba-b7e7-4f0f-aae8-460156351389\" (UID: \"b4a7e4ba-b7e7-4f0f-aae8-460156351389\") "
Oct 30 13:38:24.107026 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:24.106917    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4a7e4ba-b7e7-4f0f-aae8-460156351389-kube-api-access-zntpr\" (UniqueName: \"kubernetes.io/projected/b4a7e4ba-b7e7-4f0f-aae8-460156351389-kube-api-access-zntpr\") pod \"b4a7e4ba-b7e7-4f0f-aae8-460156351389\" (UID: \"b4a7e4ba-b7e7-4f0f-aae8-460156351389\") "
Oct 30 13:38:24.107906 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:24.107344    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4a7e4ba-b7e7-4f0f-aae8-460156351389-plugins-config-volume" pod "b4a7e4ba-b7e7-4f0f-aae8-460156351389" (UID: "b4a7e4ba-b7e7-4f0f-aae8-460156351389"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:38:24.108030 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:24.107943    4364 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4a7e4ba-b7e7-4f0f-aae8-460156351389-plugins-config-volume\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:38:24.110016 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:24.109988    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4a7e4ba-b7e7-4f0f-aae8-460156351389-kube-api-access-zntpr" pod "b4a7e4ba-b7e7-4f0f-aae8-460156351389" (UID: "b4a7e4ba-b7e7-4f0f-aae8-460156351389"). InnerVolumeSpecName "kube-api-access-zntpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:38:24.208412 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:24.208383    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb725a71-8f08-4c47-a8a6-79242757e6b2-kube-api-access-mvqg8\" (UniqueName: \"kubernetes.io/projected/cb725a71-8f08-4c47-a8a6-79242757e6b2-kube-api-access-mvqg8\") pod \"cb725a71-8f08-4c47-a8a6-79242757e6b2\" (UID: \"cb725a71-8f08-4c47-a8a6-79242757e6b2\") "
Oct 30 13:38:24.208915 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:24.208447    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb725a71-8f08-4c47-a8a6-79242757e6b2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb725a71-8f08-4c47-a8a6-79242757e6b2-plugins-config-volume\") pod \"cb725a71-8f08-4c47-a8a6-79242757e6b2\" (UID: \"cb725a71-8f08-4c47-a8a6-79242757e6b2\") "
Oct 30 13:38:24.208988 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:24.208927    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb725a71-8f08-4c47-a8a6-79242757e6b2-plugins-config-volume" pod "cb725a71-8f08-4c47-a8a6-79242757e6b2" (UID: "cb725a71-8f08-4c47-a8a6-79242757e6b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:38:24.209238 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:24.209218    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zntpr\" (UniqueName: \"kubernetes.io/projected/b4a7e4ba-b7e7-4f0f-aae8-460156351389-kube-api-access-zntpr\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:38:24.209303 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:24.209242    4364 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb725a71-8f08-4c47-a8a6-79242757e6b2-plugins-config-volume\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:38:24.210647 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:24.210571    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb725a71-8f08-4c47-a8a6-79242757e6b2-kube-api-access-mvqg8" pod "cb725a71-8f08-4c47-a8a6-79242757e6b2" (UID: "cb725a71-8f08-4c47-a8a6-79242757e6b2"). InnerVolumeSpecName "kube-api-access-mvqg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:38:24.310708 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:24.310676    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvqg8\" (UniqueName: \"kubernetes.io/projected/cb725a71-8f08-4c47-a8a6-79242757e6b2-kube-api-access-mvqg8\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:38:24.779659 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:24.779588    4364 generic.go:356] "Generic (PLEG): container finished" podID="cb725a71-8f08-4c47-a8a6-79242757e6b2" containerID="f9aaae724c4f9dee9cce90e7ab20abd99bcf64f90d336e5eecd95253f3a1a777" exitCode=0
Oct 30 13:38:24.779659 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:24.779655    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78dc89-6bd5777667-5tn7b"
Oct 30 13:38:24.779936 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:24.779628    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78dc89-6bd5777667-5tn7b" event={"ID":"cb725a71-8f08-4c47-a8a6-79242757e6b2","Type":"ContainerDied","Data":"f9aaae724c4f9dee9cce90e7ab20abd99bcf64f90d336e5eecd95253f3a1a777"}
Oct 30 13:38:24.779936 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:24.779762    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78dc89-6bd5777667-5tn7b" event={"ID":"cb725a71-8f08-4c47-a8a6-79242757e6b2","Type":"ContainerDied","Data":"7a1e13a9bfa81897c3c3c16dba874463897e74bca7d8f0056f0b922517ad9510"}
Oct 30 13:38:24.779936 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:24.779792    4364 scope.go:122] "RemoveContainer" containerID="f9aaae724c4f9dee9cce90e7ab20abd99bcf64f90d336e5eecd95253f3a1a777"
Oct 30 13:38:24.781429 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:24.781405    4364 generic.go:356] "Generic (PLEG): container finished" podID="b4a7e4ba-b7e7-4f0f-aae8-460156351389" containerID="1e86e262567304c06b16e7a0fa1f3ce2cae9a940e3226d01876629f404c00c5f" exitCode=0
Oct 30 13:38:24.781687 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:24.781668    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-78dc89-7fbb976874-xzkzv" event={"ID":"b4a7e4ba-b7e7-4f0f-aae8-460156351389","Type":"ContainerDied","Data":"1e86e262567304c06b16e7a0fa1f3ce2cae9a940e3226d01876629f404c00c5f"}
Oct 30 13:38:24.781767 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:24.781710    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-78dc89-7fbb976874-xzkzv" event={"ID":"b4a7e4ba-b7e7-4f0f-aae8-460156351389","Type":"ContainerDied","Data":"a80f38cdd3de534a07ddb4d0e92aa970dd418526b2b5cc415b8f15f16572cd2a"}
Oct 30 13:38:24.781811 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:24.781779    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-78dc89-7fbb976874-xzkzv"
Oct 30 13:38:24.784570 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:24.784547    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s" event={"ID":"0a9a3c04-3da8-4efe-8daf-b4821c563a24","Type":"ContainerStarted","Data":"70f543b1bcf791669112619fbd29585df08d5c8efeae09bb40a437fe62593169"}
Oct 30 13:38:24.784722 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:24.784622    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s" event={"ID":"0a9a3c04-3da8-4efe-8daf-b4821c563a24","Type":"ContainerStarted","Data":"d8ae01058b6db37922479aa9c5b2d466ed2b13b55a1b9f15fc5141fe693c32ff"}
Oct 30 13:38:24.793384 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:24.793367    4364 scope.go:122] "RemoveContainer" containerID="f9aaae724c4f9dee9cce90e7ab20abd99bcf64f90d336e5eecd95253f3a1a777"
Oct 30 13:38:24.793670 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:38:24.793650    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9aaae724c4f9dee9cce90e7ab20abd99bcf64f90d336e5eecd95253f3a1a777\": container with ID starting with f9aaae724c4f9dee9cce90e7ab20abd99bcf64f90d336e5eecd95253f3a1a777 not found: ID does not exist" containerID="f9aaae724c4f9dee9cce90e7ab20abd99bcf64f90d336e5eecd95253f3a1a777"
Oct 30 13:38:24.793760 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:24.793680    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9aaae724c4f9dee9cce90e7ab20abd99bcf64f90d336e5eecd95253f3a1a777"} err="failed to get container status \"f9aaae724c4f9dee9cce90e7ab20abd99bcf64f90d336e5eecd95253f3a1a777\": rpc error: code = NotFound desc = could not find container \"f9aaae724c4f9dee9cce90e7ab20abd99bcf64f90d336e5eecd95253f3a1a777\": container with ID starting with f9aaae724c4f9dee9cce90e7ab20abd99bcf64f90d336e5eecd95253f3a1a777 not found: ID does not exist"
Oct 30 13:38:24.793760 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:24.793703    4364 scope.go:122] "RemoveContainer" containerID="1e86e262567304c06b16e7a0fa1f3ce2cae9a940e3226d01876629f404c00c5f"
Oct 30 13:38:24.806756 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:24.806737    4364 scope.go:122] "RemoveContainer" containerID="1e86e262567304c06b16e7a0fa1f3ce2cae9a940e3226d01876629f404c00c5f"
Oct 30 13:38:24.807048 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:38:24.807028    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e86e262567304c06b16e7a0fa1f3ce2cae9a940e3226d01876629f404c00c5f\": container with ID starting with 1e86e262567304c06b16e7a0fa1f3ce2cae9a940e3226d01876629f404c00c5f not found: ID does not exist" containerID="1e86e262567304c06b16e7a0fa1f3ce2cae9a940e3226d01876629f404c00c5f"
Oct 30 13:38:24.807135 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:24.807052    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e86e262567304c06b16e7a0fa1f3ce2cae9a940e3226d01876629f404c00c5f"} err="failed to get container status \"1e86e262567304c06b16e7a0fa1f3ce2cae9a940e3226d01876629f404c00c5f\": rpc error: code = NotFound desc = could not find container \"1e86e262567304c06b16e7a0fa1f3ce2cae9a940e3226d01876629f404c00c5f\": container with ID starting with 1e86e262567304c06b16e7a0fa1f3ce2cae9a940e3226d01876629f404c00c5f not found: ID does not exist"
Oct 30 13:38:24.838414 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:24.838382    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-78dc89-7fbb976874-xzkzv"]
Oct 30 13:38:24.850094 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:24.850070    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-78dc89-7fbb976874-xzkzv"]
Oct 30 13:38:24.888249 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:24.888187    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s" podStartSLOduration=1.8881687409999999 podStartE2EDuration="1.888168741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:38:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:38:24.882044194 +0000 UTC m=+2804.048408843" watchObservedRunningTime="2026-10-30 13:38:24.888168741 +0000 UTC m=+2804.054533393"
Oct 30 13:38:24.923410 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:24.923378    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-78dc89-6bd5777667-5tn7b"]
Oct 30 13:38:24.930430 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:24.930404    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-78dc89-6bd5777667-5tn7b"]
Oct 30 13:38:24.974367 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:24.974337    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4a7e4ba-b7e7-4f0f-aae8-460156351389" path="/var/lib/kubelet/pods/b4a7e4ba-b7e7-4f0f-aae8-460156351389/volumes"
Oct 30 13:38:24.974841 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:24.974826    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb725a71-8f08-4c47-a8a6-79242757e6b2" path="/var/lib/kubelet/pods/cb725a71-8f08-4c47-a8a6-79242757e6b2/volumes"
Oct 30 13:38:25.475091 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:25.475052    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s"
Oct 30 13:38:25.482385 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:25.482355    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s"
Oct 30 13:38:25.795401 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:25.795309    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s"
Oct 30 13:38:25.796546 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:25.796524    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s"
Oct 30 13:38:26.098767 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:26.097436    4364 patch_prober.go:29] interesting pod/e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": read tcp 10.133.0.2:46392->10.133.0.240:15021: read: connection reset by peer" start-of-body=
Oct 30 13:38:26.098767 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:26.097512    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6" podUID="864a2c8d-97ba-4141-957c-0810628ba6ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": read tcp 10.133.0.2:46392->10.133.0.240:15021: read: connection reset by peer"
Oct 30 13:38:26.806149 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:26.806118    4364 generic.go:356] "Generic (PLEG): container finished" podID="864a2c8d-97ba-4141-957c-0810628ba6ec" containerID="385f1fb76fa6f0e60e13865dfc171e43478952e68c76add876ed08e5bbdfe068" exitCode=0
Oct 30 13:38:26.806591 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:26.806158    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6" event={"ID":"864a2c8d-97ba-4141-957c-0810628ba6ec","Type":"ContainerDied","Data":"385f1fb76fa6f0e60e13865dfc171e43478952e68c76add876ed08e5bbdfe068"}
Oct 30 13:38:26.806759 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:26.806741    4364 scope.go:122] "RemoveContainer" containerID="385f1fb76fa6f0e60e13865dfc171e43478952e68c76add876ed08e5bbdfe068"
Oct 30 13:38:26.808476 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:26.808444    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:38:26.808617 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:26.808531    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:38:26.808617 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:26.808582    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:38:27.866622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:27.866382    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6" event={"ID":"864a2c8d-97ba-4141-957c-0810628ba6ec","Type":"ContainerStarted","Data":"6a6aefb24e9046bca7c428edea73a3cde0547e75288bd6475cf1ad8eef9f3be9"}
Oct 30 13:38:27.889620 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:27.887221    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6"
Oct 30 13:38:27.889620 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:27.887508    4364 patch_prober.go:29] interesting pod/e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Oct 30 13:38:27.889620 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:27.887553    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6" podUID="864a2c8d-97ba-4141-957c-0810628ba6ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Oct 30 13:38:28.892002 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:28.891965    4364 patch_prober.go:29] interesting pod/e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Oct 30 13:38:28.892638 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:28.892586    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6" podUID="864a2c8d-97ba-4141-957c-0810628ba6ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Oct 30 13:38:29.414098 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:29.414055    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-30c004-84d79b8679-ndx6w"]
Oct 30 13:38:29.416185 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:29.416127    4364 cpu_manager.go:414] "containerMap: removing container" podUID="cb725a71-8f08-4c47-a8a6-79242757e6b2" containerName="payload-pre-processing"
Oct 30 13:38:29.416185 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:29.416183    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb725a71-8f08-4c47-a8a6-79242757e6b2" containerName="payload-pre-processing"
Oct 30 13:38:29.416416 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:29.416223    4364 cpu_manager.go:414] "containerMap: removing container" podUID="b4a7e4ba-b7e7-4f0f-aae8-460156351389" containerName="payload-processing"
Oct 30 13:38:29.416416 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:29.416230    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4a7e4ba-b7e7-4f0f-aae8-460156351389" containerName="payload-processing"
Oct 30 13:38:29.416791 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:29.416767    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb725a71-8f08-4c47-a8a6-79242757e6b2" containerName="payload-pre-processing"
Oct 30 13:38:29.416905 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:29.416807    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4a7e4ba-b7e7-4f0f-aae8-460156351389" containerName="payload-processing"
Oct 30 13:38:29.424929 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:29.424900    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30c004-84d79b8679-ndx6w"
Oct 30 13:38:29.433631 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:29.433585    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-30c004-84d79b8679-ndx6w"]
Oct 30 13:38:29.447624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:29.445301    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-30c004\""
Oct 30 13:38:29.466513 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:29.461267    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qk5f\" (UniqueName: \"kubernetes.io/projected/32995da4-65f2-4605-95e9-c2cf569f5743-kube-api-access-8qk5f\") pod \"payload-pre-processing-e2e-ipp-b-30c004-84d79b8679-ndx6w\" (UID: \"32995da4-65f2-4605-95e9-c2cf569f5743\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30c004-84d79b8679-ndx6w"
Oct 30 13:38:29.466513 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:29.461378    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32995da4-65f2-4605-95e9-c2cf569f5743-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-30c004-84d79b8679-ndx6w\" (UID: \"32995da4-65f2-4605-95e9-c2cf569f5743\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30c004-84d79b8679-ndx6w"
Oct 30 13:38:29.486791 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:29.469921    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-30c004-dockercfg-w2wp6\""
Oct 30 13:38:29.678739 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:29.596634    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qk5f\" (UniqueName: \"kubernetes.io/projected/32995da4-65f2-4605-95e9-c2cf569f5743-kube-api-access-8qk5f\") pod \"payload-pre-processing-e2e-ipp-b-30c004-84d79b8679-ndx6w\" (UID: \"32995da4-65f2-4605-95e9-c2cf569f5743\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30c004-84d79b8679-ndx6w"
Oct 30 13:38:29.685688 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:29.679055    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32995da4-65f2-4605-95e9-c2cf569f5743-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-30c004-84d79b8679-ndx6w\" (UID: \"32995da4-65f2-4605-95e9-c2cf569f5743\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30c004-84d79b8679-ndx6w"
Oct 30 13:38:29.699624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:29.680816    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32995da4-65f2-4605-95e9-c2cf569f5743-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-30c004-84d79b8679-ndx6w\" (UID: \"32995da4-65f2-4605-95e9-c2cf569f5743\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30c004-84d79b8679-ndx6w"
Oct 30 13:38:29.743113 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:29.697686    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-30c004-d665cf8b-f56dn"]
Oct 30 13:38:29.785625 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:29.760625    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-30c004-d665cf8b-f56dn"
Oct 30 13:38:29.889939 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:29.887768    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-30c004-d665cf8b-f56dn"]
Oct 30 13:38:29.892096 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:29.891039    4364 patch_prober.go:29] interesting pod/e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Oct 30 13:38:29.892096 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:29.891097    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6" podUID="864a2c8d-97ba-4141-957c-0810628ba6ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Oct 30 13:38:29.913919 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:29.913848    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qk5f\" (UniqueName: \"kubernetes.io/projected/32995da4-65f2-4605-95e9-c2cf569f5743-kube-api-access-8qk5f\") pod \"payload-pre-processing-e2e-ipp-b-30c004-84d79b8679-ndx6w\" (UID: \"32995da4-65f2-4605-95e9-c2cf569f5743\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30c004-84d79b8679-ndx6w"
Oct 30 13:38:29.939956 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:29.939874    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e262eff8-700f-4aea-b6c4-1c76a9252726-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-30c004-d665cf8b-f56dn\" (UID: \"e262eff8-700f-4aea-b6c4-1c76a9252726\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-30c004-d665cf8b-f56dn"
Oct 30 13:38:29.940114 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:29.940079    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wj8t\" (UniqueName: \"kubernetes.io/projected/e262eff8-700f-4aea-b6c4-1c76a9252726-kube-api-access-4wj8t\") pod \"payload-processing-e2e-ipp-b-30c004-d665cf8b-f56dn\" (UID: \"e262eff8-700f-4aea-b6c4-1c76a9252726\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-30c004-d665cf8b-f56dn"
Oct 30 13:38:30.042779 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:30.042317    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wj8t\" (UniqueName: \"kubernetes.io/projected/e262eff8-700f-4aea-b6c4-1c76a9252726-kube-api-access-4wj8t\") pod \"payload-processing-e2e-ipp-b-30c004-d665cf8b-f56dn\" (UID: \"e262eff8-700f-4aea-b6c4-1c76a9252726\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-30c004-d665cf8b-f56dn"
Oct 30 13:38:30.077630 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:30.053035    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30c004-84d79b8679-ndx6w"
Oct 30 13:38:30.078187 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:30.073393    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wj8t\" (UniqueName: \"kubernetes.io/projected/e262eff8-700f-4aea-b6c4-1c76a9252726-kube-api-access-4wj8t\") pod \"payload-processing-e2e-ipp-b-30c004-d665cf8b-f56dn\" (UID: \"e262eff8-700f-4aea-b6c4-1c76a9252726\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-30c004-d665cf8b-f56dn"
Oct 30 13:38:30.096207 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:30.078742    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e262eff8-700f-4aea-b6c4-1c76a9252726-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-30c004-d665cf8b-f56dn\" (UID: \"e262eff8-700f-4aea-b6c4-1c76a9252726\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-30c004-d665cf8b-f56dn"
Oct 30 13:38:30.105919 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:30.098056    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e262eff8-700f-4aea-b6c4-1c76a9252726-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-30c004-d665cf8b-f56dn\" (UID: \"e262eff8-700f-4aea-b6c4-1c76a9252726\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-30c004-d665cf8b-f56dn"
Oct 30 13:38:30.253229 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:30.253178    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-30c004-d665cf8b-f56dn"
Oct 30 13:38:30.504893 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:30.504841    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-30c004-84d79b8679-ndx6w"]
Oct 30 13:38:30.506330 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:38:30.506296    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32995da4_65f2_4605_95e9_c2cf569f5743.slice/crio-15edc26f227b3fcd61a660e33fc7900031227b70b3c940c2e8e07e186ef37630 WatchSource:0}: Error finding container 15edc26f227b3fcd61a660e33fc7900031227b70b3c940c2e8e07e186ef37630: Status 404 returned error can't find the container with id 15edc26f227b3fcd61a660e33fc7900031227b70b3c940c2e8e07e186ef37630
Oct 30 13:38:30.611985 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:30.611957    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-30c004-d665cf8b-f56dn"]
Oct 30 13:38:30.613087 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:38:30.612393    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode262eff8_700f_4aea_b6c4_1c76a9252726.slice/crio-44cbaaeb3c552ca8cad0647e13fe11c4388e5db71832dffe1aa67ac6a3feb499 WatchSource:0}: Error finding container 44cbaaeb3c552ca8cad0647e13fe11c4388e5db71832dffe1aa67ac6a3feb499: Status 404 returned error can't find the container with id 44cbaaeb3c552ca8cad0647e13fe11c4388e5db71832dffe1aa67ac6a3feb499
Oct 30 13:38:30.906685 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:30.905574    4364 patch_prober.go:29] interesting pod/e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Oct 30 13:38:30.906685 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:30.905685    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6" podUID="864a2c8d-97ba-4141-957c-0810628ba6ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Oct 30 13:38:30.915581 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:30.915550    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-30c004-d665cf8b-f56dn" event={"ID":"e262eff8-700f-4aea-b6c4-1c76a9252726","Type":"ContainerStarted","Data":"a91c0cd9d98a8154c1e0f1b8216c8dc465527ccedf6fd22bad5121c08cc07d47"}
Oct 30 13:38:30.915766 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:30.915624    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-30c004-d665cf8b-f56dn" event={"ID":"e262eff8-700f-4aea-b6c4-1c76a9252726","Type":"ContainerStarted","Data":"44cbaaeb3c552ca8cad0647e13fe11c4388e5db71832dffe1aa67ac6a3feb499"}
Oct 30 13:38:30.915766 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:30.915657    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-30c004-d665cf8b-f56dn"
Oct 30 13:38:30.917439 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:30.917411    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30c004-84d79b8679-ndx6w" event={"ID":"32995da4-65f2-4605-95e9-c2cf569f5743","Type":"ContainerStarted","Data":"f5875af59333dca9e4f3988a0035f3a533a395f3a6493fe1ea7d31aa83a90393"}
Oct 30 13:38:30.917576 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:30.917461    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30c004-84d79b8679-ndx6w" event={"ID":"32995da4-65f2-4605-95e9-c2cf569f5743","Type":"ContainerStarted","Data":"15edc26f227b3fcd61a660e33fc7900031227b70b3c940c2e8e07e186ef37630"}
Oct 30 13:38:30.917576 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:30.917486    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30c004-84d79b8679-ndx6w"
Oct 30 13:38:30.960016 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:30.959973    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-30c004-d665cf8b-f56dn" podStartSLOduration=1.95995943 podStartE2EDuration="1.95995943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:38:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:38:30.95590997 +0000 UTC m=+2810.122274620" watchObservedRunningTime="2026-10-30 13:38:30.95995943 +0000 UTC m=+2810.126324078"
Oct 30 13:38:31.004968 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:31.004867    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30c004-84d79b8679-ndx6w" podStartSLOduration=2.004849276 podStartE2EDuration="2.004849276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:38:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:38:31.004098196 +0000 UTC m=+2810.170462848" watchObservedRunningTime="2026-10-30 13:38:31.004849276 +0000 UTC m=+2810.171213925"
Oct 30 13:38:31.910636 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:31.908781    4364 patch_prober.go:29] interesting pod/e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Oct 30 13:38:31.910636 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:31.908846    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6" podUID="864a2c8d-97ba-4141-957c-0810628ba6ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Oct 30 13:38:32.902623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:32.902069    4364 patch_prober.go:29] interesting pod/e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Oct 30 13:38:32.902623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:32.902138    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6" podUID="864a2c8d-97ba-4141-957c-0810628ba6ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Oct 30 13:38:33.303289 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:33.302769    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-082748-8fcd5c46-zpgwg"]
Oct 30 13:38:33.304471 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:33.304419    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-082748-8fcd5c46-zpgwg" podUID="f77f214e-e14c-46b1-b1a7-68c693f3a055" containerName="maas-api" containerID="cri-o://3884b0921b079c7497005e7a27735da7ebad64e586106c2f99022ed3a50d6875" gracePeriod=30
Oct 30 13:38:33.666997 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:33.666965    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-082748-8fcd5c46-zpgwg"
Oct 30 13:38:33.777086 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:33.777050    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f77f214e-e14c-46b1-b1a7-68c693f3a055-kube-api-access-bvvld\" (UniqueName: \"kubernetes.io/projected/f77f214e-e14c-46b1-b1a7-68c693f3a055-kube-api-access-bvvld\") pod \"f77f214e-e14c-46b1-b1a7-68c693f3a055\" (UID: \"f77f214e-e14c-46b1-b1a7-68c693f3a055\") "
Oct 30 13:38:33.777271 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:33.777092    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f77f214e-e14c-46b1-b1a7-68c693f3a055-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f77f214e-e14c-46b1-b1a7-68c693f3a055-openshift-service-ca\") pod \"f77f214e-e14c-46b1-b1a7-68c693f3a055\" (UID: \"f77f214e-e14c-46b1-b1a7-68c693f3a055\") "
Oct 30 13:38:33.777271 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:33.777177    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f77f214e-e14c-46b1-b1a7-68c693f3a055-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f77f214e-e14c-46b1-b1a7-68c693f3a055-metrics-certs\") pod \"f77f214e-e14c-46b1-b1a7-68c693f3a055\" (UID: \"f77f214e-e14c-46b1-b1a7-68c693f3a055\") "
Oct 30 13:38:33.777412 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:33.777393    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f77f214e-e14c-46b1-b1a7-68c693f3a055-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f77f214e-e14c-46b1-b1a7-68c693f3a055-maas-api-tls\") pod \"f77f214e-e14c-46b1-b1a7-68c693f3a055\" (UID: \"f77f214e-e14c-46b1-b1a7-68c693f3a055\") "
Oct 30 13:38:33.777491 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:33.777466    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f77f214e-e14c-46b1-b1a7-68c693f3a055-openshift-service-ca" pod "f77f214e-e14c-46b1-b1a7-68c693f3a055" (UID: "f77f214e-e14c-46b1-b1a7-68c693f3a055"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:38:33.778093 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:33.778068    4364 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f77f214e-e14c-46b1-b1a7-68c693f3a055-openshift-service-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:38:33.779509 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:33.779474    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f77f214e-e14c-46b1-b1a7-68c693f3a055-kube-api-access-bvvld" pod "f77f214e-e14c-46b1-b1a7-68c693f3a055" (UID: "f77f214e-e14c-46b1-b1a7-68c693f3a055"). InnerVolumeSpecName "kube-api-access-bvvld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:38:33.779655 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:33.779539    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f77f214e-e14c-46b1-b1a7-68c693f3a055-metrics-certs" pod "f77f214e-e14c-46b1-b1a7-68c693f3a055" (UID: "f77f214e-e14c-46b1-b1a7-68c693f3a055"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:38:33.779655 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:33.779572    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f77f214e-e14c-46b1-b1a7-68c693f3a055-maas-api-tls" pod "f77f214e-e14c-46b1-b1a7-68c693f3a055" (UID: "f77f214e-e14c-46b1-b1a7-68c693f3a055"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:38:33.916634 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:33.903578    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvvld\" (UniqueName: \"kubernetes.io/projected/f77f214e-e14c-46b1-b1a7-68c693f3a055-kube-api-access-bvvld\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:38:33.932211 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:33.932180    4364 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f77f214e-e14c-46b1-b1a7-68c693f3a055-metrics-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:38:33.932467 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:33.932447    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f77f214e-e14c-46b1-b1a7-68c693f3a055-maas-api-tls\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:38:34.089621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:34.085637    4364 generic.go:356] "Generic (PLEG): container finished" podID="f77f214e-e14c-46b1-b1a7-68c693f3a055" containerID="3884b0921b079c7497005e7a27735da7ebad64e586106c2f99022ed3a50d6875" exitCode=0
Oct 30 13:38:34.089621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:34.085881    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-082748-8fcd5c46-zpgwg" event={"ID":"f77f214e-e14c-46b1-b1a7-68c693f3a055","Type":"ContainerDied","Data":"3884b0921b079c7497005e7a27735da7ebad64e586106c2f99022ed3a50d6875"}
Oct 30 13:38:34.089621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:34.085958    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-082748-8fcd5c46-zpgwg" event={"ID":"f77f214e-e14c-46b1-b1a7-68c693f3a055","Type":"ContainerDied","Data":"18d37eb8b10afee5a352d965171bb67ded3441e7f048c5b8d6e69e7befa265a9"}
Oct 30 13:38:34.089621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:34.086006    4364 scope.go:122] "RemoveContainer" containerID="3884b0921b079c7497005e7a27735da7ebad64e586106c2f99022ed3a50d6875"
Oct 30 13:38:34.089621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:34.086267    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-082748-8fcd5c46-zpgwg"
Oct 30 13:38:34.101622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:34.100515    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6"
Oct 30 13:38:34.109709 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:34.109680    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6"
Oct 30 13:38:34.120535 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:34.120504    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6"
Oct 30 13:38:34.135632 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:34.135270    4364 scope.go:122] "RemoveContainer" containerID="3884b0921b079c7497005e7a27735da7ebad64e586106c2f99022ed3a50d6875"
Oct 30 13:38:34.135966 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:38:34.135867    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3884b0921b079c7497005e7a27735da7ebad64e586106c2f99022ed3a50d6875\": container with ID starting with 3884b0921b079c7497005e7a27735da7ebad64e586106c2f99022ed3a50d6875 not found: ID does not exist" containerID="3884b0921b079c7497005e7a27735da7ebad64e586106c2f99022ed3a50d6875"
Oct 30 13:38:34.135966 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:34.135908    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3884b0921b079c7497005e7a27735da7ebad64e586106c2f99022ed3a50d6875"} err="failed to get container status \"3884b0921b079c7497005e7a27735da7ebad64e586106c2f99022ed3a50d6875\": rpc error: code = NotFound desc = could not find container \"3884b0921b079c7497005e7a27735da7ebad64e586106c2f99022ed3a50d6875\": container with ID starting with 3884b0921b079c7497005e7a27735da7ebad64e586106c2f99022ed3a50d6875 not found: ID does not exist"
Oct 30 13:38:34.247627 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:34.247501    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-082748-8fcd5c46-zpgwg"]
Oct 30 13:38:34.257945 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:34.257911    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-082748-8fcd5c46-zpgwg"]
Oct 30 13:38:34.972769 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:34.972737    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f77f214e-e14c-46b1-b1a7-68c693f3a055" path="/var/lib/kubelet/pods/f77f214e-e14c-46b1-b1a7-68c693f3a055/volumes"
Oct 30 13:38:38.164014 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:38.163980    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c54f0f-5d446cb49d-dhfks"]
Oct 30 13:38:38.164870 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:38.164853    4364 cpu_manager.go:414] "containerMap: removing container" podUID="f77f214e-e14c-46b1-b1a7-68c693f3a055" containerName="maas-api"
Oct 30 13:38:38.164926 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:38.164871    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f77f214e-e14c-46b1-b1a7-68c693f3a055" containerName="maas-api"
Oct 30 13:38:38.165004 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:38.164995    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="f77f214e-e14c-46b1-b1a7-68c693f3a055" containerName="maas-api"
Oct 30 13:38:38.170440 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:38.170421    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c54f0f-5d446cb49d-dhfks"
Oct 30 13:38:38.201245 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:38.201213    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-c54f0f-serving-cert\""
Oct 30 13:38:38.219058 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:38.219024    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c54f0f-5d446cb49d-dhfks"]
Oct 30 13:38:38.335625 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:38.335566    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89934eb4-b154-412f-9f02-2f33c0fcb2e3-metrics-certs\") pod \"maas-api-e2e-ipp-b-c54f0f-5d446cb49d-dhfks\" (UID: \"89934eb4-b154-412f-9f02-2f33c0fcb2e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c54f0f-5d446cb49d-dhfks"
Oct 30 13:38:38.335799 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:38.335694    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgxn8\" (UniqueName: \"kubernetes.io/projected/89934eb4-b154-412f-9f02-2f33c0fcb2e3-kube-api-access-rgxn8\") pod \"maas-api-e2e-ipp-b-c54f0f-5d446cb49d-dhfks\" (UID: \"89934eb4-b154-412f-9f02-2f33c0fcb2e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c54f0f-5d446cb49d-dhfks"
Oct 30 13:38:38.335858 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:38.335839    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89934eb4-b154-412f-9f02-2f33c0fcb2e3-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c54f0f-5d446cb49d-dhfks\" (UID: \"89934eb4-b154-412f-9f02-2f33c0fcb2e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c54f0f-5d446cb49d-dhfks"
Oct 30 13:38:38.336060 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:38.336046    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89934eb4-b154-412f-9f02-2f33c0fcb2e3-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c54f0f-5d446cb49d-dhfks\" (UID: \"89934eb4-b154-412f-9f02-2f33c0fcb2e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c54f0f-5d446cb49d-dhfks"
Oct 30 13:38:38.457662 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:38.454790    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89934eb4-b154-412f-9f02-2f33c0fcb2e3-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c54f0f-5d446cb49d-dhfks\" (UID: \"89934eb4-b154-412f-9f02-2f33c0fcb2e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c54f0f-5d446cb49d-dhfks"
Oct 30 13:38:38.457662 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:38.455106    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89934eb4-b154-412f-9f02-2f33c0fcb2e3-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c54f0f-5d446cb49d-dhfks\" (UID: \"89934eb4-b154-412f-9f02-2f33c0fcb2e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c54f0f-5d446cb49d-dhfks"
Oct 30 13:38:38.457662 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:38.455192    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89934eb4-b154-412f-9f02-2f33c0fcb2e3-metrics-certs\") pod \"maas-api-e2e-ipp-b-c54f0f-5d446cb49d-dhfks\" (UID: \"89934eb4-b154-412f-9f02-2f33c0fcb2e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c54f0f-5d446cb49d-dhfks"
Oct 30 13:38:38.457662 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:38.455222    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgxn8\" (UniqueName: \"kubernetes.io/projected/89934eb4-b154-412f-9f02-2f33c0fcb2e3-kube-api-access-rgxn8\") pod \"maas-api-e2e-ipp-b-c54f0f-5d446cb49d-dhfks\" (UID: \"89934eb4-b154-412f-9f02-2f33c0fcb2e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c54f0f-5d446cb49d-dhfks"
Oct 30 13:38:38.457662 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:38.456672    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89934eb4-b154-412f-9f02-2f33c0fcb2e3-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c54f0f-5d446cb49d-dhfks\" (UID: \"89934eb4-b154-412f-9f02-2f33c0fcb2e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c54f0f-5d446cb49d-dhfks"
Oct 30 13:38:38.492272 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:38.492201    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89934eb4-b154-412f-9f02-2f33c0fcb2e3-metrics-certs\") pod \"maas-api-e2e-ipp-b-c54f0f-5d446cb49d-dhfks\" (UID: \"89934eb4-b154-412f-9f02-2f33c0fcb2e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c54f0f-5d446cb49d-dhfks"
Oct 30 13:38:38.504784 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:38.504688    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgxn8\" (UniqueName: \"kubernetes.io/projected/89934eb4-b154-412f-9f02-2f33c0fcb2e3-kube-api-access-rgxn8\") pod \"maas-api-e2e-ipp-b-c54f0f-5d446cb49d-dhfks\" (UID: \"89934eb4-b154-412f-9f02-2f33c0fcb2e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c54f0f-5d446cb49d-dhfks"
Oct 30 13:38:38.505205 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:38.505173    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89934eb4-b154-412f-9f02-2f33c0fcb2e3-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c54f0f-5d446cb49d-dhfks\" (UID: \"89934eb4-b154-412f-9f02-2f33c0fcb2e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c54f0f-5d446cb49d-dhfks"
Oct 30 13:38:38.785843 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:38.785766    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c54f0f-5d446cb49d-dhfks"
Oct 30 13:38:39.119059 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:38:39.119025    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89934eb4_b154_412f_9f02_2f33c0fcb2e3.slice/crio-57c1628165ba8e87a415a5f37005bf88d2ee4ffecd6e5959dfadaee2f5228266 WatchSource:0}: Error finding container 57c1628165ba8e87a415a5f37005bf88d2ee4ffecd6e5959dfadaee2f5228266: Status 404 returned error can't find the container with id 57c1628165ba8e87a415a5f37005bf88d2ee4ffecd6e5959dfadaee2f5228266
Oct 30 13:38:39.121567 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:39.121536    4364 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 13:38:39.127619 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:39.125882    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c54f0f-5d446cb49d-dhfks"]
Oct 30 13:38:40.081835 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:40.081799    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-fca422-767b78c8cb-fzvl2"]
Oct 30 13:38:40.087506 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:40.087486    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fca422-767b78c8cb-fzvl2"
Oct 30 13:38:40.102785 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:40.102765    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-fca422-serving-cert\""
Oct 30 13:38:40.120179 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:40.120150    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-fca422-767b78c8cb-fzvl2"]
Oct 30 13:38:40.153094 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:40.153066    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c54f0f-5d446cb49d-dhfks" event={"ID":"89934eb4-b154-412f-9f02-2f33c0fcb2e3","Type":"ContainerStarted","Data":"4010466c718b75c9dc075f672ac8e28eb063aa12a6abdeb108d45a58c049ca56"}
Oct 30 13:38:40.153273 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:40.153124    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c54f0f-5d446cb49d-dhfks" event={"ID":"89934eb4-b154-412f-9f02-2f33c0fcb2e3","Type":"ContainerStarted","Data":"57c1628165ba8e87a415a5f37005bf88d2ee4ffecd6e5959dfadaee2f5228266"}
Oct 30 13:38:40.153273 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:40.153154    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c54f0f-5d446cb49d-dhfks"
Oct 30 13:38:40.187950 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:40.187920    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80f1162e-fa26-45b9-8016-c4812edb5bad-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-fca422-767b78c8cb-fzvl2\" (UID: \"80f1162e-fa26-45b9-8016-c4812edb5bad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fca422-767b78c8cb-fzvl2"
Oct 30 13:38:40.188144 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:40.188112    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80f1162e-fa26-45b9-8016-c4812edb5bad-metrics-certs\") pod \"maas-api-e2e-ipp-b-fca422-767b78c8cb-fzvl2\" (UID: \"80f1162e-fa26-45b9-8016-c4812edb5bad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fca422-767b78c8cb-fzvl2"
Oct 30 13:38:40.188271 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:40.188229    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80f1162e-fa26-45b9-8016-c4812edb5bad-maas-api-tls\") pod \"maas-api-e2e-ipp-b-fca422-767b78c8cb-fzvl2\" (UID: \"80f1162e-fa26-45b9-8016-c4812edb5bad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fca422-767b78c8cb-fzvl2"
Oct 30 13:38:40.188322 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:40.188284    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvcw7\" (UniqueName: \"kubernetes.io/projected/80f1162e-fa26-45b9-8016-c4812edb5bad-kube-api-access-xvcw7\") pod \"maas-api-e2e-ipp-b-fca422-767b78c8cb-fzvl2\" (UID: \"80f1162e-fa26-45b9-8016-c4812edb5bad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fca422-767b78c8cb-fzvl2"
Oct 30 13:38:40.207238 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:40.207193    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c54f0f-5d446cb49d-dhfks" podStartSLOduration=1.87722199 podStartE2EDuration="2.207178032s" totalImagesPullingTime="329.956042ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:38:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:38:40.202786921 +0000 UTC m=+2819.369151569" watchObservedRunningTime="2026-10-30 13:38:40.207178032 +0000 UTC m=+2819.373542699"
Oct 30 13:38:40.290158 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:40.290130    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80f1162e-fa26-45b9-8016-c4812edb5bad-metrics-certs\") pod \"maas-api-e2e-ipp-b-fca422-767b78c8cb-fzvl2\" (UID: \"80f1162e-fa26-45b9-8016-c4812edb5bad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fca422-767b78c8cb-fzvl2"
Oct 30 13:38:40.290326 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:40.290248    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80f1162e-fa26-45b9-8016-c4812edb5bad-maas-api-tls\") pod \"maas-api-e2e-ipp-b-fca422-767b78c8cb-fzvl2\" (UID: \"80f1162e-fa26-45b9-8016-c4812edb5bad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fca422-767b78c8cb-fzvl2"
Oct 30 13:38:40.290420 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:40.290342    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvcw7\" (UniqueName: \"kubernetes.io/projected/80f1162e-fa26-45b9-8016-c4812edb5bad-kube-api-access-xvcw7\") pod \"maas-api-e2e-ipp-b-fca422-767b78c8cb-fzvl2\" (UID: \"80f1162e-fa26-45b9-8016-c4812edb5bad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fca422-767b78c8cb-fzvl2"
Oct 30 13:38:40.290717 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:40.290690    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80f1162e-fa26-45b9-8016-c4812edb5bad-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-fca422-767b78c8cb-fzvl2\" (UID: \"80f1162e-fa26-45b9-8016-c4812edb5bad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fca422-767b78c8cb-fzvl2"
Oct 30 13:38:40.291397 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:40.291372    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80f1162e-fa26-45b9-8016-c4812edb5bad-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-fca422-767b78c8cb-fzvl2\" (UID: \"80f1162e-fa26-45b9-8016-c4812edb5bad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fca422-767b78c8cb-fzvl2"
Oct 30 13:38:40.292857 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:40.292835    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80f1162e-fa26-45b9-8016-c4812edb5bad-maas-api-tls\") pod \"maas-api-e2e-ipp-b-fca422-767b78c8cb-fzvl2\" (UID: \"80f1162e-fa26-45b9-8016-c4812edb5bad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fca422-767b78c8cb-fzvl2"
Oct 30 13:38:40.292947 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:40.292888    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80f1162e-fa26-45b9-8016-c4812edb5bad-metrics-certs\") pod \"maas-api-e2e-ipp-b-fca422-767b78c8cb-fzvl2\" (UID: \"80f1162e-fa26-45b9-8016-c4812edb5bad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fca422-767b78c8cb-fzvl2"
Oct 30 13:38:40.313457 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:40.313436    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvcw7\" (UniqueName: \"kubernetes.io/projected/80f1162e-fa26-45b9-8016-c4812edb5bad-kube-api-access-xvcw7\") pod \"maas-api-e2e-ipp-b-fca422-767b78c8cb-fzvl2\" (UID: \"80f1162e-fa26-45b9-8016-c4812edb5bad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fca422-767b78c8cb-fzvl2"
Oct 30 13:38:40.412621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:40.403962    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fca422-767b78c8cb-fzvl2"
Oct 30 13:38:40.759762 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:38:40.759728    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80f1162e_fa26_45b9_8016_c4812edb5bad.slice/crio-5a29604b69f7537da76437079662a747381569e3f6134a95744c79f8afe7bee4 WatchSource:0}: Error finding container 5a29604b69f7537da76437079662a747381569e3f6134a95744c79f8afe7bee4: Status 404 returned error can't find the container with id 5a29604b69f7537da76437079662a747381569e3f6134a95744c79f8afe7bee4
Oct 30 13:38:40.762109 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:40.762068    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-fca422-767b78c8cb-fzvl2"]
Oct 30 13:38:41.162121 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:41.162085    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fca422-767b78c8cb-fzvl2" event={"ID":"80f1162e-fa26-45b9-8016-c4812edb5bad","Type":"ContainerStarted","Data":"5a29604b69f7537da76437079662a747381569e3f6134a95744c79f8afe7bee4"}
Oct 30 13:38:41.604404 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:38:41.604317    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:38:41.978821 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:41.978795    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30c004-84d79b8679-ndx6w"
Oct 30 13:38:41.978999 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:41.978860    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-30c004-d665cf8b-f56dn"
Oct 30 13:38:42.092922 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:42.092883    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq"]
Oct 30 13:38:42.093161 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:42.093134    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq" podUID="64219df1-aafc-47e9-8452-b26ec5db3150" containerName="istio-proxy" containerID="cri-o://72d1e09260df7e841a4c031f44df1e1ca8037a6606d5d0e26879e1ae07e3c899" gracePeriod=30
Oct 30 13:38:42.125365 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:42.125320    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm"]
Oct 30 13:38:42.125593 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:42.125554    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm" podUID="9abdd340-6b70-4f16-a849-c53d2fe03672" containerName="istio-proxy" containerID="cri-o://dffa0b0b6a13bfc17aae55501ef04fe767dcb62f318ca0a5def37a0a131ce7eb" gracePeriod=30
Oct 30 13:38:42.173303 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:42.173271    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fca422-767b78c8cb-fzvl2" event={"ID":"80f1162e-fa26-45b9-8016-c4812edb5bad","Type":"ContainerStarted","Data":"3953071e1c98b8fa58c44729b00b66adaa01803c80ab1c2778cc33c873e9ec5c"}
Oct 30 13:38:42.173808 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:42.173357    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fca422-767b78c8cb-fzvl2"
Oct 30 13:38:42.305673 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:42.305061    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fca422-767b78c8cb-fzvl2" podStartSLOduration=1.929045036 podStartE2EDuration="2.305038937s" totalImagesPullingTime="375.993901ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:38:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:38:42.250114502 +0000 UTC m=+2821.416479153" watchObservedRunningTime="2026-10-30 13:38:42.305038937 +0000 UTC m=+2821.471403587"
Oct 30 13:38:46.168098 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:46.168069    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c54f0f-5d446cb49d-dhfks"
Oct 30 13:38:47.219777 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:47.219745    4364 generic.go:356] "Generic (PLEG): container finished" podID="64219df1-aafc-47e9-8452-b26ec5db3150" containerID="72d1e09260df7e841a4c031f44df1e1ca8037a6606d5d0e26879e1ae07e3c899" exitCode=0
Oct 30 13:38:47.220152 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:47.219789    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq" event={"ID":"64219df1-aafc-47e9-8452-b26ec5db3150","Type":"ContainerDied","Data":"72d1e09260df7e841a4c031f44df1e1ca8037a6606d5d0e26879e1ae07e3c899"}
Oct 30 13:38:47.428203 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:47.428177    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq"
Oct 30 13:38:47.475948 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:47.475926    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm"
Oct 30 13:38:47.480499 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:47.480478    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9abdd340-6b70-4f16-a849-c53d2fe03672-kube-api-access-zvk4c\" (UniqueName: \"kubernetes.io/projected/9abdd340-6b70-4f16-a849-c53d2fe03672-kube-api-access-zvk4c\") pod \"9abdd340-6b70-4f16-a849-c53d2fe03672\" (UID: \"9abdd340-6b70-4f16-a849-c53d2fe03672\") "
Oct 30 13:38:47.480576 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:47.480523    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9abdd340-6b70-4f16-a849-c53d2fe03672-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9abdd340-6b70-4f16-a849-c53d2fe03672-istiod-ca-cert\") pod \"9abdd340-6b70-4f16-a849-c53d2fe03672\" (UID: \"9abdd340-6b70-4f16-a849-c53d2fe03672\") "
Oct 30 13:38:47.480576 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:47.480557    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9abdd340-6b70-4f16-a849-c53d2fe03672-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9abdd340-6b70-4f16-a849-c53d2fe03672-workload-socket\") pod \"9abdd340-6b70-4f16-a849-c53d2fe03672\" (UID: \"9abdd340-6b70-4f16-a849-c53d2fe03672\") "
Oct 30 13:38:47.480675 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:47.480592    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9abdd340-6b70-4f16-a849-c53d2fe03672-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9abdd340-6b70-4f16-a849-c53d2fe03672-istio-envoy\") pod \"9abdd340-6b70-4f16-a849-c53d2fe03672\" (UID: \"9abdd340-6b70-4f16-a849-c53d2fe03672\") "
Oct 30 13:38:47.480675 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:47.480633    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64219df1-aafc-47e9-8452-b26ec5db3150-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64219df1-aafc-47e9-8452-b26ec5db3150-workload-socket\") pod \"64219df1-aafc-47e9-8452-b26ec5db3150\" (UID: \"64219df1-aafc-47e9-8452-b26ec5db3150\") "
Oct 30 13:38:47.480761 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:47.480675    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64219df1-aafc-47e9-8452-b26ec5db3150-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64219df1-aafc-47e9-8452-b26ec5db3150-istio-data\") pod \"64219df1-aafc-47e9-8452-b26ec5db3150\" (UID: \"64219df1-aafc-47e9-8452-b26ec5db3150\") "
Oct 30 13:38:47.480761 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:47.480720    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64219df1-aafc-47e9-8452-b26ec5db3150-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64219df1-aafc-47e9-8452-b26ec5db3150-istiod-ca-cert\") pod \"64219df1-aafc-47e9-8452-b26ec5db3150\" (UID: \"64219df1-aafc-47e9-8452-b26ec5db3150\") "
Oct 30 13:38:47.480863 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:47.480785    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64219df1-aafc-47e9-8452-b26ec5db3150-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64219df1-aafc-47e9-8452-b26ec5db3150-credential-socket\") pod \"64219df1-aafc-47e9-8452-b26ec5db3150\" (UID: \"64219df1-aafc-47e9-8452-b26ec5db3150\") "
Oct 30 13:38:47.480863 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:47.480810    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64219df1-aafc-47e9-8452-b26ec5db3150-kube-api-access-hq882\" (UniqueName: \"kubernetes.io/projected/64219df1-aafc-47e9-8452-b26ec5db3150-kube-api-access-hq882\") pod \"64219df1-aafc-47e9-8452-b26ec5db3150\" (UID: \"64219df1-aafc-47e9-8452-b26ec5db3150\") "
Oct 30 13:38:47.480863 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:47.480847    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9abdd340-6b70-4f16-a849-c53d2fe03672-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9abdd340-6b70-4f16-a849-c53d2fe03672-workload-certs\") pod \"9abdd340-6b70-4f16-a849-c53d2fe03672\" (UID: \"9abdd340-6b70-4f16-a849-c53d2fe03672\") "
Oct 30 13:38:47.481023 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:47.480892    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64219df1-aafc-47e9-8452-b26ec5db3150-istio-token\" (UniqueName: \"kubernetes.io/projected/64219df1-aafc-47e9-8452-b26ec5db3150-istio-token\") pod \"64219df1-aafc-47e9-8452-b26ec5db3150\" (UID: \"64219df1-aafc-47e9-8452-b26ec5db3150\") "
Oct 30 13:38:47.481023 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:47.480904    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64219df1-aafc-47e9-8452-b26ec5db3150-workload-socket" pod "64219df1-aafc-47e9-8452-b26ec5db3150" (UID: "64219df1-aafc-47e9-8452-b26ec5db3150"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:38:47.481023 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:47.480909    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9abdd340-6b70-4f16-a849-c53d2fe03672-workload-socket" pod "9abdd340-6b70-4f16-a849-c53d2fe03672" (UID: "9abdd340-6b70-4f16-a849-c53d2fe03672"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:38:47.481023 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:47.480955    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9abdd340-6b70-4f16-a849-c53d2fe03672-istio-token\" (UniqueName: \"kubernetes.io/projected/9abdd340-6b70-4f16-a849-c53d2fe03672-istio-token\") pod \"9abdd340-6b70-4f16-a849-c53d2fe03672\" (UID: \"9abdd340-6b70-4f16-a849-c53d2fe03672\") "
Oct 30 13:38:47.481023 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:47.480999    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9abdd340-6b70-4f16-a849-c53d2fe03672-istiod-ca-cert" pod "9abdd340-6b70-4f16-a849-c53d2fe03672" (UID: "9abdd340-6b70-4f16-a849-c53d2fe03672"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:38:47.481023 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:47.481004    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9abdd340-6b70-4f16-a849-c53d2fe03672-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9abdd340-6b70-4f16-a849-c53d2fe03672-istio-data\") pod \"9abdd340-6b70-4f16-a849-c53d2fe03672\" (UID: \"9abdd340-6b70-4f16-a849-c53d2fe03672\") "
Oct 30 13:38:47.481416 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:47.481031    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64219df1-aafc-47e9-8452-b26ec5db3150-istio-data" pod "64219df1-aafc-47e9-8452-b26ec5db3150" (UID: "64219df1-aafc-47e9-8452-b26ec5db3150"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:38:47.481416 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:47.481122    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64219df1-aafc-47e9-8452-b26ec5db3150-credential-socket" pod "64219df1-aafc-47e9-8452-b26ec5db3150" (UID: "64219df1-aafc-47e9-8452-b26ec5db3150"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:38:47.481416 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:47.481177    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64219df1-aafc-47e9-8452-b26ec5db3150-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64219df1-aafc-47e9-8452-b26ec5db3150-workload-certs\") pod \"64219df1-aafc-47e9-8452-b26ec5db3150\" (UID: \"64219df1-aafc-47e9-8452-b26ec5db3150\") "
Oct 30 13:38:47.481416 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:47.481214    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9abdd340-6b70-4f16-a849-c53d2fe03672-istio-data" pod "9abdd340-6b70-4f16-a849-c53d2fe03672" (UID: "9abdd340-6b70-4f16-a849-c53d2fe03672"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:38:47.481416 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:47.481255    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64219df1-aafc-47e9-8452-b26ec5db3150-istiod-ca-cert" pod "64219df1-aafc-47e9-8452-b26ec5db3150" (UID: "64219df1-aafc-47e9-8452-b26ec5db3150"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:38:47.481416 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:47.481259    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9abdd340-6b70-4f16-a849-c53d2fe03672-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9abdd340-6b70-4f16-a849-c53d2fe03672-istio-podinfo\") pod \"9abdd340-6b70-4f16-a849-c53d2fe03672\" (UID: \"9abdd340-6b70-4f16-a849-c53d2fe03672\") "
Oct 30 13:38:47.481781 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:47.481460    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9abdd340-6b70-4f16-a849-c53d2fe03672-workload-certs" pod "9abdd340-6b70-4f16-a849-c53d2fe03672" (UID: "9abdd340-6b70-4f16-a849-c53d2fe03672"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:38:47.481781 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:47.481492    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9abdd340-6b70-4f16-a849-c53d2fe03672-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9abdd340-6b70-4f16-a849-c53d2fe03672-credential-socket\") pod \"9abdd340-6b70-4f16-a849-c53d2fe03672\" (UID: \"9abdd340-6b70-4f16-a849-c53d2fe03672\") "
Oct 30 13:38:47.481781 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:47.481623    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64219df1-aafc-47e9-8452-b26ec5db3150-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64219df1-aafc-47e9-8452-b26ec5db3150-istio-envoy\") pod \"64219df1-aafc-47e9-8452-b26ec5db3150\" (UID: \"64219df1-aafc-47e9-8452-b26ec5db3150\") "
Oct 30 13:38:47.481781 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:47.481659    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/64219df1-aafc-47e9-8452-b26ec5db3150-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64219df1-aafc-47e9-8452-b26ec5db3150-istio-podinfo\") pod \"64219df1-aafc-47e9-8452-b26ec5db3150\" (UID: \"64219df1-aafc-47e9-8452-b26ec5db3150\") "
Oct 30 13:38:47.481781 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:47.481709    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64219df1-aafc-47e9-8452-b26ec5db3150-workload-certs" pod "64219df1-aafc-47e9-8452-b26ec5db3150" (UID: "64219df1-aafc-47e9-8452-b26ec5db3150"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:38:47.482083 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:47.481900    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9abdd340-6b70-4f16-a849-c53d2fe03672-credential-socket" pod "9abdd340-6b70-4f16-a849-c53d2fe03672" (UID: "9abdd340-6b70-4f16-a849-c53d2fe03672"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:38:47.482580 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:47.482549    4364 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64219df1-aafc-47e9-8452-b26ec5db3150-workload-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:38:47.482580 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:47.482577    4364 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64219df1-aafc-47e9-8452-b26ec5db3150-istio-data\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:38:47.482766 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:47.482590    4364 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64219df1-aafc-47e9-8452-b26ec5db3150-istiod-ca-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:38:47.482766 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:47.482620    4364 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64219df1-aafc-47e9-8452-b26ec5db3150-credential-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:38:47.482766 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:47.482633    4364 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9abdd340-6b70-4f16-a849-c53d2fe03672-workload-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:38:47.482766 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:47.482647    4364 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9abdd340-6b70-4f16-a849-c53d2fe03672-istio-data\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:38:47.482766 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:47.482658    4364 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64219df1-aafc-47e9-8452-b26ec5db3150-workload-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:38:47.482766 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:47.482670    4364 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9abdd340-6b70-4f16-a849-c53d2fe03672-credential-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:38:47.482766 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:47.482683    4364 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9abdd340-6b70-4f16-a849-c53d2fe03672-istiod-ca-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:38:47.482766 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:47.482696    4364 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9abdd340-6b70-4f16-a849-c53d2fe03672-workload-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:38:47.483332 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:47.483305    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9abdd340-6b70-4f16-a849-c53d2fe03672-istio-envoy" pod "9abdd340-6b70-4f16-a849-c53d2fe03672" (UID: "9abdd340-6b70-4f16-a849-c53d2fe03672"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:38:47.483532 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:47.483509    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64219df1-aafc-47e9-8452-b26ec5db3150-istio-token" pod "64219df1-aafc-47e9-8452-b26ec5db3150" (UID: "64219df1-aafc-47e9-8452-b26ec5db3150"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:38:47.483683 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:47.483665    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64219df1-aafc-47e9-8452-b26ec5db3150-kube-api-access-hq882" pod "64219df1-aafc-47e9-8452-b26ec5db3150" (UID: "64219df1-aafc-47e9-8452-b26ec5db3150"). InnerVolumeSpecName "kube-api-access-hq882". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:38:47.483871 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:47.483851    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9abdd340-6b70-4f16-a849-c53d2fe03672-istio-podinfo" pod "9abdd340-6b70-4f16-a849-c53d2fe03672" (UID: "9abdd340-6b70-4f16-a849-c53d2fe03672"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:38:47.483985 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:47.483967    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64219df1-aafc-47e9-8452-b26ec5db3150-istio-envoy" pod "64219df1-aafc-47e9-8452-b26ec5db3150" (UID: "64219df1-aafc-47e9-8452-b26ec5db3150"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:38:47.484204 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:47.484183    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9abdd340-6b70-4f16-a849-c53d2fe03672-istio-token" pod "9abdd340-6b70-4f16-a849-c53d2fe03672" (UID: "9abdd340-6b70-4f16-a849-c53d2fe03672"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:38:47.484650 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:47.484632    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/64219df1-aafc-47e9-8452-b26ec5db3150-istio-podinfo" pod "64219df1-aafc-47e9-8452-b26ec5db3150" (UID: "64219df1-aafc-47e9-8452-b26ec5db3150"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:38:47.484849 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:47.484831    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9abdd340-6b70-4f16-a849-c53d2fe03672-kube-api-access-zvk4c" pod "9abdd340-6b70-4f16-a849-c53d2fe03672" (UID: "9abdd340-6b70-4f16-a849-c53d2fe03672"). InnerVolumeSpecName "kube-api-access-zvk4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:38:47.584382 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:47.584344    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvk4c\" (UniqueName: \"kubernetes.io/projected/9abdd340-6b70-4f16-a849-c53d2fe03672-kube-api-access-zvk4c\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:38:47.584382 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:47.584384    4364 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9abdd340-6b70-4f16-a849-c53d2fe03672-istio-envoy\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:38:47.584627 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:47.584400    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hq882\" (UniqueName: \"kubernetes.io/projected/64219df1-aafc-47e9-8452-b26ec5db3150-kube-api-access-hq882\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:38:47.584627 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:47.584413    4364 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64219df1-aafc-47e9-8452-b26ec5db3150-istio-token\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:38:47.584627 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:47.584426    4364 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9abdd340-6b70-4f16-a849-c53d2fe03672-istio-token\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:38:47.584627 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:47.584438    4364 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9abdd340-6b70-4f16-a849-c53d2fe03672-istio-podinfo\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:38:47.584627 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:47.584451    4364 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64219df1-aafc-47e9-8452-b26ec5db3150-istio-envoy\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:38:47.584627 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:47.584465    4364 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64219df1-aafc-47e9-8452-b26ec5db3150-istio-podinfo\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:38:48.187303 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:48.187275    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fca422-767b78c8cb-fzvl2"
Oct 30 13:38:48.229910 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:48.229879    4364 generic.go:356] "Generic (PLEG): container finished" podID="9abdd340-6b70-4f16-a849-c53d2fe03672" containerID="dffa0b0b6a13bfc17aae55501ef04fe767dcb62f318ca0a5def37a0a131ce7eb" exitCode=0
Oct 30 13:38:48.230415 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:48.230087    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm" event={"ID":"9abdd340-6b70-4f16-a849-c53d2fe03672","Type":"ContainerDied","Data":"dffa0b0b6a13bfc17aae55501ef04fe767dcb62f318ca0a5def37a0a131ce7eb"}
Oct 30 13:38:48.230415 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:48.230139    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm" event={"ID":"9abdd340-6b70-4f16-a849-c53d2fe03672","Type":"ContainerDied","Data":"b3c07a6b9604b3ab6407ddce4a412b7d5f07512f502147ea89aca3a00f1f416b"}
Oct 30 13:38:48.230415 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:48.230168    4364 scope.go:122] "RemoveContainer" containerID="dffa0b0b6a13bfc17aae55501ef04fe767dcb62f318ca0a5def37a0a131ce7eb"
Oct 30 13:38:48.230415 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:48.230387    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm"
Oct 30 13:38:48.233709 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:48.233688    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq"
Oct 30 13:38:48.233814 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:48.233688    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq" event={"ID":"64219df1-aafc-47e9-8452-b26ec5db3150","Type":"ContainerDied","Data":"ecf88cbd334e59fa62b4522bdf056df0df60cff2e3c97ba412c8e60ca84d3f71"}
Oct 30 13:38:48.245142 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:48.244957    4364 scope.go:122] "RemoveContainer" containerID="dffa0b0b6a13bfc17aae55501ef04fe767dcb62f318ca0a5def37a0a131ce7eb"
Oct 30 13:38:48.245226 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:38:48.245204    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dffa0b0b6a13bfc17aae55501ef04fe767dcb62f318ca0a5def37a0a131ce7eb\": container with ID starting with dffa0b0b6a13bfc17aae55501ef04fe767dcb62f318ca0a5def37a0a131ce7eb not found: ID does not exist" containerID="dffa0b0b6a13bfc17aae55501ef04fe767dcb62f318ca0a5def37a0a131ce7eb"
Oct 30 13:38:48.245269 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:48.245233    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dffa0b0b6a13bfc17aae55501ef04fe767dcb62f318ca0a5def37a0a131ce7eb"} err="failed to get container status \"dffa0b0b6a13bfc17aae55501ef04fe767dcb62f318ca0a5def37a0a131ce7eb\": rpc error: code = NotFound desc = could not find container \"dffa0b0b6a13bfc17aae55501ef04fe767dcb62f318ca0a5def37a0a131ce7eb\": container with ID starting with dffa0b0b6a13bfc17aae55501ef04fe767dcb62f318ca0a5def37a0a131ce7eb not found: ID does not exist"
Oct 30 13:38:48.245269 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:48.245251    4364 scope.go:122] "RemoveContainer" containerID="72d1e09260df7e841a4c031f44df1e1ca8037a6606d5d0e26879e1ae07e3c899"
Oct 30 13:38:48.301431 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:48.301389    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm"]
Oct 30 13:38:48.311487 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:48.311453    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-gsjvm"]
Oct 30 13:38:48.342938 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:48.342912    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq"]
Oct 30 13:38:48.347455 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:48.347433    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-hmfcq"]
Oct 30 13:38:48.973250 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:48.973217    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64219df1-aafc-47e9-8452-b26ec5db3150" path="/var/lib/kubelet/pods/64219df1-aafc-47e9-8452-b26ec5db3150/volumes"
Oct 30 13:38:48.974537 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:48.974507    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9abdd340-6b70-4f16-a849-c53d2fe03672" path="/var/lib/kubelet/pods/9abdd340-6b70-4f16-a849-c53d2fe03672/volumes"
Oct 30 13:38:54.395681 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:38:54.392095    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:39:03.147719 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:03.147683    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l"]
Oct 30 13:39:03.148265 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:03.147915    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l" podUID="c6ba17d2-cc66-444e-9976-36496fe9db21" containerName="istio-proxy" containerID="cri-o://a3caecf61da2f0dafb8d3c999ac8b7beec600121f9c98611295dabfe46df175c" gracePeriod=30
Oct 30 13:39:05.504282 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:05.504247    4364 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:39:05.504688 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:05.504306    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l" podUID="c6ba17d2-cc66-444e-9976-36496fe9db21" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:39:08.441678 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:08.441650    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l"
Oct 30 13:39:08.576535 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:08.576499    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6ba17d2-cc66-444e-9976-36496fe9db21-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6ba17d2-cc66-444e-9976-36496fe9db21-credential-socket\") pod \"c6ba17d2-cc66-444e-9976-36496fe9db21\" (UID: \"c6ba17d2-cc66-444e-9976-36496fe9db21\") "
Oct 30 13:39:08.576853 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:08.576815    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6ba17d2-cc66-444e-9976-36496fe9db21-credential-socket" pod "c6ba17d2-cc66-444e-9976-36496fe9db21" (UID: "c6ba17d2-cc66-444e-9976-36496fe9db21"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:08.576979 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:08.576909    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6ba17d2-cc66-444e-9976-36496fe9db21-kube-api-access-gl252\" (UniqueName: \"kubernetes.io/projected/c6ba17d2-cc66-444e-9976-36496fe9db21-kube-api-access-gl252\") pod \"c6ba17d2-cc66-444e-9976-36496fe9db21\" (UID: \"c6ba17d2-cc66-444e-9976-36496fe9db21\") "
Oct 30 13:39:08.576979 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:08.576952    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6ba17d2-cc66-444e-9976-36496fe9db21-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6ba17d2-cc66-444e-9976-36496fe9db21-istiod-ca-cert\") pod \"c6ba17d2-cc66-444e-9976-36496fe9db21\" (UID: \"c6ba17d2-cc66-444e-9976-36496fe9db21\") "
Oct 30 13:39:08.577111 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:08.577019    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6ba17d2-cc66-444e-9976-36496fe9db21-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6ba17d2-cc66-444e-9976-36496fe9db21-istio-envoy\") pod \"c6ba17d2-cc66-444e-9976-36496fe9db21\" (UID: \"c6ba17d2-cc66-444e-9976-36496fe9db21\") "
Oct 30 13:39:08.577176 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:08.577130    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6ba17d2-cc66-444e-9976-36496fe9db21-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6ba17d2-cc66-444e-9976-36496fe9db21-workload-certs\") pod \"c6ba17d2-cc66-444e-9976-36496fe9db21\" (UID: \"c6ba17d2-cc66-444e-9976-36496fe9db21\") "
Oct 30 13:39:08.577242 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:08.577180    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c6ba17d2-cc66-444e-9976-36496fe9db21-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6ba17d2-cc66-444e-9976-36496fe9db21-istio-podinfo\") pod \"c6ba17d2-cc66-444e-9976-36496fe9db21\" (UID: \"c6ba17d2-cc66-444e-9976-36496fe9db21\") "
Oct 30 13:39:08.577297 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:08.577238    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6ba17d2-cc66-444e-9976-36496fe9db21-istio-token\" (UniqueName: \"kubernetes.io/projected/c6ba17d2-cc66-444e-9976-36496fe9db21-istio-token\") pod \"c6ba17d2-cc66-444e-9976-36496fe9db21\" (UID: \"c6ba17d2-cc66-444e-9976-36496fe9db21\") "
Oct 30 13:39:08.577400 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:08.577382    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6ba17d2-cc66-444e-9976-36496fe9db21-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6ba17d2-cc66-444e-9976-36496fe9db21-istio-data\") pod \"c6ba17d2-cc66-444e-9976-36496fe9db21\" (UID: \"c6ba17d2-cc66-444e-9976-36496fe9db21\") "
Oct 30 13:39:08.577466 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:08.577454    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6ba17d2-cc66-444e-9976-36496fe9db21-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6ba17d2-cc66-444e-9976-36496fe9db21-workload-socket\") pod \"c6ba17d2-cc66-444e-9976-36496fe9db21\" (UID: \"c6ba17d2-cc66-444e-9976-36496fe9db21\") "
Oct 30 13:39:08.578241 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:08.578212    4364 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6ba17d2-cc66-444e-9976-36496fe9db21-credential-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:39:08.578537 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:08.578516    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6ba17d2-cc66-444e-9976-36496fe9db21-workload-socket" pod "c6ba17d2-cc66-444e-9976-36496fe9db21" (UID: "c6ba17d2-cc66-444e-9976-36496fe9db21"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:08.578717 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:08.578693    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6ba17d2-cc66-444e-9976-36496fe9db21-workload-certs" pod "c6ba17d2-cc66-444e-9976-36496fe9db21" (UID: "c6ba17d2-cc66-444e-9976-36496fe9db21"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:08.579804 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:08.579762    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6ba17d2-cc66-444e-9976-36496fe9db21-istiod-ca-cert" pod "c6ba17d2-cc66-444e-9976-36496fe9db21" (UID: "c6ba17d2-cc66-444e-9976-36496fe9db21"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:39:08.580434 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:08.580408    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6ba17d2-cc66-444e-9976-36496fe9db21-istio-data" pod "c6ba17d2-cc66-444e-9976-36496fe9db21" (UID: "c6ba17d2-cc66-444e-9976-36496fe9db21"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:08.582333 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:08.581282    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c6ba17d2-cc66-444e-9976-36496fe9db21-istio-podinfo" pod "c6ba17d2-cc66-444e-9976-36496fe9db21" (UID: "c6ba17d2-cc66-444e-9976-36496fe9db21"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:39:08.582333 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:08.582807    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6ba17d2-cc66-444e-9976-36496fe9db21-kube-api-access-gl252" pod "c6ba17d2-cc66-444e-9976-36496fe9db21" (UID: "c6ba17d2-cc66-444e-9976-36496fe9db21"). InnerVolumeSpecName "kube-api-access-gl252". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:08.582333 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:08.584110    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6ba17d2-cc66-444e-9976-36496fe9db21-istio-envoy" pod "c6ba17d2-cc66-444e-9976-36496fe9db21" (UID: "c6ba17d2-cc66-444e-9976-36496fe9db21"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:08.588869 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:08.588839    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6ba17d2-cc66-444e-9976-36496fe9db21-istio-token" pod "c6ba17d2-cc66-444e-9976-36496fe9db21" (UID: "c6ba17d2-cc66-444e-9976-36496fe9db21"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:08.679347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:08.679271    4364 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6ba17d2-cc66-444e-9976-36496fe9db21-istio-token\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:39:08.679347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:08.679301    4364 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6ba17d2-cc66-444e-9976-36496fe9db21-istio-data\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:39:08.679347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:08.679310    4364 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6ba17d2-cc66-444e-9976-36496fe9db21-workload-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:39:08.679347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:08.679319    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gl252\" (UniqueName: \"kubernetes.io/projected/c6ba17d2-cc66-444e-9976-36496fe9db21-kube-api-access-gl252\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:39:08.679347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:08.679329    4364 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6ba17d2-cc66-444e-9976-36496fe9db21-istiod-ca-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:39:08.679347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:08.679337    4364 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6ba17d2-cc66-444e-9976-36496fe9db21-istio-envoy\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:39:08.679347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:08.679344    4364 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6ba17d2-cc66-444e-9976-36496fe9db21-workload-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:39:08.679347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:08.679352    4364 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6ba17d2-cc66-444e-9976-36496fe9db21-istio-podinfo\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:39:08.802700 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:08.802661    4364 generic.go:356] "Generic (PLEG): container finished" podID="c6ba17d2-cc66-444e-9976-36496fe9db21" containerID="a3caecf61da2f0dafb8d3c999ac8b7beec600121f9c98611295dabfe46df175c" exitCode=0
Oct 30 13:39:08.802848 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:08.802726    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l"
Oct 30 13:39:08.802848 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:08.802736    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l" event={"ID":"c6ba17d2-cc66-444e-9976-36496fe9db21","Type":"ContainerDied","Data":"a3caecf61da2f0dafb8d3c999ac8b7beec600121f9c98611295dabfe46df175c"}
Oct 30 13:39:08.802848 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:08.802818    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l" event={"ID":"c6ba17d2-cc66-444e-9976-36496fe9db21","Type":"ContainerDied","Data":"92bd143314802ad478f01ebd07ff80df31f8a63fc704d248d3d50b6b7f84cba5"}
Oct 30 13:39:08.802848 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:08.802848    4364 scope.go:122] "RemoveContainer" containerID="a3caecf61da2f0dafb8d3c999ac8b7beec600121f9c98611295dabfe46df175c"
Oct 30 13:39:08.818610 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:08.818578    4364 scope.go:122] "RemoveContainer" containerID="a3caecf61da2f0dafb8d3c999ac8b7beec600121f9c98611295dabfe46df175c"
Oct 30 13:39:08.819105 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:39:08.819071    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3caecf61da2f0dafb8d3c999ac8b7beec600121f9c98611295dabfe46df175c\": container with ID starting with a3caecf61da2f0dafb8d3c999ac8b7beec600121f9c98611295dabfe46df175c not found: ID does not exist" containerID="a3caecf61da2f0dafb8d3c999ac8b7beec600121f9c98611295dabfe46df175c"
Oct 30 13:39:08.819220 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:08.819112    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3caecf61da2f0dafb8d3c999ac8b7beec600121f9c98611295dabfe46df175c"} err="failed to get container status \"a3caecf61da2f0dafb8d3c999ac8b7beec600121f9c98611295dabfe46df175c\": rpc error: code = NotFound desc = could not find container \"a3caecf61da2f0dafb8d3c999ac8b7beec600121f9c98611295dabfe46df175c\": container with ID starting with a3caecf61da2f0dafb8d3c999ac8b7beec600121f9c98611295dabfe46df175c not found: ID does not exist"
Oct 30 13:39:08.858919 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:08.858896    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l"]
Oct 30 13:39:08.870628 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:08.870589    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-xwp4l"]
Oct 30 13:39:08.972051 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:08.971961    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6ba17d2-cc66-444e-9976-36496fe9db21" path="/var/lib/kubelet/pods/c6ba17d2-cc66-444e-9976-36496fe9db21/volumes"
Oct 30 13:39:11.506521 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:39:11.506485    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:39:17.428956 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:17.428922    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-8b77b0-75f95c8744-psdzn"]
Oct 30 13:39:17.461776 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:17.461743    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8b77b0-d46dbfdc-lkc4n"]
Oct 30 13:39:22.429416 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:22.429369    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8b77b0-75f95c8744-psdzn" podUID="91859899-765d-48fd-9c18-a37942683a5a" containerName="payload-processing" containerID="cri-o://a79a3febd438e573bd96ce4218a46b2a2cb945b852029ef8c52e05cc7fa27929" gracePeriod=25
Oct 30 13:39:22.462037 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:22.461998    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8b77b0-d46dbfdc-lkc4n" podUID="98935320-7c94-4bd4-a88a-44c9ea91fcc6" containerName="payload-pre-processing" containerID="cri-o://a8ec27f0d889c3e3fd326912856d634aa92742ae01012a0ca4b984562ba52036" gracePeriod=25
Oct 30 13:39:22.757229 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:22.757197    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8b77b0-d46dbfdc-lkc4n"
Oct 30 13:39:22.801274 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:22.801246    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98935320-7c94-4bd4-a88a-44c9ea91fcc6-kube-api-access-sx74p\" (UniqueName: \"kubernetes.io/projected/98935320-7c94-4bd4-a88a-44c9ea91fcc6-kube-api-access-sx74p\") pod \"98935320-7c94-4bd4-a88a-44c9ea91fcc6\" (UID: \"98935320-7c94-4bd4-a88a-44c9ea91fcc6\") "
Oct 30 13:39:22.801670 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:22.801641    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98935320-7c94-4bd4-a88a-44c9ea91fcc6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98935320-7c94-4bd4-a88a-44c9ea91fcc6-plugins-config-volume\") pod \"98935320-7c94-4bd4-a88a-44c9ea91fcc6\" (UID: \"98935320-7c94-4bd4-a88a-44c9ea91fcc6\") "
Oct 30 13:39:22.803010 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:22.802183    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98935320-7c94-4bd4-a88a-44c9ea91fcc6-plugins-config-volume" pod "98935320-7c94-4bd4-a88a-44c9ea91fcc6" (UID: "98935320-7c94-4bd4-a88a-44c9ea91fcc6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:39:22.803010 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:22.802367    4364 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98935320-7c94-4bd4-a88a-44c9ea91fcc6-plugins-config-volume\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:39:22.807133 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:22.807106    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98935320-7c94-4bd4-a88a-44c9ea91fcc6-kube-api-access-sx74p" pod "98935320-7c94-4bd4-a88a-44c9ea91fcc6" (UID: "98935320-7c94-4bd4-a88a-44c9ea91fcc6"). InnerVolumeSpecName "kube-api-access-sx74p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:22.903608 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:22.903565    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sx74p\" (UniqueName: \"kubernetes.io/projected/98935320-7c94-4bd4-a88a-44c9ea91fcc6-kube-api-access-sx74p\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:39:22.978712 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:22.978627    4364 generic.go:356] "Generic (PLEG): container finished" podID="98935320-7c94-4bd4-a88a-44c9ea91fcc6" containerID="a8ec27f0d889c3e3fd326912856d634aa92742ae01012a0ca4b984562ba52036" exitCode=0
Oct 30 13:39:22.978712 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:22.978676    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8b77b0-d46dbfdc-lkc4n"
Oct 30 13:39:22.978928 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:22.978708    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8b77b0-d46dbfdc-lkc4n" event={"ID":"98935320-7c94-4bd4-a88a-44c9ea91fcc6","Type":"ContainerDied","Data":"a8ec27f0d889c3e3fd326912856d634aa92742ae01012a0ca4b984562ba52036"}
Oct 30 13:39:22.978928 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:22.978767    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8b77b0-d46dbfdc-lkc4n" event={"ID":"98935320-7c94-4bd4-a88a-44c9ea91fcc6","Type":"ContainerDied","Data":"494ccbaac3e78873fe215c564d66584527f2ba8fff07f665306e6a15fec2484b"}
Oct 30 13:39:22.978928 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:22.978792    4364 scope.go:122] "RemoveContainer" containerID="a8ec27f0d889c3e3fd326912856d634aa92742ae01012a0ca4b984562ba52036"
Oct 30 13:39:22.991218 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:22.991198    4364 scope.go:122] "RemoveContainer" containerID="a8ec27f0d889c3e3fd326912856d634aa92742ae01012a0ca4b984562ba52036"
Oct 30 13:39:22.991494 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:39:22.991473    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8ec27f0d889c3e3fd326912856d634aa92742ae01012a0ca4b984562ba52036\": container with ID starting with a8ec27f0d889c3e3fd326912856d634aa92742ae01012a0ca4b984562ba52036 not found: ID does not exist" containerID="a8ec27f0d889c3e3fd326912856d634aa92742ae01012a0ca4b984562ba52036"
Oct 30 13:39:22.991547 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:22.991502    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8ec27f0d889c3e3fd326912856d634aa92742ae01012a0ca4b984562ba52036"} err="failed to get container status \"a8ec27f0d889c3e3fd326912856d634aa92742ae01012a0ca4b984562ba52036\": rpc error: code = NotFound desc = could not find container \"a8ec27f0d889c3e3fd326912856d634aa92742ae01012a0ca4b984562ba52036\": container with ID starting with a8ec27f0d889c3e3fd326912856d634aa92742ae01012a0ca4b984562ba52036 not found: ID does not exist"
Oct 30 13:39:23.018043 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:23.018010    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8b77b0-75f95c8744-psdzn" podUID="91859899-765d-48fd-9c18-a37942683a5a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:39:23.032962 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:23.032937    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8b77b0-d46dbfdc-lkc4n"]
Oct 30 13:39:23.037553 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:23.037533    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8b77b0-d46dbfdc-lkc4n"]
Oct 30 13:39:24.819191 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:24.819156    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-30c004-6f9cbc6566-9j5jr"]
Oct 30 13:39:24.820008 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:24.819988    4364 cpu_manager.go:414] "containerMap: removing container" podUID="64219df1-aafc-47e9-8452-b26ec5db3150" containerName="istio-proxy"
Oct 30 13:39:24.820008 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:24.820005    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64219df1-aafc-47e9-8452-b26ec5db3150" containerName="istio-proxy"
Oct 30 13:39:24.820198 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:24.820036    4364 cpu_manager.go:414] "containerMap: removing container" podUID="c6ba17d2-cc66-444e-9976-36496fe9db21" containerName="istio-proxy"
Oct 30 13:39:24.820198 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:24.820042    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6ba17d2-cc66-444e-9976-36496fe9db21" containerName="istio-proxy"
Oct 30 13:39:24.820198 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:24.820057    4364 cpu_manager.go:414] "containerMap: removing container" podUID="98935320-7c94-4bd4-a88a-44c9ea91fcc6" containerName="payload-pre-processing"
Oct 30 13:39:24.820198 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:24.820064    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98935320-7c94-4bd4-a88a-44c9ea91fcc6" containerName="payload-pre-processing"
Oct 30 13:39:24.820198 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:24.820098    4364 cpu_manager.go:414] "containerMap: removing container" podUID="9abdd340-6b70-4f16-a849-c53d2fe03672" containerName="istio-proxy"
Oct 30 13:39:24.820198 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:24.820104    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9abdd340-6b70-4f16-a849-c53d2fe03672" containerName="istio-proxy"
Oct 30 13:39:24.820516 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:24.820217    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6ba17d2-cc66-444e-9976-36496fe9db21" containerName="istio-proxy"
Oct 30 13:39:24.820516 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:24.820227    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="98935320-7c94-4bd4-a88a-44c9ea91fcc6" containerName="payload-pre-processing"
Oct 30 13:39:24.820516 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:24.820240    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="9abdd340-6b70-4f16-a849-c53d2fe03672" containerName="istio-proxy"
Oct 30 13:39:24.820516 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:24.820252    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="64219df1-aafc-47e9-8452-b26ec5db3150" containerName="istio-proxy"
Oct 30 13:39:24.826135 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:24.826113    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30c004-6f9cbc6566-9j5jr"
Oct 30 13:39:24.843205 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:24.843179    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-30c004-6f9cbc6566-9j5jr"]
Oct 30 13:39:24.848874 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:24.848851    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-30c004-serving-cert\""
Oct 30 13:39:24.941139 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:24.941107    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3de36fb7-459f-44d2-ba6a-b2cc20d24516-maas-api-tls\") pod \"maas-api-e2e-ipp-b-30c004-6f9cbc6566-9j5jr\" (UID: \"3de36fb7-459f-44d2-ba6a-b2cc20d24516\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30c004-6f9cbc6566-9j5jr"
Oct 30 13:39:24.941314 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:24.941152    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3de36fb7-459f-44d2-ba6a-b2cc20d24516-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-30c004-6f9cbc6566-9j5jr\" (UID: \"3de36fb7-459f-44d2-ba6a-b2cc20d24516\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30c004-6f9cbc6566-9j5jr"
Oct 30 13:39:24.941362 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:24.941333    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp7cc\" (UniqueName: \"kubernetes.io/projected/3de36fb7-459f-44d2-ba6a-b2cc20d24516-kube-api-access-wp7cc\") pod \"maas-api-e2e-ipp-b-30c004-6f9cbc6566-9j5jr\" (UID: \"3de36fb7-459f-44d2-ba6a-b2cc20d24516\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30c004-6f9cbc6566-9j5jr"
Oct 30 13:39:24.941463 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:24.941449    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3de36fb7-459f-44d2-ba6a-b2cc20d24516-metrics-certs\") pod \"maas-api-e2e-ipp-b-30c004-6f9cbc6566-9j5jr\" (UID: \"3de36fb7-459f-44d2-ba6a-b2cc20d24516\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30c004-6f9cbc6566-9j5jr"
Oct 30 13:39:24.972317 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:24.972286    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98935320-7c94-4bd4-a88a-44c9ea91fcc6" path="/var/lib/kubelet/pods/98935320-7c94-4bd4-a88a-44c9ea91fcc6/volumes"
Oct 30 13:39:25.060619 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:25.058912    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp7cc\" (UniqueName: \"kubernetes.io/projected/3de36fb7-459f-44d2-ba6a-b2cc20d24516-kube-api-access-wp7cc\") pod \"maas-api-e2e-ipp-b-30c004-6f9cbc6566-9j5jr\" (UID: \"3de36fb7-459f-44d2-ba6a-b2cc20d24516\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30c004-6f9cbc6566-9j5jr"
Oct 30 13:39:25.060619 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:25.059120    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3de36fb7-459f-44d2-ba6a-b2cc20d24516-metrics-certs\") pod \"maas-api-e2e-ipp-b-30c004-6f9cbc6566-9j5jr\" (UID: \"3de36fb7-459f-44d2-ba6a-b2cc20d24516\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30c004-6f9cbc6566-9j5jr"
Oct 30 13:39:25.060619 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:25.059327    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3de36fb7-459f-44d2-ba6a-b2cc20d24516-maas-api-tls\") pod \"maas-api-e2e-ipp-b-30c004-6f9cbc6566-9j5jr\" (UID: \"3de36fb7-459f-44d2-ba6a-b2cc20d24516\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30c004-6f9cbc6566-9j5jr"
Oct 30 13:39:25.060619 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:25.059367    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3de36fb7-459f-44d2-ba6a-b2cc20d24516-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-30c004-6f9cbc6566-9j5jr\" (UID: \"3de36fb7-459f-44d2-ba6a-b2cc20d24516\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30c004-6f9cbc6566-9j5jr"
Oct 30 13:39:25.061169 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:39:25.061143    4364 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-30c004-serving-cert: secret "maas-api-e2e-ipp-b-30c004-serving-cert" not found
Oct 30 13:39:25.061733 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:39:25.061711    4364 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3de36fb7-459f-44d2-ba6a-b2cc20d24516-maas-api-tls podName:3de36fb7-459f-44d2-ba6a-b2cc20d24516 nodeName:}" failed. No retries permitted until 2026-10-30 13:39:25.561685509 +0000 UTC m=+2864.728050161 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3de36fb7-459f-44d2-ba6a-b2cc20d24516-maas-api-tls") pod "maas-api-e2e-ipp-b-30c004-6f9cbc6566-9j5jr" (UID: "3de36fb7-459f-44d2-ba6a-b2cc20d24516") : secret "maas-api-e2e-ipp-b-30c004-serving-cert" not found
Oct 30 13:39:25.083709 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:25.083403    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3de36fb7-459f-44d2-ba6a-b2cc20d24516-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-30c004-6f9cbc6566-9j5jr\" (UID: \"3de36fb7-459f-44d2-ba6a-b2cc20d24516\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30c004-6f9cbc6566-9j5jr"
Oct 30 13:39:25.106222 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:25.093255    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3de36fb7-459f-44d2-ba6a-b2cc20d24516-metrics-certs\") pod \"maas-api-e2e-ipp-b-30c004-6f9cbc6566-9j5jr\" (UID: \"3de36fb7-459f-44d2-ba6a-b2cc20d24516\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30c004-6f9cbc6566-9j5jr"
Oct 30 13:39:25.106222 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:25.095223    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp7cc\" (UniqueName: \"kubernetes.io/projected/3de36fb7-459f-44d2-ba6a-b2cc20d24516-kube-api-access-wp7cc\") pod \"maas-api-e2e-ipp-b-30c004-6f9cbc6566-9j5jr\" (UID: \"3de36fb7-459f-44d2-ba6a-b2cc20d24516\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30c004-6f9cbc6566-9j5jr"
Oct 30 13:39:25.567297 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:25.567263    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3de36fb7-459f-44d2-ba6a-b2cc20d24516-maas-api-tls\") pod \"maas-api-e2e-ipp-b-30c004-6f9cbc6566-9j5jr\" (UID: \"3de36fb7-459f-44d2-ba6a-b2cc20d24516\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30c004-6f9cbc6566-9j5jr"
Oct 30 13:39:25.569767 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:25.569736    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3de36fb7-459f-44d2-ba6a-b2cc20d24516-maas-api-tls\") pod \"maas-api-e2e-ipp-b-30c004-6f9cbc6566-9j5jr\" (UID: \"3de36fb7-459f-44d2-ba6a-b2cc20d24516\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30c004-6f9cbc6566-9j5jr"
Oct 30 13:39:25.744366 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:25.744277    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30c004-6f9cbc6566-9j5jr"
Oct 30 13:39:26.125283 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:26.125254    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-30c004-6f9cbc6566-9j5jr"]
Oct 30 13:39:26.126430 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:39:26.126402    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3de36fb7_459f_44d2_ba6a_b2cc20d24516.slice/crio-c7703ef93e5ced2dbfb4f38e8fb48a33b6dd8c8e74f725144f5432f008fe6150 WatchSource:0}: Error finding container c7703ef93e5ced2dbfb4f38e8fb48a33b6dd8c8e74f725144f5432f008fe6150: Status 404 returned error can't find the container with id c7703ef93e5ced2dbfb4f38e8fb48a33b6dd8c8e74f725144f5432f008fe6150
Oct 30 13:39:27.061143 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:27.060318    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30c004-6f9cbc6566-9j5jr" event={"ID":"3de36fb7-459f-44d2-ba6a-b2cc20d24516","Type":"ContainerStarted","Data":"f565a988e9f8597c8d33b8f2f0d2ba8635fbe3d0df87479e05d0ead60b818fcd"}
Oct 30 13:39:27.061143 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:27.060434    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30c004-6f9cbc6566-9j5jr" event={"ID":"3de36fb7-459f-44d2-ba6a-b2cc20d24516","Type":"ContainerStarted","Data":"c7703ef93e5ced2dbfb4f38e8fb48a33b6dd8c8e74f725144f5432f008fe6150"}
Oct 30 13:39:27.062669 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:27.062571    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30c004-6f9cbc6566-9j5jr"
Oct 30 13:39:27.111271 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:27.111207    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30c004-6f9cbc6566-9j5jr" podStartSLOduration=2.519302058 podStartE2EDuration="3.111192095s" totalImagesPullingTime="591.890037ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:39:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:39:27.105509771 +0000 UTC m=+2866.271874418" watchObservedRunningTime="2026-10-30 13:39:27.111192095 +0000 UTC m=+2866.277556743"
Oct 30 13:39:32.599280 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:32.599239    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8b77b0-5f7b7ff96f-wggzj"]
Oct 30 13:39:32.599810 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:32.599501    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8b77b0-5f7b7ff96f-wggzj" podUID="eb144ffb-880b-44cb-ac19-bbd878eb3bff" containerName="maas-api" containerID="cri-o://b31895906ec859a4e4800a143ad66c2e83e3afb043fec3f536201a930184b7a3" gracePeriod=30
Oct 30 13:39:33.041622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:33.041482    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8b77b0-75f95c8744-psdzn" podUID="91859899-765d-48fd-9c18-a37942683a5a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:39:33.041825 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:33.041787    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8b77b0-5f7b7ff96f-wggzj"
Oct 30 13:39:33.097375 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:33.097334    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb144ffb-880b-44cb-ac19-bbd878eb3bff-kube-api-access-zs5gj\" (UniqueName: \"kubernetes.io/projected/eb144ffb-880b-44cb-ac19-bbd878eb3bff-kube-api-access-zs5gj\") pod \"eb144ffb-880b-44cb-ac19-bbd878eb3bff\" (UID: \"eb144ffb-880b-44cb-ac19-bbd878eb3bff\") "
Oct 30 13:39:33.097573 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:33.097430    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb144ffb-880b-44cb-ac19-bbd878eb3bff-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb144ffb-880b-44cb-ac19-bbd878eb3bff-openshift-service-ca\") pod \"eb144ffb-880b-44cb-ac19-bbd878eb3bff\" (UID: \"eb144ffb-880b-44cb-ac19-bbd878eb3bff\") "
Oct 30 13:39:33.097573 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:33.097523    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb144ffb-880b-44cb-ac19-bbd878eb3bff-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb144ffb-880b-44cb-ac19-bbd878eb3bff-maas-api-tls\") pod \"eb144ffb-880b-44cb-ac19-bbd878eb3bff\" (UID: \"eb144ffb-880b-44cb-ac19-bbd878eb3bff\") "
Oct 30 13:39:33.097707 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:33.097641    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb144ffb-880b-44cb-ac19-bbd878eb3bff-metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb144ffb-880b-44cb-ac19-bbd878eb3bff-metrics-certs\") pod \"eb144ffb-880b-44cb-ac19-bbd878eb3bff\" (UID: \"eb144ffb-880b-44cb-ac19-bbd878eb3bff\") "
Oct 30 13:39:33.098726 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:33.098643    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb144ffb-880b-44cb-ac19-bbd878eb3bff-openshift-service-ca" pod "eb144ffb-880b-44cb-ac19-bbd878eb3bff" (UID: "eb144ffb-880b-44cb-ac19-bbd878eb3bff"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:39:33.113827 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:33.113729    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb144ffb-880b-44cb-ac19-bbd878eb3bff-kube-api-access-zs5gj" pod "eb144ffb-880b-44cb-ac19-bbd878eb3bff" (UID: "eb144ffb-880b-44cb-ac19-bbd878eb3bff"). InnerVolumeSpecName "kube-api-access-zs5gj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:33.114114 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:33.114079    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb144ffb-880b-44cb-ac19-bbd878eb3bff-maas-api-tls" pod "eb144ffb-880b-44cb-ac19-bbd878eb3bff" (UID: "eb144ffb-880b-44cb-ac19-bbd878eb3bff"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:39:33.140889 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:33.140825    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb144ffb-880b-44cb-ac19-bbd878eb3bff-metrics-certs" pod "eb144ffb-880b-44cb-ac19-bbd878eb3bff" (UID: "eb144ffb-880b-44cb-ac19-bbd878eb3bff"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:39:33.172811 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:33.172780    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30c004-6f9cbc6566-9j5jr"
Oct 30 13:39:33.206088 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:33.206051    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zs5gj\" (UniqueName: \"kubernetes.io/projected/eb144ffb-880b-44cb-ac19-bbd878eb3bff-kube-api-access-zs5gj\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:39:33.211055 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:33.211017    4364 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb144ffb-880b-44cb-ac19-bbd878eb3bff-openshift-service-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:39:33.211287 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:33.211269    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb144ffb-880b-44cb-ac19-bbd878eb3bff-maas-api-tls\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:39:33.211394 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:33.211383    4364 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb144ffb-880b-44cb-ac19-bbd878eb3bff-metrics-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:39:33.232257 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:33.210732    4364 generic.go:356] "Generic (PLEG): container finished" podID="eb144ffb-880b-44cb-ac19-bbd878eb3bff" containerID="b31895906ec859a4e4800a143ad66c2e83e3afb043fec3f536201a930184b7a3" exitCode=0
Oct 30 13:39:33.232257 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:33.210778    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8b77b0-5f7b7ff96f-wggzj" event={"ID":"eb144ffb-880b-44cb-ac19-bbd878eb3bff","Type":"ContainerDied","Data":"b31895906ec859a4e4800a143ad66c2e83e3afb043fec3f536201a930184b7a3"}
Oct 30 13:39:33.232257 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:33.228019    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8b77b0-5f7b7ff96f-wggzj" event={"ID":"eb144ffb-880b-44cb-ac19-bbd878eb3bff","Type":"ContainerDied","Data":"2d39dd89d030028abf8150d90848e60f3b21a49ed661b05be89d11c2c22d60c7"}
Oct 30 13:39:33.232257 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:33.228059    4364 scope.go:122] "RemoveContainer" containerID="b31895906ec859a4e4800a143ad66c2e83e3afb043fec3f536201a930184b7a3"
Oct 30 13:39:33.232257 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:33.210866    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8b77b0-5f7b7ff96f-wggzj"
Oct 30 13:39:33.325723 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:33.325696    4364 scope.go:122] "RemoveContainer" containerID="b31895906ec859a4e4800a143ad66c2e83e3afb043fec3f536201a930184b7a3"
Oct 30 13:39:33.326555 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:39:33.326474    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b31895906ec859a4e4800a143ad66c2e83e3afb043fec3f536201a930184b7a3\": container with ID starting with b31895906ec859a4e4800a143ad66c2e83e3afb043fec3f536201a930184b7a3 not found: ID does not exist" containerID="b31895906ec859a4e4800a143ad66c2e83e3afb043fec3f536201a930184b7a3"
Oct 30 13:39:33.326555 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:33.326513    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b31895906ec859a4e4800a143ad66c2e83e3afb043fec3f536201a930184b7a3"} err="failed to get container status \"b31895906ec859a4e4800a143ad66c2e83e3afb043fec3f536201a930184b7a3\": rpc error: code = NotFound desc = could not find container \"b31895906ec859a4e4800a143ad66c2e83e3afb043fec3f536201a930184b7a3\": container with ID starting with b31895906ec859a4e4800a143ad66c2e83e3afb043fec3f536201a930184b7a3 not found: ID does not exist"
Oct 30 13:39:33.337588 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:33.337560    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8b77b0-5f7b7ff96f-wggzj"]
Oct 30 13:39:33.346818 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:33.346783    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8b77b0-5f7b7ff96f-wggzj"]
Oct 30 13:39:34.973253 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:34.973218    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb144ffb-880b-44cb-ac19-bbd878eb3bff" path="/var/lib/kubelet/pods/eb144ffb-880b-44cb-ac19-bbd878eb3bff/volumes"
Oct 30 13:39:37.189979 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:37.189933    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:39:40.875721 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:40.875684    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85"]
Oct 30 13:39:40.876239 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:40.875961    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85" podUID="bdd3c1f5-f9fd-400c-90f2-f9693008003a" containerName="istio-proxy" containerID="cri-o://c7e6d0a63ede59639ee66ff16092e925ed35cd6e45acf3c202b86c8db382a944" gracePeriod=30
Oct 30 13:39:41.508810 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:39:41.508775    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:39:42.072126 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:42.071268    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c54f0f-6f599994bb-ntwlx"]
Oct 30 13:39:42.082389 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:42.082354    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c54f0f-6dc7885d9f-h9ghd"]
Oct 30 13:39:42.981913 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:42.981881    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-fca422-69f9fc7b66-ktsvw"]
Oct 30 13:39:42.989189 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:42.989166    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8b77b0-75f95c8744-psdzn"
Oct 30 13:39:43.024021 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:43.023995    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91859899-765d-48fd-9c18-a37942683a5a-kube-api-access-swk67\" (UniqueName: \"kubernetes.io/projected/91859899-765d-48fd-9c18-a37942683a5a-kube-api-access-swk67\") pod \"91859899-765d-48fd-9c18-a37942683a5a\" (UID: \"91859899-765d-48fd-9c18-a37942683a5a\") "
Oct 30 13:39:43.024242 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:43.024084    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91859899-765d-48fd-9c18-a37942683a5a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91859899-765d-48fd-9c18-a37942683a5a-plugins-config-volume\") pod \"91859899-765d-48fd-9c18-a37942683a5a\" (UID: \"91859899-765d-48fd-9c18-a37942683a5a\") "
Oct 30 13:39:43.024664 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:43.024639    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91859899-765d-48fd-9c18-a37942683a5a-plugins-config-volume" pod "91859899-765d-48fd-9c18-a37942683a5a" (UID: "91859899-765d-48fd-9c18-a37942683a5a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:39:43.025118 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:43.025094    4364 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91859899-765d-48fd-9c18-a37942683a5a-plugins-config-volume\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:39:43.026256 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:43.026228    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91859899-765d-48fd-9c18-a37942683a5a-kube-api-access-swk67" pod "91859899-765d-48fd-9c18-a37942683a5a" (UID: "91859899-765d-48fd-9c18-a37942683a5a"). InnerVolumeSpecName "kube-api-access-swk67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:43.126868 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:43.126830    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swk67\" (UniqueName: \"kubernetes.io/projected/91859899-765d-48fd-9c18-a37942683a5a-kube-api-access-swk67\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:39:43.352954 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:43.352869    4364 generic.go:356] "Generic (PLEG): container finished" podID="91859899-765d-48fd-9c18-a37942683a5a" containerID="a79a3febd438e573bd96ce4218a46b2a2cb945b852029ef8c52e05cc7fa27929" exitCode=0
Oct 30 13:39:43.352954 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:43.352913    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8b77b0-75f95c8744-psdzn"
Oct 30 13:39:43.353214 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:43.352953    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8b77b0-75f95c8744-psdzn" event={"ID":"91859899-765d-48fd-9c18-a37942683a5a","Type":"ContainerDied","Data":"a79a3febd438e573bd96ce4218a46b2a2cb945b852029ef8c52e05cc7fa27929"}
Oct 30 13:39:43.353214 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:43.353006    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8b77b0-75f95c8744-psdzn" event={"ID":"91859899-765d-48fd-9c18-a37942683a5a","Type":"ContainerDied","Data":"bba8c450f9b9dade358bee6a130c06032cd6f9767285f94c3ffca67902b482e6"}
Oct 30 13:39:43.353214 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:43.353033    4364 scope.go:122] "RemoveContainer" containerID="a79a3febd438e573bd96ce4218a46b2a2cb945b852029ef8c52e05cc7fa27929"
Oct 30 13:39:43.364370 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:43.364354    4364 scope.go:122] "RemoveContainer" containerID="a79a3febd438e573bd96ce4218a46b2a2cb945b852029ef8c52e05cc7fa27929"
Oct 30 13:39:43.364667 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:39:43.364644    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a79a3febd438e573bd96ce4218a46b2a2cb945b852029ef8c52e05cc7fa27929\": container with ID starting with a79a3febd438e573bd96ce4218a46b2a2cb945b852029ef8c52e05cc7fa27929 not found: ID does not exist" containerID="a79a3febd438e573bd96ce4218a46b2a2cb945b852029ef8c52e05cc7fa27929"
Oct 30 13:39:43.364754 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:43.364672    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a79a3febd438e573bd96ce4218a46b2a2cb945b852029ef8c52e05cc7fa27929"} err="failed to get container status \"a79a3febd438e573bd96ce4218a46b2a2cb945b852029ef8c52e05cc7fa27929\": rpc error: code = NotFound desc = could not find container \"a79a3febd438e573bd96ce4218a46b2a2cb945b852029ef8c52e05cc7fa27929\": container with ID starting with a79a3febd438e573bd96ce4218a46b2a2cb945b852029ef8c52e05cc7fa27929 not found: ID does not exist"
Oct 30 13:39:43.411832 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:43.411805    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-8b77b0-75f95c8744-psdzn"]
Oct 30 13:39:43.418587 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:43.418562    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-8b77b0-75f95c8744-psdzn"]
Oct 30 13:39:44.972565 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:44.972528    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91859899-765d-48fd-9c18-a37942683a5a" path="/var/lib/kubelet/pods/91859899-765d-48fd-9c18-a37942683a5a/volumes"
Oct 30 13:39:45.636358 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:45.636326    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-b4fb25/ipp-route-b4fb25-kserve-5d46888457-25qdx"]
Oct 30 13:39:45.637206 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:45.637185    4364 cpu_manager.go:414] "containerMap: removing container" podUID="91859899-765d-48fd-9c18-a37942683a5a" containerName="payload-processing"
Oct 30 13:39:45.637206 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:45.637205    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91859899-765d-48fd-9c18-a37942683a5a" containerName="payload-processing"
Oct 30 13:39:45.637406 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:45.637255    4364 cpu_manager.go:414] "containerMap: removing container" podUID="eb144ffb-880b-44cb-ac19-bbd878eb3bff" containerName="maas-api"
Oct 30 13:39:45.637406 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:45.637263    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb144ffb-880b-44cb-ac19-bbd878eb3bff" containerName="maas-api"
Oct 30 13:39:45.637512 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:45.637434    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="91859899-765d-48fd-9c18-a37942683a5a" containerName="payload-processing"
Oct 30 13:39:45.637512 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:45.637455    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb144ffb-880b-44cb-ac19-bbd878eb3bff" containerName="maas-api"
Oct 30 13:39:45.643168 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:45.643142    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-b4fb25/ipp-route-b4fb25-kserve-5d46888457-25qdx"
Oct 30 13:39:45.663055 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:45.663028    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-b4fb25/ipp-route-b4fb25-kserve-5d46888457-25qdx"]
Oct 30 13:39:45.667647 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:45.666763    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-b4fb25\"/\"default-dockercfg-2r5z7\""
Oct 30 13:39:45.667647 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:45.666773    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-b4fb25\"/\"kube-root-ca.crt\""
Oct 30 13:39:45.670652 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:45.670627    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-b4fb25\"/\"ipp-route-b4fb25-kserve-self-signed-certs\""
Oct 30 13:39:45.670788 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:45.670717    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-b4fb25\"/\"openshift-service-ca.crt\""
Oct 30 13:39:45.704904 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:45.704872    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e77381ab-005b-4b76-81b0-ada11566adca-dshm\") pod \"ipp-route-b4fb25-kserve-5d46888457-25qdx\" (UID: \"e77381ab-005b-4b76-81b0-ada11566adca\") " pod="ai-tenant-e2e-ipp-a-b4fb25/ipp-route-b4fb25-kserve-5d46888457-25qdx"
Oct 30 13:39:45.705066 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:45.704996    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e77381ab-005b-4b76-81b0-ada11566adca-model-cache\") pod \"ipp-route-b4fb25-kserve-5d46888457-25qdx\" (UID: \"e77381ab-005b-4b76-81b0-ada11566adca\") " pod="ai-tenant-e2e-ipp-a-b4fb25/ipp-route-b4fb25-kserve-5d46888457-25qdx"
Oct 30 13:39:45.705176 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:45.705160    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e77381ab-005b-4b76-81b0-ada11566adca-tls-certs\") pod \"ipp-route-b4fb25-kserve-5d46888457-25qdx\" (UID: \"e77381ab-005b-4b76-81b0-ada11566adca\") " pod="ai-tenant-e2e-ipp-a-b4fb25/ipp-route-b4fb25-kserve-5d46888457-25qdx"
Oct 30 13:39:45.705238 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:45.705226    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvnr4\" (UniqueName: \"kubernetes.io/projected/e77381ab-005b-4b76-81b0-ada11566adca-kube-api-access-wvnr4\") pod \"ipp-route-b4fb25-kserve-5d46888457-25qdx\" (UID: \"e77381ab-005b-4b76-81b0-ada11566adca\") " pod="ai-tenant-e2e-ipp-a-b4fb25/ipp-route-b4fb25-kserve-5d46888457-25qdx"
Oct 30 13:39:45.705292 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:45.705243    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e77381ab-005b-4b76-81b0-ada11566adca-home\") pod \"ipp-route-b4fb25-kserve-5d46888457-25qdx\" (UID: \"e77381ab-005b-4b76-81b0-ada11566adca\") " pod="ai-tenant-e2e-ipp-a-b4fb25/ipp-route-b4fb25-kserve-5d46888457-25qdx"
Oct 30 13:39:45.705341 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:45.705294    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e77381ab-005b-4b76-81b0-ada11566adca-tmp-dir\") pod \"ipp-route-b4fb25-kserve-5d46888457-25qdx\" (UID: \"e77381ab-005b-4b76-81b0-ada11566adca\") " pod="ai-tenant-e2e-ipp-a-b4fb25/ipp-route-b4fb25-kserve-5d46888457-25qdx"
Oct 30 13:39:45.806952 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:45.806922    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e77381ab-005b-4b76-81b0-ada11566adca-dshm\") pod \"ipp-route-b4fb25-kserve-5d46888457-25qdx\" (UID: \"e77381ab-005b-4b76-81b0-ada11566adca\") " pod="ai-tenant-e2e-ipp-a-b4fb25/ipp-route-b4fb25-kserve-5d46888457-25qdx"
Oct 30 13:39:45.807122 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:45.807006    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e77381ab-005b-4b76-81b0-ada11566adca-model-cache\") pod \"ipp-route-b4fb25-kserve-5d46888457-25qdx\" (UID: \"e77381ab-005b-4b76-81b0-ada11566adca\") " pod="ai-tenant-e2e-ipp-a-b4fb25/ipp-route-b4fb25-kserve-5d46888457-25qdx"
Oct 30 13:39:45.807179 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:45.807119    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e77381ab-005b-4b76-81b0-ada11566adca-tls-certs\") pod \"ipp-route-b4fb25-kserve-5d46888457-25qdx\" (UID: \"e77381ab-005b-4b76-81b0-ada11566adca\") " pod="ai-tenant-e2e-ipp-a-b4fb25/ipp-route-b4fb25-kserve-5d46888457-25qdx"
Oct 30 13:39:45.807238 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:45.807208    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvnr4\" (UniqueName: \"kubernetes.io/projected/e77381ab-005b-4b76-81b0-ada11566adca-kube-api-access-wvnr4\") pod \"ipp-route-b4fb25-kserve-5d46888457-25qdx\" (UID: \"e77381ab-005b-4b76-81b0-ada11566adca\") " pod="ai-tenant-e2e-ipp-a-b4fb25/ipp-route-b4fb25-kserve-5d46888457-25qdx"
Oct 30 13:39:45.807346 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:45.807321    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e77381ab-005b-4b76-81b0-ada11566adca-home\") pod \"ipp-route-b4fb25-kserve-5d46888457-25qdx\" (UID: \"e77381ab-005b-4b76-81b0-ada11566adca\") " pod="ai-tenant-e2e-ipp-a-b4fb25/ipp-route-b4fb25-kserve-5d46888457-25qdx"
Oct 30 13:39:45.807414 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:45.807365    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e77381ab-005b-4b76-81b0-ada11566adca-tmp-dir\") pod \"ipp-route-b4fb25-kserve-5d46888457-25qdx\" (UID: \"e77381ab-005b-4b76-81b0-ada11566adca\") " pod="ai-tenant-e2e-ipp-a-b4fb25/ipp-route-b4fb25-kserve-5d46888457-25qdx"
Oct 30 13:39:45.807470 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:45.807424    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e77381ab-005b-4b76-81b0-ada11566adca-model-cache\") pod \"ipp-route-b4fb25-kserve-5d46888457-25qdx\" (UID: \"e77381ab-005b-4b76-81b0-ada11566adca\") " pod="ai-tenant-e2e-ipp-a-b4fb25/ipp-route-b4fb25-kserve-5d46888457-25qdx"
Oct 30 13:39:45.807741 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:45.807712    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e77381ab-005b-4b76-81b0-ada11566adca-home\") pod \"ipp-route-b4fb25-kserve-5d46888457-25qdx\" (UID: \"e77381ab-005b-4b76-81b0-ada11566adca\") " pod="ai-tenant-e2e-ipp-a-b4fb25/ipp-route-b4fb25-kserve-5d46888457-25qdx"
Oct 30 13:39:45.807865 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:45.807748    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e77381ab-005b-4b76-81b0-ada11566adca-tmp-dir\") pod \"ipp-route-b4fb25-kserve-5d46888457-25qdx\" (UID: \"e77381ab-005b-4b76-81b0-ada11566adca\") " pod="ai-tenant-e2e-ipp-a-b4fb25/ipp-route-b4fb25-kserve-5d46888457-25qdx"
Oct 30 13:39:45.809670 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:45.809647    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e77381ab-005b-4b76-81b0-ada11566adca-dshm\") pod \"ipp-route-b4fb25-kserve-5d46888457-25qdx\" (UID: \"e77381ab-005b-4b76-81b0-ada11566adca\") " pod="ai-tenant-e2e-ipp-a-b4fb25/ipp-route-b4fb25-kserve-5d46888457-25qdx"
Oct 30 13:39:45.809968 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:45.809946    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e77381ab-005b-4b76-81b0-ada11566adca-tls-certs\") pod \"ipp-route-b4fb25-kserve-5d46888457-25qdx\" (UID: \"e77381ab-005b-4b76-81b0-ada11566adca\") " pod="ai-tenant-e2e-ipp-a-b4fb25/ipp-route-b4fb25-kserve-5d46888457-25qdx"
Oct 30 13:39:45.841812 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:45.841787    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvnr4\" (UniqueName: \"kubernetes.io/projected/e77381ab-005b-4b76-81b0-ada11566adca-kube-api-access-wvnr4\") pod \"ipp-route-b4fb25-kserve-5d46888457-25qdx\" (UID: \"e77381ab-005b-4b76-81b0-ada11566adca\") " pod="ai-tenant-e2e-ipp-a-b4fb25/ipp-route-b4fb25-kserve-5d46888457-25qdx"
Oct 30 13:39:46.094801 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:46.094752    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-b4fb25/ipp-route-b4fb25-kserve-5d46888457-25qdx"
Oct 30 13:39:46.256209 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:46.254982    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85"
Oct 30 13:39:46.377078 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:46.377046    4364 generic.go:356] "Generic (PLEG): container finished" podID="bdd3c1f5-f9fd-400c-90f2-f9693008003a" containerID="c7e6d0a63ede59639ee66ff16092e925ed35cd6e45acf3c202b86c8db382a944" exitCode=0
Oct 30 13:39:46.377246 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:46.377153    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85"
Oct 30 13:39:46.377246 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:46.377177    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85" event={"ID":"bdd3c1f5-f9fd-400c-90f2-f9693008003a","Type":"ContainerDied","Data":"c7e6d0a63ede59639ee66ff16092e925ed35cd6e45acf3c202b86c8db382a944"}
Oct 30 13:39:46.377246 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:46.377238    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85" event={"ID":"bdd3c1f5-f9fd-400c-90f2-f9693008003a","Type":"ContainerDied","Data":"a2f3eef98f6456a6d5769a7dd5c317aa94440179a5836a64dff6e377fa564dd0"}
Oct 30 13:39:46.377380 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:46.377270    4364 scope.go:122] "RemoveContainer" containerID="c7e6d0a63ede59639ee66ff16092e925ed35cd6e45acf3c202b86c8db382a944"
Oct 30 13:39:46.388116 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:46.388096    4364 scope.go:122] "RemoveContainer" containerID="3ba0c9e791b155c0c55ef49e6410558e335a26c9456e87f0bfb5ec5a720c041d"
Oct 30 13:39:46.399570 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:46.399541    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdd3c1f5-f9fd-400c-90f2-f9693008003a-istio-token\" (UniqueName: \"kubernetes.io/projected/bdd3c1f5-f9fd-400c-90f2-f9693008003a-istio-token\") pod \"bdd3c1f5-f9fd-400c-90f2-f9693008003a\" (UID: \"bdd3c1f5-f9fd-400c-90f2-f9693008003a\") "
Oct 30 13:39:46.399729 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:46.399583    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bdd3c1f5-f9fd-400c-90f2-f9693008003a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdd3c1f5-f9fd-400c-90f2-f9693008003a-istio-podinfo\") pod \"bdd3c1f5-f9fd-400c-90f2-f9693008003a\" (UID: \"bdd3c1f5-f9fd-400c-90f2-f9693008003a\") "
Oct 30 13:39:46.399814 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:46.399797    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdd3c1f5-f9fd-400c-90f2-f9693008003a-kube-api-access-7ns6p\" (UniqueName: \"kubernetes.io/projected/bdd3c1f5-f9fd-400c-90f2-f9693008003a-kube-api-access-7ns6p\") pod \"bdd3c1f5-f9fd-400c-90f2-f9693008003a\" (UID: \"bdd3c1f5-f9fd-400c-90f2-f9693008003a\") "
Oct 30 13:39:46.399880 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:46.399830    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdd3c1f5-f9fd-400c-90f2-f9693008003a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdd3c1f5-f9fd-400c-90f2-f9693008003a-istio-data\") pod \"bdd3c1f5-f9fd-400c-90f2-f9693008003a\" (UID: \"bdd3c1f5-f9fd-400c-90f2-f9693008003a\") "
Oct 30 13:39:46.399932 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:46.399893    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdd3c1f5-f9fd-400c-90f2-f9693008003a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdd3c1f5-f9fd-400c-90f2-f9693008003a-istiod-ca-cert\") pod \"bdd3c1f5-f9fd-400c-90f2-f9693008003a\" (UID: \"bdd3c1f5-f9fd-400c-90f2-f9693008003a\") "
Oct 30 13:39:46.400004 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:46.399959    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdd3c1f5-f9fd-400c-90f2-f9693008003a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdd3c1f5-f9fd-400c-90f2-f9693008003a-workload-certs\") pod \"bdd3c1f5-f9fd-400c-90f2-f9693008003a\" (UID: \"bdd3c1f5-f9fd-400c-90f2-f9693008003a\") "
Oct 30 13:39:46.400060 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:46.400003    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdd3c1f5-f9fd-400c-90f2-f9693008003a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdd3c1f5-f9fd-400c-90f2-f9693008003a-workload-socket\") pod \"bdd3c1f5-f9fd-400c-90f2-f9693008003a\" (UID: \"bdd3c1f5-f9fd-400c-90f2-f9693008003a\") "
Oct 30 13:39:46.400060 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:46.400044    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdd3c1f5-f9fd-400c-90f2-f9693008003a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdd3c1f5-f9fd-400c-90f2-f9693008003a-credential-socket\") pod \"bdd3c1f5-f9fd-400c-90f2-f9693008003a\" (UID: \"bdd3c1f5-f9fd-400c-90f2-f9693008003a\") "
Oct 30 13:39:46.400205 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:46.400190    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdd3c1f5-f9fd-400c-90f2-f9693008003a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdd3c1f5-f9fd-400c-90f2-f9693008003a-istio-envoy\") pod \"bdd3c1f5-f9fd-400c-90f2-f9693008003a\" (UID: \"bdd3c1f5-f9fd-400c-90f2-f9693008003a\") "
Oct 30 13:39:46.402082 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:46.400671    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdd3c1f5-f9fd-400c-90f2-f9693008003a-istio-data" pod "bdd3c1f5-f9fd-400c-90f2-f9693008003a" (UID: "bdd3c1f5-f9fd-400c-90f2-f9693008003a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:46.402082 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:46.400971    4364 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdd3c1f5-f9fd-400c-90f2-f9693008003a-istio-data\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:39:46.402082 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:46.401208    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdd3c1f5-f9fd-400c-90f2-f9693008003a-workload-certs" pod "bdd3c1f5-f9fd-400c-90f2-f9693008003a" (UID: "bdd3c1f5-f9fd-400c-90f2-f9693008003a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:46.402082 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:46.401489    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdd3c1f5-f9fd-400c-90f2-f9693008003a-istiod-ca-cert" pod "bdd3c1f5-f9fd-400c-90f2-f9693008003a" (UID: "bdd3c1f5-f9fd-400c-90f2-f9693008003a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:39:46.402679 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:46.402643    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdd3c1f5-f9fd-400c-90f2-f9693008003a-workload-socket" pod "bdd3c1f5-f9fd-400c-90f2-f9693008003a" (UID: "bdd3c1f5-f9fd-400c-90f2-f9693008003a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:46.402783 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:46.402688    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdd3c1f5-f9fd-400c-90f2-f9693008003a-credential-socket" pod "bdd3c1f5-f9fd-400c-90f2-f9693008003a" (UID: "bdd3c1f5-f9fd-400c-90f2-f9693008003a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:46.403389 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:46.403360    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdd3c1f5-f9fd-400c-90f2-f9693008003a-istio-token" pod "bdd3c1f5-f9fd-400c-90f2-f9693008003a" (UID: "bdd3c1f5-f9fd-400c-90f2-f9693008003a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:46.404126 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:46.404005    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bdd3c1f5-f9fd-400c-90f2-f9693008003a-istio-podinfo" pod "bdd3c1f5-f9fd-400c-90f2-f9693008003a" (UID: "bdd3c1f5-f9fd-400c-90f2-f9693008003a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:39:46.404126 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:46.404008    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdd3c1f5-f9fd-400c-90f2-f9693008003a-istio-envoy" pod "bdd3c1f5-f9fd-400c-90f2-f9693008003a" (UID: "bdd3c1f5-f9fd-400c-90f2-f9693008003a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:46.404428 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:46.404406    4364 scope.go:122] "RemoveContainer" containerID="c7e6d0a63ede59639ee66ff16092e925ed35cd6e45acf3c202b86c8db382a944"
Oct 30 13:39:46.404517 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:46.404502    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdd3c1f5-f9fd-400c-90f2-f9693008003a-kube-api-access-7ns6p" pod "bdd3c1f5-f9fd-400c-90f2-f9693008003a" (UID: "bdd3c1f5-f9fd-400c-90f2-f9693008003a"). InnerVolumeSpecName "kube-api-access-7ns6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:46.404780 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:39:46.404755    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7e6d0a63ede59639ee66ff16092e925ed35cd6e45acf3c202b86c8db382a944\": container with ID starting with c7e6d0a63ede59639ee66ff16092e925ed35cd6e45acf3c202b86c8db382a944 not found: ID does not exist" containerID="c7e6d0a63ede59639ee66ff16092e925ed35cd6e45acf3c202b86c8db382a944"
Oct 30 13:39:46.404835 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:46.404788    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7e6d0a63ede59639ee66ff16092e925ed35cd6e45acf3c202b86c8db382a944"} err="failed to get container status \"c7e6d0a63ede59639ee66ff16092e925ed35cd6e45acf3c202b86c8db382a944\": rpc error: code = NotFound desc = could not find container \"c7e6d0a63ede59639ee66ff16092e925ed35cd6e45acf3c202b86c8db382a944\": container with ID starting with c7e6d0a63ede59639ee66ff16092e925ed35cd6e45acf3c202b86c8db382a944 not found: ID does not exist"
Oct 30 13:39:46.404835 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:46.404807    4364 scope.go:122] "RemoveContainer" containerID="3ba0c9e791b155c0c55ef49e6410558e335a26c9456e87f0bfb5ec5a720c041d"
Oct 30 13:39:46.405059 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:39:46.405043    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ba0c9e791b155c0c55ef49e6410558e335a26c9456e87f0bfb5ec5a720c041d\": container with ID starting with 3ba0c9e791b155c0c55ef49e6410558e335a26c9456e87f0bfb5ec5a720c041d not found: ID does not exist" containerID="3ba0c9e791b155c0c55ef49e6410558e335a26c9456e87f0bfb5ec5a720c041d"
Oct 30 13:39:46.405109 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:46.405065    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ba0c9e791b155c0c55ef49e6410558e335a26c9456e87f0bfb5ec5a720c041d"} err="failed to get container status \"3ba0c9e791b155c0c55ef49e6410558e335a26c9456e87f0bfb5ec5a720c041d\": rpc error: code = NotFound desc = could not find container \"3ba0c9e791b155c0c55ef49e6410558e335a26c9456e87f0bfb5ec5a720c041d\": container with ID starting with 3ba0c9e791b155c0c55ef49e6410558e335a26c9456e87f0bfb5ec5a720c041d not found: ID does not exist"
Oct 30 13:39:46.422096 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:46.422076    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-b4fb25/ipp-route-b4fb25-kserve-5d46888457-25qdx"]
Oct 30 13:39:46.422798 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:39:46.422778    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode77381ab_005b_4b76_81b0_ada11566adca.slice/crio-b4caf1a8e9e7675e69a6aee6da3623afd9a8e84a9f13c4039e8074a8bdadfb16 WatchSource:0}: Error finding container b4caf1a8e9e7675e69a6aee6da3623afd9a8e84a9f13c4039e8074a8bdadfb16: Status 404 returned error can't find the container with id b4caf1a8e9e7675e69a6aee6da3623afd9a8e84a9f13c4039e8074a8bdadfb16
Oct 30 13:39:46.502551 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:46.502523    4364 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdd3c1f5-f9fd-400c-90f2-f9693008003a-istio-envoy\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:39:46.502551 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:46.502549    4364 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdd3c1f5-f9fd-400c-90f2-f9693008003a-istio-token\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:39:46.502726 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:46.502560    4364 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdd3c1f5-f9fd-400c-90f2-f9693008003a-istio-podinfo\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:39:46.502726 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:46.502569    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ns6p\" (UniqueName: \"kubernetes.io/projected/bdd3c1f5-f9fd-400c-90f2-f9693008003a-kube-api-access-7ns6p\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:39:46.502726 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:46.502580    4364 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdd3c1f5-f9fd-400c-90f2-f9693008003a-istiod-ca-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:39:46.502726 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:46.502590    4364 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdd3c1f5-f9fd-400c-90f2-f9693008003a-workload-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:39:46.502726 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:46.502613    4364 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdd3c1f5-f9fd-400c-90f2-f9693008003a-workload-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:39:46.502726 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:46.502625    4364 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdd3c1f5-f9fd-400c-90f2-f9693008003a-credential-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:39:46.726739 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:46.726710    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85"]
Oct 30 13:39:46.731145 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:46.731121    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-8b77b0-openshift-default-9c8bd5559-2hr85"]
Oct 30 13:39:46.972754 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:46.972673    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdd3c1f5-f9fd-400c-90f2-f9693008003a" path="/var/lib/kubelet/pods/bdd3c1f5-f9fd-400c-90f2-f9693008003a/volumes"
Oct 30 13:39:47.071681 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:47.071646    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-c54f0f-6f599994bb-ntwlx" podUID="094f69e3-b678-4281-9d9b-c0a4b25d0895" containerName="payload-processing" containerID="cri-o://862e3ddccd982605540d1143036989b15d27bc21934c7242d3358f498eeac9f3" gracePeriod=25
Oct 30 13:39:47.083470 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:47.083432    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c54f0f-6dc7885d9f-h9ghd" podUID="88af8672-a246-457e-9cfa-534c8d1d604d" containerName="payload-pre-processing" containerID="cri-o://30344301d8d5cdb8c4ed00fbef02bdc48933aa0e2a787c2a5d84ab56a66eb7c4" gracePeriod=25
Oct 30 13:39:47.396478 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:47.396040    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-b4fb25/ipp-route-b4fb25-kserve-5d46888457-25qdx" event={"ID":"e77381ab-005b-4b76-81b0-ada11566adca","Type":"ContainerStarted","Data":"878db7143ceb0dea950ced2298052a25023c1bb671c22e6967424f5e335a500d"}
Oct 30 13:39:47.396478 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:47.396094    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-b4fb25/ipp-route-b4fb25-kserve-5d46888457-25qdx" event={"ID":"e77381ab-005b-4b76-81b0-ada11566adca","Type":"ContainerStarted","Data":"b4caf1a8e9e7675e69a6aee6da3623afd9a8e84a9f13c4039e8074a8bdadfb16"}
Oct 30 13:39:47.397818 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:47.397797    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c54f0f-6f599994bb-ntwlx"
Oct 30 13:39:47.397941 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:47.397818    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c54f0f-6dc7885d9f-h9ghd" event={"ID":"88af8672-a246-457e-9cfa-534c8d1d604d","Type":"ContainerDied","Data":"30344301d8d5cdb8c4ed00fbef02bdc48933aa0e2a787c2a5d84ab56a66eb7c4"}
Oct 30 13:39:47.397941 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:47.397799    4364 generic.go:356] "Generic (PLEG): container finished" podID="88af8672-a246-457e-9cfa-534c8d1d604d" containerID="30344301d8d5cdb8c4ed00fbef02bdc48933aa0e2a787c2a5d84ab56a66eb7c4" exitCode=0
Oct 30 13:39:47.399364 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:47.399341    4364 generic.go:356] "Generic (PLEG): container finished" podID="094f69e3-b678-4281-9d9b-c0a4b25d0895" containerID="862e3ddccd982605540d1143036989b15d27bc21934c7242d3358f498eeac9f3" exitCode=0
Oct 30 13:39:47.399478 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:47.399370    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c54f0f-6f599994bb-ntwlx"
Oct 30 13:39:47.399478 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:47.399416    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c54f0f-6f599994bb-ntwlx" event={"ID":"094f69e3-b678-4281-9d9b-c0a4b25d0895","Type":"ContainerDied","Data":"862e3ddccd982605540d1143036989b15d27bc21934c7242d3358f498eeac9f3"}
Oct 30 13:39:47.399478 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:47.399461    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c54f0f-6f599994bb-ntwlx" event={"ID":"094f69e3-b678-4281-9d9b-c0a4b25d0895","Type":"ContainerDied","Data":"7005b971ab9f796b5cb1d90e9bf76260163d49b79add8afaca2a971c7e4f1ef5"}
Oct 30 13:39:47.399689 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:47.399486    4364 scope.go:122] "RemoveContainer" containerID="862e3ddccd982605540d1143036989b15d27bc21934c7242d3358f498eeac9f3"
Oct 30 13:39:47.411695 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:47.411636    4364 scope.go:122] "RemoveContainer" containerID="862e3ddccd982605540d1143036989b15d27bc21934c7242d3358f498eeac9f3"
Oct 30 13:39:47.411920 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:39:47.411898    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"862e3ddccd982605540d1143036989b15d27bc21934c7242d3358f498eeac9f3\": container with ID starting with 862e3ddccd982605540d1143036989b15d27bc21934c7242d3358f498eeac9f3 not found: ID does not exist" containerID="862e3ddccd982605540d1143036989b15d27bc21934c7242d3358f498eeac9f3"
Oct 30 13:39:47.411983 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:47.411927    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"862e3ddccd982605540d1143036989b15d27bc21934c7242d3358f498eeac9f3"} err="failed to get container status \"862e3ddccd982605540d1143036989b15d27bc21934c7242d3358f498eeac9f3\": rpc error: code = NotFound desc = could not find container \"862e3ddccd982605540d1143036989b15d27bc21934c7242d3358f498eeac9f3\": container with ID starting with 862e3ddccd982605540d1143036989b15d27bc21934c7242d3358f498eeac9f3 not found: ID does not exist"
Oct 30 13:39:47.413725 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:47.413705    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c54f0f-6dc7885d9f-h9ghd"
Oct 30 13:39:47.414031 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:47.414012    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/094f69e3-b678-4281-9d9b-c0a4b25d0895-kube-api-access-lr2wp\" (UniqueName: \"kubernetes.io/projected/094f69e3-b678-4281-9d9b-c0a4b25d0895-kube-api-access-lr2wp\") pod \"094f69e3-b678-4281-9d9b-c0a4b25d0895\" (UID: \"094f69e3-b678-4281-9d9b-c0a4b25d0895\") "
Oct 30 13:39:47.414096 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:47.414086    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/094f69e3-b678-4281-9d9b-c0a4b25d0895-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/094f69e3-b678-4281-9d9b-c0a4b25d0895-plugins-config-volume\") pod \"094f69e3-b678-4281-9d9b-c0a4b25d0895\" (UID: \"094f69e3-b678-4281-9d9b-c0a4b25d0895\") "
Oct 30 13:39:47.414447 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:47.414423    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/094f69e3-b678-4281-9d9b-c0a4b25d0895-plugins-config-volume" pod "094f69e3-b678-4281-9d9b-c0a4b25d0895" (UID: "094f69e3-b678-4281-9d9b-c0a4b25d0895"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:39:47.414748 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:47.414728    4364 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/094f69e3-b678-4281-9d9b-c0a4b25d0895-plugins-config-volume\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:39:47.416022 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:47.415990    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/094f69e3-b678-4281-9d9b-c0a4b25d0895-kube-api-access-lr2wp" pod "094f69e3-b678-4281-9d9b-c0a4b25d0895" (UID: "094f69e3-b678-4281-9d9b-c0a4b25d0895"). InnerVolumeSpecName "kube-api-access-lr2wp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:47.441213 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:47.441172    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-b4fb25/ipp-route-b4fb25-kserve-5d46888457-25qdx" podStartSLOduration=2.441161587 podStartE2EDuration="2.441161587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:39:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:39:47.436706865 +0000 UTC m=+2886.603071512" watchObservedRunningTime="2026-10-30 13:39:47.441161587 +0000 UTC m=+2886.607526234"
Oct 30 13:39:47.515350 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:47.515272    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88af8672-a246-457e-9cfa-534c8d1d604d-kube-api-access-8tjxq\" (UniqueName: \"kubernetes.io/projected/88af8672-a246-457e-9cfa-534c8d1d604d-kube-api-access-8tjxq\") pod \"88af8672-a246-457e-9cfa-534c8d1d604d\" (UID: \"88af8672-a246-457e-9cfa-534c8d1d604d\") "
Oct 30 13:39:47.515528 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:47.515509    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88af8672-a246-457e-9cfa-534c8d1d604d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88af8672-a246-457e-9cfa-534c8d1d604d-plugins-config-volume\") pod \"88af8672-a246-457e-9cfa-534c8d1d604d\" (UID: \"88af8672-a246-457e-9cfa-534c8d1d604d\") "
Oct 30 13:39:47.516195 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:47.516165    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88af8672-a246-457e-9cfa-534c8d1d604d-plugins-config-volume" pod "88af8672-a246-457e-9cfa-534c8d1d604d" (UID: "88af8672-a246-457e-9cfa-534c8d1d604d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:39:47.516195 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:47.516178    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lr2wp\" (UniqueName: \"kubernetes.io/projected/094f69e3-b678-4281-9d9b-c0a4b25d0895-kube-api-access-lr2wp\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:39:47.517711 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:47.517688    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88af8672-a246-457e-9cfa-534c8d1d604d-kube-api-access-8tjxq" pod "88af8672-a246-457e-9cfa-534c8d1d604d" (UID: "88af8672-a246-457e-9cfa-534c8d1d604d"). InnerVolumeSpecName "kube-api-access-8tjxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:47.623624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:47.621683    4364 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88af8672-a246-457e-9cfa-534c8d1d604d-plugins-config-volume\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:39:47.623624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:47.621723    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tjxq\" (UniqueName: \"kubernetes.io/projected/88af8672-a246-457e-9cfa-534c8d1d604d-kube-api-access-8tjxq\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:39:47.784718 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:47.783115    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c54f0f-6f599994bb-ntwlx"]
Oct 30 13:39:47.792053 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:47.791156    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c54f0f-6f599994bb-ntwlx"]
Oct 30 13:39:47.983283 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:47.983239    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-fca422-69f9fc7b66-ktsvw" podUID="4e1fd541-b715-4ba4-a160-bd86c2076d92" containerName="payload-processing" containerID="cri-o://92ef052f857a14f10f76eb71294f0b152b2f5a332d88c8a7210c9fe130190f81" gracePeriod=25
Oct 30 13:39:48.098014 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:48.097935    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-b4fb25/ipp-route-b4fb25-kserve-5d46888457-25qdx"
Oct 30 13:39:48.275723 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:48.275694    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-fca422-69f9fc7b66-ktsvw"
Oct 30 13:39:48.335739 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:48.335703    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e1fd541-b715-4ba4-a160-bd86c2076d92-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e1fd541-b715-4ba4-a160-bd86c2076d92-plugins-config-volume\") pod \"4e1fd541-b715-4ba4-a160-bd86c2076d92\" (UID: \"4e1fd541-b715-4ba4-a160-bd86c2076d92\") "
Oct 30 13:39:48.335919 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:48.335845    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e1fd541-b715-4ba4-a160-bd86c2076d92-kube-api-access-kl2kn\" (UniqueName: \"kubernetes.io/projected/4e1fd541-b715-4ba4-a160-bd86c2076d92-kube-api-access-kl2kn\") pod \"4e1fd541-b715-4ba4-a160-bd86c2076d92\" (UID: \"4e1fd541-b715-4ba4-a160-bd86c2076d92\") "
Oct 30 13:39:48.336118 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:48.336089    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e1fd541-b715-4ba4-a160-bd86c2076d92-plugins-config-volume" pod "4e1fd541-b715-4ba4-a160-bd86c2076d92" (UID: "4e1fd541-b715-4ba4-a160-bd86c2076d92"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:39:48.336758 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:48.336734    4364 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e1fd541-b715-4ba4-a160-bd86c2076d92-plugins-config-volume\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:39:48.338248 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:48.338206    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e1fd541-b715-4ba4-a160-bd86c2076d92-kube-api-access-kl2kn" pod "4e1fd541-b715-4ba4-a160-bd86c2076d92" (UID: "4e1fd541-b715-4ba4-a160-bd86c2076d92"). InnerVolumeSpecName "kube-api-access-kl2kn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:48.411224 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:48.411191    4364 generic.go:356] "Generic (PLEG): container finished" podID="4e1fd541-b715-4ba4-a160-bd86c2076d92" containerID="92ef052f857a14f10f76eb71294f0b152b2f5a332d88c8a7210c9fe130190f81" exitCode=0
Oct 30 13:39:48.411669 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:48.411237    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-fca422-69f9fc7b66-ktsvw"
Oct 30 13:39:48.411669 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:48.411269    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-fca422-69f9fc7b66-ktsvw" event={"ID":"4e1fd541-b715-4ba4-a160-bd86c2076d92","Type":"ContainerDied","Data":"92ef052f857a14f10f76eb71294f0b152b2f5a332d88c8a7210c9fe130190f81"}
Oct 30 13:39:48.411669 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:48.411324    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-fca422-69f9fc7b66-ktsvw" event={"ID":"4e1fd541-b715-4ba4-a160-bd86c2076d92","Type":"ContainerDied","Data":"46e6a7af5c5c6321ca0118f8ca3ba5c434715b48a0cc8d3d441e3c643a7bdd7b"}
Oct 30 13:39:48.411669 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:48.411353    4364 scope.go:122] "RemoveContainer" containerID="92ef052f857a14f10f76eb71294f0b152b2f5a332d88c8a7210c9fe130190f81"
Oct 30 13:39:48.413019 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:48.412998    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c54f0f-6dc7885d9f-h9ghd"
Oct 30 13:39:48.413109 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:48.413020    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c54f0f-6dc7885d9f-h9ghd" event={"ID":"88af8672-a246-457e-9cfa-534c8d1d604d","Type":"ContainerDied","Data":"baaa43a30e120d3d307443df64f4c160fbe1f302c9d68e804d705f8e38f8f1d3"}
Oct 30 13:39:48.426467 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:48.426443    4364 scope.go:122] "RemoveContainer" containerID="92ef052f857a14f10f76eb71294f0b152b2f5a332d88c8a7210c9fe130190f81"
Oct 30 13:39:48.426778 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:39:48.426753    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92ef052f857a14f10f76eb71294f0b152b2f5a332d88c8a7210c9fe130190f81\": container with ID starting with 92ef052f857a14f10f76eb71294f0b152b2f5a332d88c8a7210c9fe130190f81 not found: ID does not exist" containerID="92ef052f857a14f10f76eb71294f0b152b2f5a332d88c8a7210c9fe130190f81"
Oct 30 13:39:48.426894 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:48.426789    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92ef052f857a14f10f76eb71294f0b152b2f5a332d88c8a7210c9fe130190f81"} err="failed to get container status \"92ef052f857a14f10f76eb71294f0b152b2f5a332d88c8a7210c9fe130190f81\": rpc error: code = NotFound desc = could not find container \"92ef052f857a14f10f76eb71294f0b152b2f5a332d88c8a7210c9fe130190f81\": container with ID starting with 92ef052f857a14f10f76eb71294f0b152b2f5a332d88c8a7210c9fe130190f81 not found: ID does not exist"
Oct 30 13:39:48.426894 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:48.426810    4364 scope.go:122] "RemoveContainer" containerID="30344301d8d5cdb8c4ed00fbef02bdc48933aa0e2a787c2a5d84ab56a66eb7c4"
Oct 30 13:39:48.438713 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:48.438694    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kl2kn\" (UniqueName: \"kubernetes.io/projected/4e1fd541-b715-4ba4-a160-bd86c2076d92-kube-api-access-kl2kn\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:39:48.466389 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:48.465851    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-fca422-69f9fc7b66-ktsvw"]
Oct 30 13:39:48.470538 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:48.470510    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-fca422-69f9fc7b66-ktsvw"]
Oct 30 13:39:48.503111 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:48.503079    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c54f0f-6dc7885d9f-h9ghd"]
Oct 30 13:39:48.509721 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:48.509694    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c54f0f-6dc7885d9f-h9ghd"]
Oct 30 13:39:48.973591 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:48.973545    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="094f69e3-b678-4281-9d9b-c0a4b25d0895" path="/var/lib/kubelet/pods/094f69e3-b678-4281-9d9b-c0a4b25d0895/volumes"
Oct 30 13:39:48.974184 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:48.974154    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e1fd541-b715-4ba4-a160-bd86c2076d92" path="/var/lib/kubelet/pods/4e1fd541-b715-4ba4-a160-bd86c2076d92/volumes"
Oct 30 13:39:48.974592 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:48.974576    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88af8672-a246-457e-9cfa-534c8d1d604d" path="/var/lib/kubelet/pods/88af8672-a246-457e-9cfa-534c8d1d604d/volumes"
Oct 30 13:39:56.097636 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:56.097590    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-b4fb25/ipp-route-b4fb25-kserve-5d46888457-25qdx"
Oct 30 13:39:56.110339 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:56.110314    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-b4fb25/ipp-route-b4fb25-kserve-5d46888457-25qdx"
Oct 30 13:39:56.489709 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:39:56.489678    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-b4fb25/ipp-route-b4fb25-kserve-5d46888457-25qdx"
Oct 30 13:40:05.391906 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:05.391874    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e3ebf5-76b5bd46b4-8ll5d"]
Oct 30 13:40:05.392705 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:05.392688    4364 cpu_manager.go:414] "containerMap: removing container" podUID="88af8672-a246-457e-9cfa-534c8d1d604d" containerName="payload-pre-processing"
Oct 30 13:40:05.392759 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:05.392705    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88af8672-a246-457e-9cfa-534c8d1d604d" containerName="payload-pre-processing"
Oct 30 13:40:05.392759 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:05.392716    4364 cpu_manager.go:414] "containerMap: removing container" podUID="bdd3c1f5-f9fd-400c-90f2-f9693008003a" containerName="istio-proxy"
Oct 30 13:40:05.392759 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:05.392724    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdd3c1f5-f9fd-400c-90f2-f9693008003a" containerName="istio-proxy"
Oct 30 13:40:05.392759 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:05.392735    4364 cpu_manager.go:414] "containerMap: removing container" podUID="bdd3c1f5-f9fd-400c-90f2-f9693008003a" containerName="istio-proxy"
Oct 30 13:40:05.392759 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:05.392740    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdd3c1f5-f9fd-400c-90f2-f9693008003a" containerName="istio-proxy"
Oct 30 13:40:05.392759 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:05.392752    4364 cpu_manager.go:414] "containerMap: removing container" podUID="094f69e3-b678-4281-9d9b-c0a4b25d0895" containerName="payload-processing"
Oct 30 13:40:05.392759 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:05.392757    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="094f69e3-b678-4281-9d9b-c0a4b25d0895" containerName="payload-processing"
Oct 30 13:40:05.392990 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:05.392763    4364 cpu_manager.go:414] "containerMap: removing container" podUID="4e1fd541-b715-4ba4-a160-bd86c2076d92" containerName="payload-processing"
Oct 30 13:40:05.392990 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:05.392768    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e1fd541-b715-4ba4-a160-bd86c2076d92" containerName="payload-processing"
Oct 30 13:40:05.392990 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:05.392891    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="88af8672-a246-457e-9cfa-534c8d1d604d" containerName="payload-pre-processing"
Oct 30 13:40:05.392990 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:05.392899    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdd3c1f5-f9fd-400c-90f2-f9693008003a" containerName="istio-proxy"
Oct 30 13:40:05.392990 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:05.392906    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdd3c1f5-f9fd-400c-90f2-f9693008003a" containerName="istio-proxy"
Oct 30 13:40:05.392990 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:05.392914    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="094f69e3-b678-4281-9d9b-c0a4b25d0895" containerName="payload-processing"
Oct 30 13:40:05.392990 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:05.392920    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e1fd541-b715-4ba4-a160-bd86c2076d92" containerName="payload-processing"
Oct 30 13:40:05.398382 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:05.398366    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e3ebf5-76b5bd46b4-8ll5d"
Oct 30 13:40:05.416015 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:05.415986    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e3ebf5-76b5bd46b4-8ll5d"]
Oct 30 13:40:05.423901 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:05.423871    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-e3ebf5-serving-cert\""
Oct 30 13:40:05.532524 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:05.532491    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b5b1e9d-9ea8-430d-9ff0-42b382a5f6a6-maas-api-tls\") pod \"maas-api-e2e-ipp-b-e3ebf5-76b5bd46b4-8ll5d\" (UID: \"0b5b1e9d-9ea8-430d-9ff0-42b382a5f6a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e3ebf5-76b5bd46b4-8ll5d"
Oct 30 13:40:05.532724 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:05.532687    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssq9z\" (UniqueName: \"kubernetes.io/projected/0b5b1e9d-9ea8-430d-9ff0-42b382a5f6a6-kube-api-access-ssq9z\") pod \"maas-api-e2e-ipp-b-e3ebf5-76b5bd46b4-8ll5d\" (UID: \"0b5b1e9d-9ea8-430d-9ff0-42b382a5f6a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e3ebf5-76b5bd46b4-8ll5d"
Oct 30 13:40:05.532795 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:05.532761    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b5b1e9d-9ea8-430d-9ff0-42b382a5f6a6-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-e3ebf5-76b5bd46b4-8ll5d\" (UID: \"0b5b1e9d-9ea8-430d-9ff0-42b382a5f6a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e3ebf5-76b5bd46b4-8ll5d"
Oct 30 13:40:05.532795 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:05.532785    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b5b1e9d-9ea8-430d-9ff0-42b382a5f6a6-metrics-certs\") pod \"maas-api-e2e-ipp-b-e3ebf5-76b5bd46b4-8ll5d\" (UID: \"0b5b1e9d-9ea8-430d-9ff0-42b382a5f6a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e3ebf5-76b5bd46b4-8ll5d"
Oct 30 13:40:05.640230 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:05.640167    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b5b1e9d-9ea8-430d-9ff0-42b382a5f6a6-maas-api-tls\") pod \"maas-api-e2e-ipp-b-e3ebf5-76b5bd46b4-8ll5d\" (UID: \"0b5b1e9d-9ea8-430d-9ff0-42b382a5f6a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e3ebf5-76b5bd46b4-8ll5d"
Oct 30 13:40:05.644800 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:05.644710    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssq9z\" (UniqueName: \"kubernetes.io/projected/0b5b1e9d-9ea8-430d-9ff0-42b382a5f6a6-kube-api-access-ssq9z\") pod \"maas-api-e2e-ipp-b-e3ebf5-76b5bd46b4-8ll5d\" (UID: \"0b5b1e9d-9ea8-430d-9ff0-42b382a5f6a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e3ebf5-76b5bd46b4-8ll5d"
Oct 30 13:40:05.645131 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:05.645104    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b5b1e9d-9ea8-430d-9ff0-42b382a5f6a6-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-e3ebf5-76b5bd46b4-8ll5d\" (UID: \"0b5b1e9d-9ea8-430d-9ff0-42b382a5f6a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e3ebf5-76b5bd46b4-8ll5d"
Oct 30 13:40:05.645297 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:05.645273    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b5b1e9d-9ea8-430d-9ff0-42b382a5f6a6-metrics-certs\") pod \"maas-api-e2e-ipp-b-e3ebf5-76b5bd46b4-8ll5d\" (UID: \"0b5b1e9d-9ea8-430d-9ff0-42b382a5f6a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e3ebf5-76b5bd46b4-8ll5d"
Oct 30 13:40:05.646305 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:40:05.646285    4364 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-e3ebf5-serving-cert: secret "maas-api-e2e-ipp-b-e3ebf5-serving-cert" not found
Oct 30 13:40:05.665818 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:40:05.665788    4364 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b5b1e9d-9ea8-430d-9ff0-42b382a5f6a6-maas-api-tls podName:0b5b1e9d-9ea8-430d-9ff0-42b382a5f6a6 nodeName:}" failed. No retries permitted until 2026-10-30 13:40:06.165759652 +0000 UTC m=+2905.332124285 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0b5b1e9d-9ea8-430d-9ff0-42b382a5f6a6-maas-api-tls") pod "maas-api-e2e-ipp-b-e3ebf5-76b5bd46b4-8ll5d" (UID: "0b5b1e9d-9ea8-430d-9ff0-42b382a5f6a6") : secret "maas-api-e2e-ipp-b-e3ebf5-serving-cert" not found
Oct 30 13:40:05.666397 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:05.666372    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b5b1e9d-9ea8-430d-9ff0-42b382a5f6a6-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-e3ebf5-76b5bd46b4-8ll5d\" (UID: \"0b5b1e9d-9ea8-430d-9ff0-42b382a5f6a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e3ebf5-76b5bd46b4-8ll5d"
Oct 30 13:40:05.676940 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:05.676915    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b5b1e9d-9ea8-430d-9ff0-42b382a5f6a6-metrics-certs\") pod \"maas-api-e2e-ipp-b-e3ebf5-76b5bd46b4-8ll5d\" (UID: \"0b5b1e9d-9ea8-430d-9ff0-42b382a5f6a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e3ebf5-76b5bd46b4-8ll5d"
Oct 30 13:40:05.678184 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:05.678158    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssq9z\" (UniqueName: \"kubernetes.io/projected/0b5b1e9d-9ea8-430d-9ff0-42b382a5f6a6-kube-api-access-ssq9z\") pod \"maas-api-e2e-ipp-b-e3ebf5-76b5bd46b4-8ll5d\" (UID: \"0b5b1e9d-9ea8-430d-9ff0-42b382a5f6a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e3ebf5-76b5bd46b4-8ll5d"
Oct 30 13:40:06.172679 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:06.171587    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b5b1e9d-9ea8-430d-9ff0-42b382a5f6a6-maas-api-tls\") pod \"maas-api-e2e-ipp-b-e3ebf5-76b5bd46b4-8ll5d\" (UID: \"0b5b1e9d-9ea8-430d-9ff0-42b382a5f6a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e3ebf5-76b5bd46b4-8ll5d"
Oct 30 13:40:06.202621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:06.202116    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b5b1e9d-9ea8-430d-9ff0-42b382a5f6a6-maas-api-tls\") pod \"maas-api-e2e-ipp-b-e3ebf5-76b5bd46b4-8ll5d\" (UID: \"0b5b1e9d-9ea8-430d-9ff0-42b382a5f6a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e3ebf5-76b5bd46b4-8ll5d"
Oct 30 13:40:06.313673 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:06.312879    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e3ebf5-76b5bd46b4-8ll5d"
Oct 30 13:40:06.848679 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:06.847980    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e3ebf5-76b5bd46b4-8ll5d"]
Oct 30 13:40:07.615266 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:07.615172    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e3ebf5-76b5bd46b4-8ll5d" event={"ID":"0b5b1e9d-9ea8-430d-9ff0-42b382a5f6a6","Type":"ContainerStarted","Data":"24d0b7697755d1209ab721fdca2f6a469fd54fa5d6a3ebe3dcb72d28187584df"}
Oct 30 13:40:07.615266 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:07.615245    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e3ebf5-76b5bd46b4-8ll5d" event={"ID":"0b5b1e9d-9ea8-430d-9ff0-42b382a5f6a6","Type":"ContainerStarted","Data":"e20a2756b8afa144f3a5c3263e72d9740c44eb3a9e08fc3ca09e052d1d3fc697"}
Oct 30 13:40:07.615471 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:07.615288    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e3ebf5-76b5bd46b4-8ll5d"
Oct 30 13:40:07.657639 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:07.657577    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e3ebf5-76b5bd46b4-8ll5d" podStartSLOduration=2.186637608 podStartE2EDuration="2.657564045s" totalImagesPullingTime="470.926437ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:40:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:40:07.656683035 +0000 UTC m=+2906.823047682" watchObservedRunningTime="2026-10-30 13:40:07.657564045 +0000 UTC m=+2906.823928705"
Oct 30 13:40:11.529615 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:40:11.529566    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:40:13.628850 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:13.628816    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e3ebf5-76b5bd46b4-8ll5d"
Oct 30 13:40:16.696113 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:16.696076    4364 generic.go:356] "Generic (PLEG): container finished" podID="0a9a3c04-3da8-4efe-8daf-b4821c563a24" containerID="70f543b1bcf791669112619fbd29585df08d5c8efeae09bb40a437fe62593169" exitCode=0
Oct 30 13:40:16.696550 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:16.696151    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s" event={"ID":"0a9a3c04-3da8-4efe-8daf-b4821c563a24","Type":"ContainerDied","Data":"70f543b1bcf791669112619fbd29585df08d5c8efeae09bb40a437fe62593169"}
Oct 30 13:40:16.696629 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:16.696574    4364 scope.go:122] "RemoveContainer" containerID="70f543b1bcf791669112619fbd29585df08d5c8efeae09bb40a437fe62593169"
Oct 30 13:40:16.698725 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:16.698691    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:40:16.698842 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:16.698760    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:40:16.698842 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:16.698796    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:40:17.713446 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:17.713411    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s" event={"ID":"0a9a3c04-3da8-4efe-8daf-b4821c563a24","Type":"ContainerStarted","Data":"961fa587390df58fd5669c5aaabc2151bc0a9d4f83dc505576583e277c9b4e17"}
Oct 30 13:40:18.475151 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:18.475116    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s"
Oct 30 13:40:18.475343 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:18.475295    4364 patch_prober.go:29] interesting pod/e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body=
Oct 30 13:40:18.475395 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:18.475351    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s" podUID="0a9a3c04-3da8-4efe-8daf-b4821c563a24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused"
Oct 30 13:40:19.474954 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:19.474919    4364 patch_prober.go:29] interesting pod/e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body=
Oct 30 13:40:19.475444 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:19.474981    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s" podUID="0a9a3c04-3da8-4efe-8daf-b4821c563a24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused"
Oct 30 13:40:20.474529 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:20.474491    4364 patch_prober.go:29] interesting pod/e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body=
Oct 30 13:40:20.474841 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:20.474549    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s" podUID="0a9a3c04-3da8-4efe-8daf-b4821c563a24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused"
Oct 30 13:40:21.475225 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:21.475183    4364 patch_prober.go:29] interesting pod/e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body=
Oct 30 13:40:21.475716 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:21.475253    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s" podUID="0a9a3c04-3da8-4efe-8daf-b4821c563a24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused"
Oct 30 13:40:22.474861 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:22.474824    4364 patch_prober.go:29] interesting pod/e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body=
Oct 30 13:40:22.475067 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:22.474906    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s" podUID="0a9a3c04-3da8-4efe-8daf-b4821c563a24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused"
Oct 30 13:40:23.508063 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:23.508029    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s"
Oct 30 13:40:23.508655 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:23.508633    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s"
Oct 30 13:40:23.510691 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:23.510668    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s"
Oct 30 13:40:25.030529 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:25.030490    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-30c004-d665cf8b-f56dn"]
Oct 30 13:40:25.061983 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:25.061947    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-30c004-84d79b8679-ndx6w"]
Oct 30 13:40:26.319636 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:26.319584    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8ee35e-6968f44cd7-7xvp8"]
Oct 30 13:40:26.324130 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:26.324108    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8ee35e-6968f44cd7-7xvp8"
Oct 30 13:40:26.344798 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:26.344769    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8ee35e-6968f44cd7-7xvp8"]
Oct 30 13:40:26.438291 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:26.438262    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-8ee35e-serving-cert\""
Oct 30 13:40:26.496504 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:26.496466    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-559wt\" (UniqueName: \"kubernetes.io/projected/d98dc22b-5142-4d00-bf71-27ead6f76d3e-kube-api-access-559wt\") pod \"maas-api-e2e-ipp-b-8ee35e-6968f44cd7-7xvp8\" (UID: \"d98dc22b-5142-4d00-bf71-27ead6f76d3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8ee35e-6968f44cd7-7xvp8"
Oct 30 13:40:26.496725 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:26.496576    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d98dc22b-5142-4d00-bf71-27ead6f76d3e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-8ee35e-6968f44cd7-7xvp8\" (UID: \"d98dc22b-5142-4d00-bf71-27ead6f76d3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8ee35e-6968f44cd7-7xvp8"
Oct 30 13:40:26.496818 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:26.496803    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d98dc22b-5142-4d00-bf71-27ead6f76d3e-metrics-certs\") pod \"maas-api-e2e-ipp-b-8ee35e-6968f44cd7-7xvp8\" (UID: \"d98dc22b-5142-4d00-bf71-27ead6f76d3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8ee35e-6968f44cd7-7xvp8"
Oct 30 13:40:26.496866 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:26.496854    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d98dc22b-5142-4d00-bf71-27ead6f76d3e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-8ee35e-6968f44cd7-7xvp8\" (UID: \"d98dc22b-5142-4d00-bf71-27ead6f76d3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8ee35e-6968f44cd7-7xvp8"
Oct 30 13:40:26.602674 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:26.599410    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-559wt\" (UniqueName: \"kubernetes.io/projected/d98dc22b-5142-4d00-bf71-27ead6f76d3e-kube-api-access-559wt\") pod \"maas-api-e2e-ipp-b-8ee35e-6968f44cd7-7xvp8\" (UID: \"d98dc22b-5142-4d00-bf71-27ead6f76d3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8ee35e-6968f44cd7-7xvp8"
Oct 30 13:40:26.602674 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:26.599977    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d98dc22b-5142-4d00-bf71-27ead6f76d3e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-8ee35e-6968f44cd7-7xvp8\" (UID: \"d98dc22b-5142-4d00-bf71-27ead6f76d3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8ee35e-6968f44cd7-7xvp8"
Oct 30 13:40:26.602674 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:26.600737    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d98dc22b-5142-4d00-bf71-27ead6f76d3e-metrics-certs\") pod \"maas-api-e2e-ipp-b-8ee35e-6968f44cd7-7xvp8\" (UID: \"d98dc22b-5142-4d00-bf71-27ead6f76d3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8ee35e-6968f44cd7-7xvp8"
Oct 30 13:40:26.610618 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:26.608261    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d98dc22b-5142-4d00-bf71-27ead6f76d3e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-8ee35e-6968f44cd7-7xvp8\" (UID: \"d98dc22b-5142-4d00-bf71-27ead6f76d3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8ee35e-6968f44cd7-7xvp8"
Oct 30 13:40:26.610618 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:26.609337    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d98dc22b-5142-4d00-bf71-27ead6f76d3e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-8ee35e-6968f44cd7-7xvp8\" (UID: \"d98dc22b-5142-4d00-bf71-27ead6f76d3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8ee35e-6968f44cd7-7xvp8"
Oct 30 13:40:26.635724 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:26.635689    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d98dc22b-5142-4d00-bf71-27ead6f76d3e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-8ee35e-6968f44cd7-7xvp8\" (UID: \"d98dc22b-5142-4d00-bf71-27ead6f76d3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8ee35e-6968f44cd7-7xvp8"
Oct 30 13:40:26.649717 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:26.640222    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d98dc22b-5142-4d00-bf71-27ead6f76d3e-metrics-certs\") pod \"maas-api-e2e-ipp-b-8ee35e-6968f44cd7-7xvp8\" (UID: \"d98dc22b-5142-4d00-bf71-27ead6f76d3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8ee35e-6968f44cd7-7xvp8"
Oct 30 13:40:26.659801 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:26.659387    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-559wt\" (UniqueName: \"kubernetes.io/projected/d98dc22b-5142-4d00-bf71-27ead6f76d3e-kube-api-access-559wt\") pod \"maas-api-e2e-ipp-b-8ee35e-6968f44cd7-7xvp8\" (UID: \"d98dc22b-5142-4d00-bf71-27ead6f76d3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8ee35e-6968f44cd7-7xvp8"
Oct 30 13:40:26.938090 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:26.938059    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8ee35e-6968f44cd7-7xvp8"
Oct 30 13:40:27.247427 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:27.247401    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8ee35e-6968f44cd7-7xvp8"]
Oct 30 13:40:27.248963 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:40:27.248933    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd98dc22b_5142_4d00_bf71_27ead6f76d3e.slice/crio-50062c71cbac4238139907a2908756acf8fceffc142d353b60f103959d397bdb WatchSource:0}: Error finding container 50062c71cbac4238139907a2908756acf8fceffc142d353b60f103959d397bdb: Status 404 returned error can't find the container with id 50062c71cbac4238139907a2908756acf8fceffc142d353b60f103959d397bdb
Oct 30 13:40:27.894665 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:27.892432    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8ee35e-6968f44cd7-7xvp8" event={"ID":"d98dc22b-5142-4d00-bf71-27ead6f76d3e","Type":"ContainerStarted","Data":"b84de1bf2aa1b743bd937b3b1ae6fa988ca585c67ee55b21dd6e1181cd02db99"}
Oct 30 13:40:27.894665 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:27.892504    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8ee35e-6968f44cd7-7xvp8" event={"ID":"d98dc22b-5142-4d00-bf71-27ead6f76d3e","Type":"ContainerStarted","Data":"50062c71cbac4238139907a2908756acf8fceffc142d353b60f103959d397bdb"}
Oct 30 13:40:27.894665 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:27.893682    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8ee35e-6968f44cd7-7xvp8"
Oct 30 13:40:27.963614 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:27.963536    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8ee35e-6968f44cd7-7xvp8" podStartSLOduration=1.679859865 podStartE2EDuration="1.963517032s" totalImagesPullingTime="283.657167ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:40:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:40:27.947100921 +0000 UTC m=+2927.113465570" watchObservedRunningTime="2026-10-30 13:40:27.963517032 +0000 UTC m=+2927.129881681"
Oct 30 13:40:30.031061 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:30.031022    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-30c004-d665cf8b-f56dn" podUID="e262eff8-700f-4aea-b6c4-1c76a9252726" containerName="payload-processing" containerID="cri-o://a91c0cd9d98a8154c1e0f1b8216c8dc465527ccedf6fd22bad5121c08cc07d47" gracePeriod=25
Oct 30 13:40:30.062404 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:30.062369    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30c004-84d79b8679-ndx6w" podUID="32995da4-65f2-4605-95e9-c2cf569f5743" containerName="payload-pre-processing" containerID="cri-o://f5875af59333dca9e4f3988a0035f3a533a395f3a6493fe1ea7d31aa83a90393" gracePeriod=25
Oct 30 13:40:30.353412 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:30.353383    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-30c004-d665cf8b-f56dn"
Oct 30 13:40:30.391671 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:30.391625    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e262eff8-700f-4aea-b6c4-1c76a9252726-kube-api-access-4wj8t\" (UniqueName: \"kubernetes.io/projected/e262eff8-700f-4aea-b6c4-1c76a9252726-kube-api-access-4wj8t\") pod \"e262eff8-700f-4aea-b6c4-1c76a9252726\" (UID: \"e262eff8-700f-4aea-b6c4-1c76a9252726\") "
Oct 30 13:40:30.391861 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:30.391780    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e262eff8-700f-4aea-b6c4-1c76a9252726-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e262eff8-700f-4aea-b6c4-1c76a9252726-plugins-config-volume\") pod \"e262eff8-700f-4aea-b6c4-1c76a9252726\" (UID: \"e262eff8-700f-4aea-b6c4-1c76a9252726\") "
Oct 30 13:40:30.392104 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:30.392079    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e262eff8-700f-4aea-b6c4-1c76a9252726-plugins-config-volume" pod "e262eff8-700f-4aea-b6c4-1c76a9252726" (UID: "e262eff8-700f-4aea-b6c4-1c76a9252726"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:40:30.392761 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:30.392738    4364 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e262eff8-700f-4aea-b6c4-1c76a9252726-plugins-config-volume\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:40:30.393847 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:30.393824    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e262eff8-700f-4aea-b6c4-1c76a9252726-kube-api-access-4wj8t" pod "e262eff8-700f-4aea-b6c4-1c76a9252726" (UID: "e262eff8-700f-4aea-b6c4-1c76a9252726"). InnerVolumeSpecName "kube-api-access-4wj8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:30.426986 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:30.426949    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30c004-84d79b8679-ndx6w"
Oct 30 13:40:30.493534 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:30.493489    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32995da4-65f2-4605-95e9-c2cf569f5743-kube-api-access-8qk5f\" (UniqueName: \"kubernetes.io/projected/32995da4-65f2-4605-95e9-c2cf569f5743-kube-api-access-8qk5f\") pod \"32995da4-65f2-4605-95e9-c2cf569f5743\" (UID: \"32995da4-65f2-4605-95e9-c2cf569f5743\") "
Oct 30 13:40:30.493856 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:30.493837    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32995da4-65f2-4605-95e9-c2cf569f5743-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32995da4-65f2-4605-95e9-c2cf569f5743-plugins-config-volume\") pod \"32995da4-65f2-4605-95e9-c2cf569f5743\" (UID: \"32995da4-65f2-4605-95e9-c2cf569f5743\") "
Oct 30 13:40:30.494336 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:30.494308    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32995da4-65f2-4605-95e9-c2cf569f5743-plugins-config-volume" pod "32995da4-65f2-4605-95e9-c2cf569f5743" (UID: "32995da4-65f2-4605-95e9-c2cf569f5743"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:40:30.494548 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:30.494529    4364 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32995da4-65f2-4605-95e9-c2cf569f5743-plugins-config-volume\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:40:30.494646 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:30.494552    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wj8t\" (UniqueName: \"kubernetes.io/projected/e262eff8-700f-4aea-b6c4-1c76a9252726-kube-api-access-4wj8t\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:40:30.496349 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:30.496317    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32995da4-65f2-4605-95e9-c2cf569f5743-kube-api-access-8qk5f" pod "32995da4-65f2-4605-95e9-c2cf569f5743" (UID: "32995da4-65f2-4605-95e9-c2cf569f5743"). InnerVolumeSpecName "kube-api-access-8qk5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:30.597001 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:30.596916    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qk5f\" (UniqueName: \"kubernetes.io/projected/32995da4-65f2-4605-95e9-c2cf569f5743-kube-api-access-8qk5f\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:40:30.919495 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:30.919455    4364 generic.go:356] "Generic (PLEG): container finished" podID="e262eff8-700f-4aea-b6c4-1c76a9252726" containerID="a91c0cd9d98a8154c1e0f1b8216c8dc465527ccedf6fd22bad5121c08cc07d47" exitCode=0
Oct 30 13:40:30.919495 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:30.919497    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-30c004-d665cf8b-f56dn"
Oct 30 13:40:30.919877 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:30.919535    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-30c004-d665cf8b-f56dn" event={"ID":"e262eff8-700f-4aea-b6c4-1c76a9252726","Type":"ContainerDied","Data":"a91c0cd9d98a8154c1e0f1b8216c8dc465527ccedf6fd22bad5121c08cc07d47"}
Oct 30 13:40:30.919877 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:30.919593    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-30c004-d665cf8b-f56dn" event={"ID":"e262eff8-700f-4aea-b6c4-1c76a9252726","Type":"ContainerDied","Data":"44cbaaeb3c552ca8cad0647e13fe11c4388e5db71832dffe1aa67ac6a3feb499"}
Oct 30 13:40:30.919877 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:30.919642    4364 scope.go:122] "RemoveContainer" containerID="a91c0cd9d98a8154c1e0f1b8216c8dc465527ccedf6fd22bad5121c08cc07d47"
Oct 30 13:40:30.921277 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:30.921250    4364 generic.go:356] "Generic (PLEG): container finished" podID="32995da4-65f2-4605-95e9-c2cf569f5743" containerID="f5875af59333dca9e4f3988a0035f3a533a395f3a6493fe1ea7d31aa83a90393" exitCode=0
Oct 30 13:40:30.921395 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:30.921307    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30c004-84d79b8679-ndx6w"
Oct 30 13:40:30.921395 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:30.921342    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30c004-84d79b8679-ndx6w" event={"ID":"32995da4-65f2-4605-95e9-c2cf569f5743","Type":"ContainerDied","Data":"f5875af59333dca9e4f3988a0035f3a533a395f3a6493fe1ea7d31aa83a90393"}
Oct 30 13:40:30.921482 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:30.921397    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30c004-84d79b8679-ndx6w" event={"ID":"32995da4-65f2-4605-95e9-c2cf569f5743","Type":"ContainerDied","Data":"15edc26f227b3fcd61a660e33fc7900031227b70b3c940c2e8e07e186ef37630"}
Oct 30 13:40:30.932584 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:30.932567    4364 scope.go:122] "RemoveContainer" containerID="a91c0cd9d98a8154c1e0f1b8216c8dc465527ccedf6fd22bad5121c08cc07d47"
Oct 30 13:40:30.932880 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:40:30.932856    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a91c0cd9d98a8154c1e0f1b8216c8dc465527ccedf6fd22bad5121c08cc07d47\": container with ID starting with a91c0cd9d98a8154c1e0f1b8216c8dc465527ccedf6fd22bad5121c08cc07d47 not found: ID does not exist" containerID="a91c0cd9d98a8154c1e0f1b8216c8dc465527ccedf6fd22bad5121c08cc07d47"
Oct 30 13:40:30.932977 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:30.932891    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a91c0cd9d98a8154c1e0f1b8216c8dc465527ccedf6fd22bad5121c08cc07d47"} err="failed to get container status \"a91c0cd9d98a8154c1e0f1b8216c8dc465527ccedf6fd22bad5121c08cc07d47\": rpc error: code = NotFound desc = could not find container \"a91c0cd9d98a8154c1e0f1b8216c8dc465527ccedf6fd22bad5121c08cc07d47\": container with ID starting with a91c0cd9d98a8154c1e0f1b8216c8dc465527ccedf6fd22bad5121c08cc07d47 not found: ID does not exist"
Oct 30 13:40:30.932977 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:30.932909    4364 scope.go:122] "RemoveContainer" containerID="f5875af59333dca9e4f3988a0035f3a533a395f3a6493fe1ea7d31aa83a90393"
Oct 30 13:40:30.944174 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:30.944157    4364 scope.go:122] "RemoveContainer" containerID="f5875af59333dca9e4f3988a0035f3a533a395f3a6493fe1ea7d31aa83a90393"
Oct 30 13:40:30.944460 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:40:30.944438    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5875af59333dca9e4f3988a0035f3a533a395f3a6493fe1ea7d31aa83a90393\": container with ID starting with f5875af59333dca9e4f3988a0035f3a533a395f3a6493fe1ea7d31aa83a90393 not found: ID does not exist" containerID="f5875af59333dca9e4f3988a0035f3a533a395f3a6493fe1ea7d31aa83a90393"
Oct 30 13:40:30.944534 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:30.944469    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5875af59333dca9e4f3988a0035f3a533a395f3a6493fe1ea7d31aa83a90393"} err="failed to get container status \"f5875af59333dca9e4f3988a0035f3a533a395f3a6493fe1ea7d31aa83a90393\": rpc error: code = NotFound desc = could not find container \"f5875af59333dca9e4f3988a0035f3a533a395f3a6493fe1ea7d31aa83a90393\": container with ID starting with f5875af59333dca9e4f3988a0035f3a533a395f3a6493fe1ea7d31aa83a90393 not found: ID does not exist"
Oct 30 13:40:30.975954 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:30.975932    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-30c004-d665cf8b-f56dn"]
Oct 30 13:40:30.980392 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:30.980368    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-30c004-d665cf8b-f56dn"]
Oct 30 13:40:31.019886 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:31.019856    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-30c004-84d79b8679-ndx6w"]
Oct 30 13:40:31.030701 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:31.030671    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-30c004-84d79b8679-ndx6w"]
Oct 30 13:40:32.972278 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:32.972245    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32995da4-65f2-4605-95e9-c2cf569f5743" path="/var/lib/kubelet/pods/32995da4-65f2-4605-95e9-c2cf569f5743/volumes"
Oct 30 13:40:32.972857 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:32.972837    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e262eff8-700f-4aea-b6c4-1c76a9252726" path="/var/lib/kubelet/pods/e262eff8-700f-4aea-b6c4-1c76a9252726/volumes"
Oct 30 13:40:34.914684 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:34.914656    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8ee35e-6968f44cd7-7xvp8"
Oct 30 13:40:41.556551 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:40:41.556509    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:40:44.238794 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:44.238757    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:40:45.387809 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:45.387770    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf"]
Oct 30 13:40:45.388272 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:45.388071    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf" podUID="b6d93861-6630-4e50-95b8-d34b8e83e92e" containerName="istio-proxy" containerID="cri-o://474c7970fef0a0f1e8e43f9e5a6ca011a12260ceffa91f1b32b8d27648a6b363" gracePeriod=30
Oct 30 13:40:50.729214 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:50.729187    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf"
Oct 30 13:40:50.832809 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:50.832776    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b6d93861-6630-4e50-95b8-d34b8e83e92e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6d93861-6630-4e50-95b8-d34b8e83e92e-istio-podinfo\") pod \"b6d93861-6630-4e50-95b8-d34b8e83e92e\" (UID: \"b6d93861-6630-4e50-95b8-d34b8e83e92e\") "
Oct 30 13:40:50.833006 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:50.832946    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6d93861-6630-4e50-95b8-d34b8e83e92e-kube-api-access-7cvz5\" (UniqueName: \"kubernetes.io/projected/b6d93861-6630-4e50-95b8-d34b8e83e92e-kube-api-access-7cvz5\") pod \"b6d93861-6630-4e50-95b8-d34b8e83e92e\" (UID: \"b6d93861-6630-4e50-95b8-d34b8e83e92e\") "
Oct 30 13:40:50.833085 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:50.833069    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6d93861-6630-4e50-95b8-d34b8e83e92e-istio-token\" (UniqueName: \"kubernetes.io/projected/b6d93861-6630-4e50-95b8-d34b8e83e92e-istio-token\") pod \"b6d93861-6630-4e50-95b8-d34b8e83e92e\" (UID: \"b6d93861-6630-4e50-95b8-d34b8e83e92e\") "
Oct 30 13:40:50.834157 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:50.833174    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6d93861-6630-4e50-95b8-d34b8e83e92e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6d93861-6630-4e50-95b8-d34b8e83e92e-credential-socket\") pod \"b6d93861-6630-4e50-95b8-d34b8e83e92e\" (UID: \"b6d93861-6630-4e50-95b8-d34b8e83e92e\") "
Oct 30 13:40:50.834157 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:50.833228    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6d93861-6630-4e50-95b8-d34b8e83e92e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6d93861-6630-4e50-95b8-d34b8e83e92e-istio-envoy\") pod \"b6d93861-6630-4e50-95b8-d34b8e83e92e\" (UID: \"b6d93861-6630-4e50-95b8-d34b8e83e92e\") "
Oct 30 13:40:50.834157 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:50.833254    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6d93861-6630-4e50-95b8-d34b8e83e92e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6d93861-6630-4e50-95b8-d34b8e83e92e-workload-certs\") pod \"b6d93861-6630-4e50-95b8-d34b8e83e92e\" (UID: \"b6d93861-6630-4e50-95b8-d34b8e83e92e\") "
Oct 30 13:40:50.834157 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:50.833290    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6d93861-6630-4e50-95b8-d34b8e83e92e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6d93861-6630-4e50-95b8-d34b8e83e92e-workload-socket\") pod \"b6d93861-6630-4e50-95b8-d34b8e83e92e\" (UID: \"b6d93861-6630-4e50-95b8-d34b8e83e92e\") "
Oct 30 13:40:50.834157 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:50.833343    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6d93861-6630-4e50-95b8-d34b8e83e92e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6d93861-6630-4e50-95b8-d34b8e83e92e-istiod-ca-cert\") pod \"b6d93861-6630-4e50-95b8-d34b8e83e92e\" (UID: \"b6d93861-6630-4e50-95b8-d34b8e83e92e\") "
Oct 30 13:40:50.834157 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:50.833389    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6d93861-6630-4e50-95b8-d34b8e83e92e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6d93861-6630-4e50-95b8-d34b8e83e92e-istio-data\") pod \"b6d93861-6630-4e50-95b8-d34b8e83e92e\" (UID: \"b6d93861-6630-4e50-95b8-d34b8e83e92e\") "
Oct 30 13:40:50.834157 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:50.833432    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6d93861-6630-4e50-95b8-d34b8e83e92e-credential-socket" pod "b6d93861-6630-4e50-95b8-d34b8e83e92e" (UID: "b6d93861-6630-4e50-95b8-d34b8e83e92e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:50.834157 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:50.833773    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6d93861-6630-4e50-95b8-d34b8e83e92e-istiod-ca-cert" pod "b6d93861-6630-4e50-95b8-d34b8e83e92e" (UID: "b6d93861-6630-4e50-95b8-d34b8e83e92e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:40:50.834157 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:50.833926    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6d93861-6630-4e50-95b8-d34b8e83e92e-workload-certs" pod "b6d93861-6630-4e50-95b8-d34b8e83e92e" (UID: "b6d93861-6630-4e50-95b8-d34b8e83e92e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:50.834157 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:50.833983    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6d93861-6630-4e50-95b8-d34b8e83e92e-workload-socket" pod "b6d93861-6630-4e50-95b8-d34b8e83e92e" (UID: "b6d93861-6630-4e50-95b8-d34b8e83e92e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:50.834157 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:50.834003    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6d93861-6630-4e50-95b8-d34b8e83e92e-istio-data" pod "b6d93861-6630-4e50-95b8-d34b8e83e92e" (UID: "b6d93861-6630-4e50-95b8-d34b8e83e92e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:50.834863 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:50.834274    4364 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6d93861-6630-4e50-95b8-d34b8e83e92e-credential-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:40:50.834863 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:50.834291    4364 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6d93861-6630-4e50-95b8-d34b8e83e92e-workload-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:40:50.834863 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:50.834301    4364 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6d93861-6630-4e50-95b8-d34b8e83e92e-workload-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:40:50.834863 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:50.834309    4364 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6d93861-6630-4e50-95b8-d34b8e83e92e-istiod-ca-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:40:50.834863 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:50.834316    4364 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6d93861-6630-4e50-95b8-d34b8e83e92e-istio-data\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:40:50.835433 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:50.835401    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b6d93861-6630-4e50-95b8-d34b8e83e92e-istio-podinfo" pod "b6d93861-6630-4e50-95b8-d34b8e83e92e" (UID: "b6d93861-6630-4e50-95b8-d34b8e83e92e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:40:50.835860 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:50.835832    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6d93861-6630-4e50-95b8-d34b8e83e92e-kube-api-access-7cvz5" pod "b6d93861-6630-4e50-95b8-d34b8e83e92e" (UID: "b6d93861-6630-4e50-95b8-d34b8e83e92e"). InnerVolumeSpecName "kube-api-access-7cvz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:50.836581 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:50.836557    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6d93861-6630-4e50-95b8-d34b8e83e92e-istio-token" pod "b6d93861-6630-4e50-95b8-d34b8e83e92e" (UID: "b6d93861-6630-4e50-95b8-d34b8e83e92e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:50.836938 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:50.836918    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6d93861-6630-4e50-95b8-d34b8e83e92e-istio-envoy" pod "b6d93861-6630-4e50-95b8-d34b8e83e92e" (UID: "b6d93861-6630-4e50-95b8-d34b8e83e92e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:50.935989 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:50.935958    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cvz5\" (UniqueName: \"kubernetes.io/projected/b6d93861-6630-4e50-95b8-d34b8e83e92e-kube-api-access-7cvz5\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:40:50.935989 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:50.935987    4364 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6d93861-6630-4e50-95b8-d34b8e83e92e-istio-token\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:40:50.935989 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:50.935998    4364 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6d93861-6630-4e50-95b8-d34b8e83e92e-istio-envoy\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:40:50.936240 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:50.936006    4364 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6d93861-6630-4e50-95b8-d34b8e83e92e-istio-podinfo\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:40:51.294447 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:51.294354    4364 generic.go:356] "Generic (PLEG): container finished" podID="b6d93861-6630-4e50-95b8-d34b8e83e92e" containerID="474c7970fef0a0f1e8e43f9e5a6ca011a12260ceffa91f1b32b8d27648a6b363" exitCode=0
Oct 30 13:40:51.294447 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:51.294398    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf"
Oct 30 13:40:51.294447 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:51.294435    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf" event={"ID":"b6d93861-6630-4e50-95b8-d34b8e83e92e","Type":"ContainerDied","Data":"474c7970fef0a0f1e8e43f9e5a6ca011a12260ceffa91f1b32b8d27648a6b363"}
Oct 30 13:40:51.294751 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:51.294494    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf" event={"ID":"b6d93861-6630-4e50-95b8-d34b8e83e92e","Type":"ContainerDied","Data":"7ee3eb1a4c9a5739de21b0c2b5a12614817ac8a52324110f8e829ab72a9c5ce6"}
Oct 30 13:40:51.294751 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:51.294524    4364 scope.go:122] "RemoveContainer" containerID="474c7970fef0a0f1e8e43f9e5a6ca011a12260ceffa91f1b32b8d27648a6b363"
Oct 30 13:40:51.305912 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:51.305894    4364 scope.go:122] "RemoveContainer" containerID="474c7970fef0a0f1e8e43f9e5a6ca011a12260ceffa91f1b32b8d27648a6b363"
Oct 30 13:40:51.306202 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:40:51.306183    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"474c7970fef0a0f1e8e43f9e5a6ca011a12260ceffa91f1b32b8d27648a6b363\": container with ID starting with 474c7970fef0a0f1e8e43f9e5a6ca011a12260ceffa91f1b32b8d27648a6b363 not found: ID does not exist" containerID="474c7970fef0a0f1e8e43f9e5a6ca011a12260ceffa91f1b32b8d27648a6b363"
Oct 30 13:40:51.306278 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:51.306210    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"474c7970fef0a0f1e8e43f9e5a6ca011a12260ceffa91f1b32b8d27648a6b363"} err="failed to get container status \"474c7970fef0a0f1e8e43f9e5a6ca011a12260ceffa91f1b32b8d27648a6b363\": rpc error: code = NotFound desc = could not find container \"474c7970fef0a0f1e8e43f9e5a6ca011a12260ceffa91f1b32b8d27648a6b363\": container with ID starting with 474c7970fef0a0f1e8e43f9e5a6ca011a12260ceffa91f1b32b8d27648a6b363 not found: ID does not exist"
Oct 30 13:40:51.360552 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:51.360522    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf"]
Oct 30 13:40:51.371850 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:51.371825    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-ddkkf"]
Oct 30 13:40:52.472078 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:52.472032    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp"]
Oct 30 13:40:52.472590 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:52.472357    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp" podUID="09bcc5ff-a33b-4699-a797-58b18a5f63bd" containerName="istio-proxy" containerID="cri-o://5e78e34aa30d512d552790b9f84b026d6c4a5dca1b4ea04c843cefa29daad33e" gracePeriod=30
Oct 30 13:40:52.971864 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:52.971829    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6d93861-6630-4e50-95b8-d34b8e83e92e" path="/var/lib/kubelet/pods/b6d93861-6630-4e50-95b8-d34b8e83e92e/volumes"
Oct 30 13:40:57.653674 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.653629    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj"]
Oct 30 13:40:57.654950 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.654927    4364 cpu_manager.go:414] "containerMap: removing container" podUID="e262eff8-700f-4aea-b6c4-1c76a9252726" containerName="payload-processing"
Oct 30 13:40:57.655019 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.654953    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e262eff8-700f-4aea-b6c4-1c76a9252726" containerName="payload-processing"
Oct 30 13:40:57.655019 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.655001    4364 cpu_manager.go:414] "containerMap: removing container" podUID="b6d93861-6630-4e50-95b8-d34b8e83e92e" containerName="istio-proxy"
Oct 30 13:40:57.655019 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.655008    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6d93861-6630-4e50-95b8-d34b8e83e92e" containerName="istio-proxy"
Oct 30 13:40:57.655126 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.655043    4364 cpu_manager.go:414] "containerMap: removing container" podUID="32995da4-65f2-4605-95e9-c2cf569f5743" containerName="payload-pre-processing"
Oct 30 13:40:57.655126 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.655051    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32995da4-65f2-4605-95e9-c2cf569f5743" containerName="payload-pre-processing"
Oct 30 13:40:57.655224 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.655211    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="e262eff8-700f-4aea-b6c4-1c76a9252726" containerName="payload-processing"
Oct 30 13:40:57.655262 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.655236    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6d93861-6630-4e50-95b8-d34b8e83e92e" containerName="istio-proxy"
Oct 30 13:40:57.655262 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.655249    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="32995da4-65f2-4605-95e9-c2cf569f5743" containerName="payload-pre-processing"
Oct 30 13:40:57.659210 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.659185    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj"
Oct 30 13:40:57.674679 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.674642    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq"]
Oct 30 13:40:57.682762 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.682297    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq"
Oct 30 13:40:57.688992 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.688966    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-8ee35e-openshift-default-dockercfg-27hz2\""
Oct 30 13:40:57.697575 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.697535    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj"]
Oct 30 13:40:57.718543 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.718513    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq"]
Oct 30 13:40:57.752098 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.752017    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09c95c32-f60a-4f60-90ff-90f447247cce-istio-data\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq\" (UID: \"09c95c32-f60a-4f60-90ff-90f447247cce\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq"
Oct 30 13:40:57.752263 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.752119    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23116724-6a57-401b-8ba4-a105a71a5e6d-workload-certs\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj\" (UID: \"23116724-6a57-401b-8ba4-a105a71a5e6d\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj"
Oct 30 13:40:57.752263 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.752164    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c95c32-f60a-4f60-90ff-90f447247cce-credential-socket\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq\" (UID: \"09c95c32-f60a-4f60-90ff-90f447247cce\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq"
Oct 30 13:40:57.752263 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.752200    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23116724-6a57-401b-8ba4-a105a71a5e6d-istio-podinfo\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj\" (UID: \"23116724-6a57-401b-8ba4-a105a71a5e6d\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj"
Oct 30 13:40:57.752263 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.752243    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23116724-6a57-401b-8ba4-a105a71a5e6d-workload-socket\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj\" (UID: \"23116724-6a57-401b-8ba4-a105a71a5e6d\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj"
Oct 30 13:40:57.752510 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.752281    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23116724-6a57-401b-8ba4-a105a71a5e6d-istio-envoy\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj\" (UID: \"23116724-6a57-401b-8ba4-a105a71a5e6d\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj"
Oct 30 13:40:57.752510 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.752314    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09c95c32-f60a-4f60-90ff-90f447247cce-istio-podinfo\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq\" (UID: \"09c95c32-f60a-4f60-90ff-90f447247cce\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq"
Oct 30 13:40:57.752510 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.752496    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srhwh\" (UniqueName: \"kubernetes.io/projected/23116724-6a57-401b-8ba4-a105a71a5e6d-kube-api-access-srhwh\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj\" (UID: \"23116724-6a57-401b-8ba4-a105a71a5e6d\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj"
Oct 30 13:40:57.752697 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.752561    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwc5b\" (UniqueName: \"kubernetes.io/projected/09c95c32-f60a-4f60-90ff-90f447247cce-kube-api-access-pwc5b\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq\" (UID: \"09c95c32-f60a-4f60-90ff-90f447247cce\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq"
Oct 30 13:40:57.752697 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.752683    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09c95c32-f60a-4f60-90ff-90f447247cce-workload-certs\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq\" (UID: \"09c95c32-f60a-4f60-90ff-90f447247cce\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq"
Oct 30 13:40:57.752815 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.752712    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23116724-6a57-401b-8ba4-a105a71a5e6d-credential-socket\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj\" (UID: \"23116724-6a57-401b-8ba4-a105a71a5e6d\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj"
Oct 30 13:40:57.752815 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.752756    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09c95c32-f60a-4f60-90ff-90f447247cce-istio-envoy\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq\" (UID: \"09c95c32-f60a-4f60-90ff-90f447247cce\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq"
Oct 30 13:40:57.752815 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.752800    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23116724-6a57-401b-8ba4-a105a71a5e6d-istio-data\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj\" (UID: \"23116724-6a57-401b-8ba4-a105a71a5e6d\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj"
Oct 30 13:40:57.752978 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.752851    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23116724-6a57-401b-8ba4-a105a71a5e6d-istio-token\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj\" (UID: \"23116724-6a57-401b-8ba4-a105a71a5e6d\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj"
Oct 30 13:40:57.752978 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.752885    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c95c32-f60a-4f60-90ff-90f447247cce-workload-socket\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq\" (UID: \"09c95c32-f60a-4f60-90ff-90f447247cce\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq"
Oct 30 13:40:57.753115 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.753098    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23116724-6a57-401b-8ba4-a105a71a5e6d-istiod-ca-cert\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj\" (UID: \"23116724-6a57-401b-8ba4-a105a71a5e6d\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj"
Oct 30 13:40:57.753207 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.753192    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09c95c32-f60a-4f60-90ff-90f447247cce-istio-token\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq\" (UID: \"09c95c32-f60a-4f60-90ff-90f447247cce\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq"
Oct 30 13:40:57.753267 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.753241    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09c95c32-f60a-4f60-90ff-90f447247cce-istiod-ca-cert\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq\" (UID: \"09c95c32-f60a-4f60-90ff-90f447247cce\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq"
Oct 30 13:40:57.813459 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.813433    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp"
Oct 30 13:40:57.854613 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.854570    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23116724-6a57-401b-8ba4-a105a71a5e6d-istio-token\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj\" (UID: \"23116724-6a57-401b-8ba4-a105a71a5e6d\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj"
Oct 30 13:40:57.854781 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.854637    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c95c32-f60a-4f60-90ff-90f447247cce-workload-socket\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq\" (UID: \"09c95c32-f60a-4f60-90ff-90f447247cce\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq"
Oct 30 13:40:57.854781 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.854726    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23116724-6a57-401b-8ba4-a105a71a5e6d-istiod-ca-cert\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj\" (UID: \"23116724-6a57-401b-8ba4-a105a71a5e6d\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj"
Oct 30 13:40:57.854910 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.854830    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09c95c32-f60a-4f60-90ff-90f447247cce-istio-token\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq\" (UID: \"09c95c32-f60a-4f60-90ff-90f447247cce\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq"
Oct 30 13:40:57.854910 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.854880    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09c95c32-f60a-4f60-90ff-90f447247cce-istiod-ca-cert\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq\" (UID: \"09c95c32-f60a-4f60-90ff-90f447247cce\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq"
Oct 30 13:40:57.855055 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.854996    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09c95c32-f60a-4f60-90ff-90f447247cce-istio-data\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq\" (UID: \"09c95c32-f60a-4f60-90ff-90f447247cce\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq"
Oct 30 13:40:57.855125 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.855065    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23116724-6a57-401b-8ba4-a105a71a5e6d-workload-certs\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj\" (UID: \"23116724-6a57-401b-8ba4-a105a71a5e6d\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj"
Oct 30 13:40:57.855125 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.855108    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c95c32-f60a-4f60-90ff-90f447247cce-credential-socket\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq\" (UID: \"09c95c32-f60a-4f60-90ff-90f447247cce\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq"
Oct 30 13:40:57.855236 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.855141    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23116724-6a57-401b-8ba4-a105a71a5e6d-istio-podinfo\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj\" (UID: \"23116724-6a57-401b-8ba4-a105a71a5e6d\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj"
Oct 30 13:40:57.855236 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.855186    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23116724-6a57-401b-8ba4-a105a71a5e6d-workload-socket\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj\" (UID: \"23116724-6a57-401b-8ba4-a105a71a5e6d\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj"
Oct 30 13:40:57.855236 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.855227    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23116724-6a57-401b-8ba4-a105a71a5e6d-istio-envoy\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj\" (UID: \"23116724-6a57-401b-8ba4-a105a71a5e6d\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj"
Oct 30 13:40:57.855403 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.855250    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09c95c32-f60a-4f60-90ff-90f447247cce-istio-podinfo\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq\" (UID: \"09c95c32-f60a-4f60-90ff-90f447247cce\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq"
Oct 30 13:40:57.855403 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.855289    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srhwh\" (UniqueName: \"kubernetes.io/projected/23116724-6a57-401b-8ba4-a105a71a5e6d-kube-api-access-srhwh\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj\" (UID: \"23116724-6a57-401b-8ba4-a105a71a5e6d\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj"
Oct 30 13:40:57.855403 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.855337    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwc5b\" (UniqueName: \"kubernetes.io/projected/09c95c32-f60a-4f60-90ff-90f447247cce-kube-api-access-pwc5b\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq\" (UID: \"09c95c32-f60a-4f60-90ff-90f447247cce\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq"
Oct 30 13:40:57.855567 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.855405    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09c95c32-f60a-4f60-90ff-90f447247cce-workload-certs\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq\" (UID: \"09c95c32-f60a-4f60-90ff-90f447247cce\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq"
Oct 30 13:40:57.855567 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.855425    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23116724-6a57-401b-8ba4-a105a71a5e6d-credential-socket\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj\" (UID: \"23116724-6a57-401b-8ba4-a105a71a5e6d\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj"
Oct 30 13:40:57.855567 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.855462    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09c95c32-f60a-4f60-90ff-90f447247cce-istio-envoy\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq\" (UID: \"09c95c32-f60a-4f60-90ff-90f447247cce\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq"
Oct 30 13:40:57.855567 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.855502    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23116724-6a57-401b-8ba4-a105a71a5e6d-istio-data\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj\" (UID: \"23116724-6a57-401b-8ba4-a105a71a5e6d\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj"
Oct 30 13:40:57.855567 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.855542    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23116724-6a57-401b-8ba4-a105a71a5e6d-workload-certs\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj\" (UID: \"23116724-6a57-401b-8ba4-a105a71a5e6d\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj"
Oct 30 13:40:57.856010 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.855593    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23116724-6a57-401b-8ba4-a105a71a5e6d-istiod-ca-cert\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj\" (UID: \"23116724-6a57-401b-8ba4-a105a71a5e6d\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj"
Oct 30 13:40:57.856010 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.855807    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c95c32-f60a-4f60-90ff-90f447247cce-workload-socket\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq\" (UID: \"09c95c32-f60a-4f60-90ff-90f447247cce\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq"
Oct 30 13:40:57.856010 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.855929    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23116724-6a57-401b-8ba4-a105a71a5e6d-credential-socket\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj\" (UID: \"23116724-6a57-401b-8ba4-a105a71a5e6d\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj"
Oct 30 13:40:57.856921 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.856293    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23116724-6a57-401b-8ba4-a105a71a5e6d-workload-socket\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj\" (UID: \"23116724-6a57-401b-8ba4-a105a71a5e6d\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj"
Oct 30 13:40:57.856921 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.856492    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23116724-6a57-401b-8ba4-a105a71a5e6d-istio-data\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj\" (UID: \"23116724-6a57-401b-8ba4-a105a71a5e6d\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj"
Oct 30 13:40:57.856921 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.856527    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09c95c32-f60a-4f60-90ff-90f447247cce-istio-data\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq\" (UID: \"09c95c32-f60a-4f60-90ff-90f447247cce\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq"
Oct 30 13:40:57.856921 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.856849    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09c95c32-f60a-4f60-90ff-90f447247cce-workload-certs\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq\" (UID: \"09c95c32-f60a-4f60-90ff-90f447247cce\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq"
Oct 30 13:40:57.856921 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.856915    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c95c32-f60a-4f60-90ff-90f447247cce-credential-socket\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq\" (UID: \"09c95c32-f60a-4f60-90ff-90f447247cce\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq"
Oct 30 13:40:57.857469 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.857446    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09c95c32-f60a-4f60-90ff-90f447247cce-istiod-ca-cert\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq\" (UID: \"09c95c32-f60a-4f60-90ff-90f447247cce\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq"
Oct 30 13:40:57.858389 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.858368    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23116724-6a57-401b-8ba4-a105a71a5e6d-istio-envoy\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj\" (UID: \"23116724-6a57-401b-8ba4-a105a71a5e6d\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj"
Oct 30 13:40:57.858781 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.858756    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23116724-6a57-401b-8ba4-a105a71a5e6d-istio-podinfo\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj\" (UID: \"23116724-6a57-401b-8ba4-a105a71a5e6d\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj"
Oct 30 13:40:57.859243 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.859217    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09c95c32-f60a-4f60-90ff-90f447247cce-istio-envoy\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq\" (UID: \"09c95c32-f60a-4f60-90ff-90f447247cce\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq"
Oct 30 13:40:57.859341 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.859249    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09c95c32-f60a-4f60-90ff-90f447247cce-istio-podinfo\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq\" (UID: \"09c95c32-f60a-4f60-90ff-90f447247cce\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq"
Oct 30 13:40:57.879704 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.879671    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23116724-6a57-401b-8ba4-a105a71a5e6d-istio-token\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj\" (UID: \"23116724-6a57-401b-8ba4-a105a71a5e6d\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj"
Oct 30 13:40:57.882815 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.882762    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09c95c32-f60a-4f60-90ff-90f447247cce-istio-token\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq\" (UID: \"09c95c32-f60a-4f60-90ff-90f447247cce\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq"
Oct 30 13:40:57.882958 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.882834    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwc5b\" (UniqueName: \"kubernetes.io/projected/09c95c32-f60a-4f60-90ff-90f447247cce-kube-api-access-pwc5b\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq\" (UID: \"09c95c32-f60a-4f60-90ff-90f447247cce\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq"
Oct 30 13:40:57.884343 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.884314    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srhwh\" (UniqueName: \"kubernetes.io/projected/23116724-6a57-401b-8ba4-a105a71a5e6d-kube-api-access-srhwh\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj\" (UID: \"23116724-6a57-401b-8ba4-a105a71a5e6d\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj"
Oct 30 13:40:57.956742 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.956709    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09bcc5ff-a33b-4699-a797-58b18a5f63bd-istio-token\" (UniqueName: \"kubernetes.io/projected/09bcc5ff-a33b-4699-a797-58b18a5f63bd-istio-token\") pod \"09bcc5ff-a33b-4699-a797-58b18a5f63bd\" (UID: \"09bcc5ff-a33b-4699-a797-58b18a5f63bd\") "
Oct 30 13:40:57.956890 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.956829    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/09bcc5ff-a33b-4699-a797-58b18a5f63bd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09bcc5ff-a33b-4699-a797-58b18a5f63bd-istio-podinfo\") pod \"09bcc5ff-a33b-4699-a797-58b18a5f63bd\" (UID: \"09bcc5ff-a33b-4699-a797-58b18a5f63bd\") "
Oct 30 13:40:57.956977 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.956906    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09bcc5ff-a33b-4699-a797-58b18a5f63bd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09bcc5ff-a33b-4699-a797-58b18a5f63bd-istiod-ca-cert\") pod \"09bcc5ff-a33b-4699-a797-58b18a5f63bd\" (UID: \"09bcc5ff-a33b-4699-a797-58b18a5f63bd\") "
Oct 30 13:40:57.957039 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.956988    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09bcc5ff-a33b-4699-a797-58b18a5f63bd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09bcc5ff-a33b-4699-a797-58b18a5f63bd-workload-socket\") pod \"09bcc5ff-a33b-4699-a797-58b18a5f63bd\" (UID: \"09bcc5ff-a33b-4699-a797-58b18a5f63bd\") "
Oct 30 13:40:57.957039 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.957020    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09bcc5ff-a33b-4699-a797-58b18a5f63bd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09bcc5ff-a33b-4699-a797-58b18a5f63bd-istio-envoy\") pod \"09bcc5ff-a33b-4699-a797-58b18a5f63bd\" (UID: \"09bcc5ff-a33b-4699-a797-58b18a5f63bd\") "
Oct 30 13:40:57.957146 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.957089    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09bcc5ff-a33b-4699-a797-58b18a5f63bd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09bcc5ff-a33b-4699-a797-58b18a5f63bd-istio-data\") pod \"09bcc5ff-a33b-4699-a797-58b18a5f63bd\" (UID: \"09bcc5ff-a33b-4699-a797-58b18a5f63bd\") "
Oct 30 13:40:57.957236 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.957216    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09bcc5ff-a33b-4699-a797-58b18a5f63bd-kube-api-access-f5sdd\" (UniqueName: \"kubernetes.io/projected/09bcc5ff-a33b-4699-a797-58b18a5f63bd-kube-api-access-f5sdd\") pod \"09bcc5ff-a33b-4699-a797-58b18a5f63bd\" (UID: \"09bcc5ff-a33b-4699-a797-58b18a5f63bd\") "
Oct 30 13:40:57.957302 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.957242    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09bcc5ff-a33b-4699-a797-58b18a5f63bd-workload-socket" pod "09bcc5ff-a33b-4699-a797-58b18a5f63bd" (UID: "09bcc5ff-a33b-4699-a797-58b18a5f63bd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:57.957302 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.957259    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09bcc5ff-a33b-4699-a797-58b18a5f63bd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09bcc5ff-a33b-4699-a797-58b18a5f63bd-workload-certs\") pod \"09bcc5ff-a33b-4699-a797-58b18a5f63bd\" (UID: \"09bcc5ff-a33b-4699-a797-58b18a5f63bd\") "
Oct 30 13:40:57.957434 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.957400    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09bcc5ff-a33b-4699-a797-58b18a5f63bd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09bcc5ff-a33b-4699-a797-58b18a5f63bd-credential-socket\") pod \"09bcc5ff-a33b-4699-a797-58b18a5f63bd\" (UID: \"09bcc5ff-a33b-4699-a797-58b18a5f63bd\") "
Oct 30 13:40:57.957649 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.957398    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09bcc5ff-a33b-4699-a797-58b18a5f63bd-istio-data" pod "09bcc5ff-a33b-4699-a797-58b18a5f63bd" (UID: "09bcc5ff-a33b-4699-a797-58b18a5f63bd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:57.957746 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.957445    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09bcc5ff-a33b-4699-a797-58b18a5f63bd-istiod-ca-cert" pod "09bcc5ff-a33b-4699-a797-58b18a5f63bd" (UID: "09bcc5ff-a33b-4699-a797-58b18a5f63bd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:40:57.957746 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.957679    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09bcc5ff-a33b-4699-a797-58b18a5f63bd-workload-certs" pod "09bcc5ff-a33b-4699-a797-58b18a5f63bd" (UID: "09bcc5ff-a33b-4699-a797-58b18a5f63bd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:57.957746 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.957696    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09bcc5ff-a33b-4699-a797-58b18a5f63bd-credential-socket" pod "09bcc5ff-a33b-4699-a797-58b18a5f63bd" (UID: "09bcc5ff-a33b-4699-a797-58b18a5f63bd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:57.958310 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.958283    4364 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09bcc5ff-a33b-4699-a797-58b18a5f63bd-workload-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:40:57.958432 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.958314    4364 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09bcc5ff-a33b-4699-a797-58b18a5f63bd-credential-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:40:57.958432 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.958330    4364 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09bcc5ff-a33b-4699-a797-58b18a5f63bd-istiod-ca-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:40:57.958432 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.958345    4364 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09bcc5ff-a33b-4699-a797-58b18a5f63bd-workload-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:40:57.958432 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.958361    4364 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09bcc5ff-a33b-4699-a797-58b18a5f63bd-istio-data\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:40:57.959269 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.959243    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09bcc5ff-a33b-4699-a797-58b18a5f63bd-istio-podinfo" pod "09bcc5ff-a33b-4699-a797-58b18a5f63bd" (UID: "09bcc5ff-a33b-4699-a797-58b18a5f63bd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:40:57.959358 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.959252    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09bcc5ff-a33b-4699-a797-58b18a5f63bd-istio-token" pod "09bcc5ff-a33b-4699-a797-58b18a5f63bd" (UID: "09bcc5ff-a33b-4699-a797-58b18a5f63bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:57.959486 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.959464    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09bcc5ff-a33b-4699-a797-58b18a5f63bd-kube-api-access-f5sdd" pod "09bcc5ff-a33b-4699-a797-58b18a5f63bd" (UID: "09bcc5ff-a33b-4699-a797-58b18a5f63bd"). InnerVolumeSpecName "kube-api-access-f5sdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:57.959831 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.959804    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09bcc5ff-a33b-4699-a797-58b18a5f63bd-istio-envoy" pod "09bcc5ff-a33b-4699-a797-58b18a5f63bd" (UID: "09bcc5ff-a33b-4699-a797-58b18a5f63bd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:57.978245 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:57.978199    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj"
Oct 30 13:40:58.028099 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:58.028025    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq"
Oct 30 13:40:58.061223 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:58.061185    4364 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09bcc5ff-a33b-4699-a797-58b18a5f63bd-istio-token\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:40:58.061368 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:58.061228    4364 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09bcc5ff-a33b-4699-a797-58b18a5f63bd-istio-podinfo\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:40:58.061368 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:58.061245    4364 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09bcc5ff-a33b-4699-a797-58b18a5f63bd-istio-envoy\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:40:58.061368 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:58.061260    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5sdd\" (UniqueName: \"kubernetes.io/projected/09bcc5ff-a33b-4699-a797-58b18a5f63bd-kube-api-access-f5sdd\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:40:58.303912 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:58.303885    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj"]
Oct 30 13:40:58.305440 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:40:58.305408    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23116724_6a57_401b_8ba4_a105a71a5e6d.slice/crio-a4b2a8849de9b57834cc3d4d09ce1112258ce9e34da69e77d6871ea30438d87c WatchSource:0}: Error finding container a4b2a8849de9b57834cc3d4d09ce1112258ce9e34da69e77d6871ea30438d87c: Status 404 returned error can't find the container with id a4b2a8849de9b57834cc3d4d09ce1112258ce9e34da69e77d6871ea30438d87c
Oct 30 13:40:58.308437 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:58.308395    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:40:58.308647 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:58.308496    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:40:58.308647 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:58.308545    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:40:58.352836 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:58.352803    4364 generic.go:356] "Generic (PLEG): container finished" podID="09bcc5ff-a33b-4699-a797-58b18a5f63bd" containerID="5e78e34aa30d512d552790b9f84b026d6c4a5dca1b4ea04c843cefa29daad33e" exitCode=0
Oct 30 13:40:58.352990 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:58.352855    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp"
Oct 30 13:40:58.352990 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:58.352879    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp" event={"ID":"09bcc5ff-a33b-4699-a797-58b18a5f63bd","Type":"ContainerDied","Data":"5e78e34aa30d512d552790b9f84b026d6c4a5dca1b4ea04c843cefa29daad33e"}
Oct 30 13:40:58.352990 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:58.352958    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp" event={"ID":"09bcc5ff-a33b-4699-a797-58b18a5f63bd","Type":"ContainerDied","Data":"b863f27ba21dac577c59806167067db7a02c727cda2a36bd96f315c0195c38b0"}
Oct 30 13:40:58.353149 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:58.352994    4364 scope.go:122] "RemoveContainer" containerID="5e78e34aa30d512d552790b9f84b026d6c4a5dca1b4ea04c843cefa29daad33e"
Oct 30 13:40:58.355111 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:58.355089    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj" event={"ID":"23116724-6a57-401b-8ba4-a105a71a5e6d","Type":"ContainerStarted","Data":"a4b2a8849de9b57834cc3d4d09ce1112258ce9e34da69e77d6871ea30438d87c"}
Oct 30 13:40:58.377756 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:58.377052    4364 scope.go:122] "RemoveContainer" containerID="5e78e34aa30d512d552790b9f84b026d6c4a5dca1b4ea04c843cefa29daad33e"
Oct 30 13:40:58.377756 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:40:58.377374    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e78e34aa30d512d552790b9f84b026d6c4a5dca1b4ea04c843cefa29daad33e\": container with ID starting with 5e78e34aa30d512d552790b9f84b026d6c4a5dca1b4ea04c843cefa29daad33e not found: ID does not exist" containerID="5e78e34aa30d512d552790b9f84b026d6c4a5dca1b4ea04c843cefa29daad33e"
Oct 30 13:40:58.377756 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:58.377399    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e78e34aa30d512d552790b9f84b026d6c4a5dca1b4ea04c843cefa29daad33e"} err="failed to get container status \"5e78e34aa30d512d552790b9f84b026d6c4a5dca1b4ea04c843cefa29daad33e\": rpc error: code = NotFound desc = could not find container \"5e78e34aa30d512d552790b9f84b026d6c4a5dca1b4ea04c843cefa29daad33e\": container with ID starting with 5e78e34aa30d512d552790b9f84b026d6c4a5dca1b4ea04c843cefa29daad33e not found: ID does not exist"
Oct 30 13:40:58.381807 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:58.381780    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq"]
Oct 30 13:40:58.382740 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:40:58.382716    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09c95c32_f60a_4f60_90ff_90f447247cce.slice/crio-6435d08519d683b06c67039264e3ab9e7860e19b8b4dd1b78c6a17a62b2a5acb WatchSource:0}: Error finding container 6435d08519d683b06c67039264e3ab9e7860e19b8b4dd1b78c6a17a62b2a5acb: Status 404 returned error can't find the container with id 6435d08519d683b06c67039264e3ab9e7860e19b8b4dd1b78c6a17a62b2a5acb
Oct 30 13:40:58.385486 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:58.385457    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:40:58.385619 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:58.385519    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:40:58.385619 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:58.385547    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:40:58.414918 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:58.414786    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp"]
Oct 30 13:40:58.421467 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:58.421438    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-qdmmp"]
Oct 30 13:40:58.972244 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:58.972201    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09bcc5ff-a33b-4699-a797-58b18a5f63bd" path="/var/lib/kubelet/pods/09bcc5ff-a33b-4699-a797-58b18a5f63bd/volumes"
Oct 30 13:40:59.365925 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:59.365842    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj" event={"ID":"23116724-6a57-401b-8ba4-a105a71a5e6d","Type":"ContainerStarted","Data":"6865a7361de91246a238324421ad90d014329021e8a40a8a1b7978c00063bd13"}
Oct 30 13:40:59.368185 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:59.368152    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq" event={"ID":"09c95c32-f60a-4f60-90ff-90f447247cce","Type":"ContainerStarted","Data":"a526058e96bf8fe46f5fac380c980be2b9991ed7f86bf8b2825b56a417636a73"}
Oct 30 13:40:59.368301 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:59.368200    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq" event={"ID":"09c95c32-f60a-4f60-90ff-90f447247cce","Type":"ContainerStarted","Data":"6435d08519d683b06c67039264e3ab9e7860e19b8b4dd1b78c6a17a62b2a5acb"}
Oct 30 13:40:59.450226 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:59.450171    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj" podStartSLOduration=2.450156661 podStartE2EDuration="2.450156661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:40:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:40:59.435086531 +0000 UTC m=+2958.601451204" watchObservedRunningTime="2026-10-30 13:40:59.450156661 +0000 UTC m=+2958.616521308"
Oct 30 13:40:59.533618 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:59.533059    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq" podStartSLOduration=2.533038242 podStartE2EDuration="2.533038242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:40:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:40:59.501970646 +0000 UTC m=+2958.668335300" watchObservedRunningTime="2026-10-30 13:40:59.533038242 +0000 UTC m=+2958.699402891"
Oct 30 13:41:00.006762 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:59.984131    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj"
Oct 30 13:41:00.006762 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:59.984295    4364 patch_prober.go:29] interesting pod/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body=
Oct 30 13:41:00.006762 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:40:59.984338    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj" podUID="23116724-6a57-401b-8ba4-a105a71a5e6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused"
Oct 30 13:41:00.046707 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:00.046665    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq"
Oct 30 13:41:00.047974 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:00.047942    4364 patch_prober.go:29] interesting pod/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" start-of-body=
Oct 30 13:41:00.048121 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:00.048016    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq" podUID="09c95c32-f60a-4f60-90ff-90f447247cce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused"
Oct 30 13:41:01.009623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:01.000352    4364 patch_prober.go:29] interesting pod/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body=
Oct 30 13:41:01.009623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:01.000471    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj" podUID="23116724-6a57-401b-8ba4-a105a71a5e6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused"
Oct 30 13:41:01.029417 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:01.029375    4364 patch_prober.go:29] interesting pod/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" start-of-body=
Oct 30 13:41:01.029744 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:01.029697    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq" podUID="09c95c32-f60a-4f60-90ff-90f447247cce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused"
Oct 30 13:41:01.521624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:01.519928    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr"]
Oct 30 13:41:02.021622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:02.021187    4364 patch_prober.go:29] interesting pod/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body=
Oct 30 13:41:02.021622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:02.021252    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj" podUID="23116724-6a57-401b-8ba4-a105a71a5e6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused"
Oct 30 13:41:02.033126 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:02.033078    4364 patch_prober.go:29] interesting pod/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" start-of-body=
Oct 30 13:41:02.033406 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:02.033362    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq" podUID="09c95c32-f60a-4f60-90ff-90f447247cce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused"
Oct 30 13:41:03.010625 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:03.007631    4364 patch_prober.go:29] interesting pod/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body=
Oct 30 13:41:03.010625 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:03.007695    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj" podUID="23116724-6a57-401b-8ba4-a105a71a5e6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused"
Oct 30 13:41:03.032081 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:03.028487    4364 patch_prober.go:29] interesting pod/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" start-of-body=
Oct 30 13:41:03.032081 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:03.028544    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq" podUID="09c95c32-f60a-4f60-90ff-90f447247cce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused"
Oct 30 13:41:03.986628 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:03.983029    4364 patch_prober.go:29] interesting pod/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body=
Oct 30 13:41:03.986628 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:03.983094    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj" podUID="23116724-6a57-401b-8ba4-a105a71a5e6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused"
Oct 30 13:41:04.029068 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:04.029034    4364 patch_prober.go:29] interesting pod/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" start-of-body=
Oct 30 13:41:04.029392 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:04.029347    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq" podUID="09c95c32-f60a-4f60-90ff-90f447247cce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused"
Oct 30 13:41:04.980009 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:04.979975    4364 patch_prober.go:29] interesting pod/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body=
Oct 30 13:41:04.980443 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:04.980040    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj" podUID="23116724-6a57-401b-8ba4-a105a71a5e6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused"
Oct 30 13:41:05.028801 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:05.028761    4364 patch_prober.go:29] interesting pod/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" start-of-body=
Oct 30 13:41:05.028995 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:05.028845    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq" podUID="09c95c32-f60a-4f60-90ff-90f447247cce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused"
Oct 30 13:41:05.984273 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:05.984242    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj"
Oct 30 13:41:05.984736 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:05.984640    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj"
Oct 30 13:41:05.985114 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:05.985097    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj"
Oct 30 13:41:06.033268 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:06.033237    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq"
Oct 30 13:41:06.033526 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:06.033507    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq"
Oct 30 13:41:06.034245 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:06.034218    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq"
Oct 30 13:41:07.097760 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:07.097723    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-cb58b8-b449f8b58-r97fs"]
Oct 30 13:41:07.135439 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:07.135409    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-cb58b8-78744489c9-lshhg"]
Oct 30 13:41:11.507612 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:41:11.507563    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:41:12.098381 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:12.098340    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-cb58b8-b449f8b58-r97fs" podUID="f42a48a8-5f5e-459c-8276-dd30d02052a7" containerName="payload-processing" containerID="cri-o://4dee5c972eb3862e27b351da3a3746d5616c2fc60f0d461cf7e7cc0dc3ea87d3" gracePeriod=25
Oct 30 13:41:12.136703 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:12.136671    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-cb58b8-78744489c9-lshhg" podUID="0b45c4c8-a116-43ce-9e06-6b4ceca907e2" containerName="payload-pre-processing" containerID="cri-o://063041b92ffbf4394ddc2c754fcddb54dfb173cc1618a5510d9fb1273f0b2ed3" gracePeriod=25
Oct 30 13:41:12.419425 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:12.419390    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-cb58b8-b449f8b58-r97fs"
Oct 30 13:41:12.424814 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:12.424780    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f42a48a8-5f5e-459c-8276-dd30d02052a7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f42a48a8-5f5e-459c-8276-dd30d02052a7-plugins-config-volume\") pod \"f42a48a8-5f5e-459c-8276-dd30d02052a7\" (UID: \"f42a48a8-5f5e-459c-8276-dd30d02052a7\") "
Oct 30 13:41:12.425143 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:12.425120    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f42a48a8-5f5e-459c-8276-dd30d02052a7-plugins-config-volume" pod "f42a48a8-5f5e-459c-8276-dd30d02052a7" (UID: "f42a48a8-5f5e-459c-8276-dd30d02052a7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:41:12.425246 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:12.425141    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f42a48a8-5f5e-459c-8276-dd30d02052a7-kube-api-access-n7mlx\" (UniqueName: \"kubernetes.io/projected/f42a48a8-5f5e-459c-8276-dd30d02052a7-kube-api-access-n7mlx\") pod \"f42a48a8-5f5e-459c-8276-dd30d02052a7\" (UID: \"f42a48a8-5f5e-459c-8276-dd30d02052a7\") "
Oct 30 13:41:12.425929 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:12.425902    4364 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f42a48a8-5f5e-459c-8276-dd30d02052a7-plugins-config-volume\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:41:12.427278 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:12.427250    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f42a48a8-5f5e-459c-8276-dd30d02052a7-kube-api-access-n7mlx" pod "f42a48a8-5f5e-459c-8276-dd30d02052a7" (UID: "f42a48a8-5f5e-459c-8276-dd30d02052a7"). InnerVolumeSpecName "kube-api-access-n7mlx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:41:12.470796 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:12.470770    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-cb58b8-78744489c9-lshhg"
Oct 30 13:41:12.527179 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:12.527138    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b45c4c8-a116-43ce-9e06-6b4ceca907e2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b45c4c8-a116-43ce-9e06-6b4ceca907e2-plugins-config-volume\") pod \"0b45c4c8-a116-43ce-9e06-6b4ceca907e2\" (UID: \"0b45c4c8-a116-43ce-9e06-6b4ceca907e2\") "
Oct 30 13:41:12.527711 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:12.527283    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b45c4c8-a116-43ce-9e06-6b4ceca907e2-kube-api-access-p25fk\" (UniqueName: \"kubernetes.io/projected/0b45c4c8-a116-43ce-9e06-6b4ceca907e2-kube-api-access-p25fk\") pod \"0b45c4c8-a116-43ce-9e06-6b4ceca907e2\" (UID: \"0b45c4c8-a116-43ce-9e06-6b4ceca907e2\") "
Oct 30 13:41:12.527711 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:12.527488    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b45c4c8-a116-43ce-9e06-6b4ceca907e2-plugins-config-volume" pod "0b45c4c8-a116-43ce-9e06-6b4ceca907e2" (UID: "0b45c4c8-a116-43ce-9e06-6b4ceca907e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:41:12.528402 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:12.528378    4364 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b45c4c8-a116-43ce-9e06-6b4ceca907e2-plugins-config-volume\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:41:12.528402 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:12.528400    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7mlx\" (UniqueName: \"kubernetes.io/projected/f42a48a8-5f5e-459c-8276-dd30d02052a7-kube-api-access-n7mlx\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:41:12.530406 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:12.530373    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b45c4c8-a116-43ce-9e06-6b4ceca907e2-kube-api-access-p25fk" pod "0b45c4c8-a116-43ce-9e06-6b4ceca907e2" (UID: "0b45c4c8-a116-43ce-9e06-6b4ceca907e2"). InnerVolumeSpecName "kube-api-access-p25fk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:41:12.570552 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:12.570522    4364 generic.go:356] "Generic (PLEG): container finished" podID="0b45c4c8-a116-43ce-9e06-6b4ceca907e2" containerID="063041b92ffbf4394ddc2c754fcddb54dfb173cc1618a5510d9fb1273f0b2ed3" exitCode=0
Oct 30 13:41:12.570727 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:12.570582    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-cb58b8-78744489c9-lshhg"
Oct 30 13:41:12.570727 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:12.570628    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-cb58b8-78744489c9-lshhg" event={"ID":"0b45c4c8-a116-43ce-9e06-6b4ceca907e2","Type":"ContainerDied","Data":"063041b92ffbf4394ddc2c754fcddb54dfb173cc1618a5510d9fb1273f0b2ed3"}
Oct 30 13:41:12.570727 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:12.570688    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-cb58b8-78744489c9-lshhg" event={"ID":"0b45c4c8-a116-43ce-9e06-6b4ceca907e2","Type":"ContainerDied","Data":"0e8dd6761ea66081d27d6f36c998af0a00ee82f5ceb374b12478fb33bb0350e4"}
Oct 30 13:41:12.570727 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:12.570714    4364 scope.go:122] "RemoveContainer" containerID="063041b92ffbf4394ddc2c754fcddb54dfb173cc1618a5510d9fb1273f0b2ed3"
Oct 30 13:41:12.572767 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:12.572743    4364 generic.go:356] "Generic (PLEG): container finished" podID="f42a48a8-5f5e-459c-8276-dd30d02052a7" containerID="4dee5c972eb3862e27b351da3a3746d5616c2fc60f0d461cf7e7cc0dc3ea87d3" exitCode=0
Oct 30 13:41:12.572897 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:12.572790    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-cb58b8-b449f8b58-r97fs" event={"ID":"f42a48a8-5f5e-459c-8276-dd30d02052a7","Type":"ContainerDied","Data":"4dee5c972eb3862e27b351da3a3746d5616c2fc60f0d461cf7e7cc0dc3ea87d3"}
Oct 30 13:41:12.572897 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:12.572825    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-cb58b8-b449f8b58-r97fs" event={"ID":"f42a48a8-5f5e-459c-8276-dd30d02052a7","Type":"ContainerDied","Data":"42d3168f4971db601c8f2a2ebcf5ac1263a13aff27c7ba46ee30c5893497c35e"}
Oct 30 13:41:12.572897 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:12.572796    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-cb58b8-b449f8b58-r97fs"
Oct 30 13:41:12.585114 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:12.585087    4364 scope.go:122] "RemoveContainer" containerID="063041b92ffbf4394ddc2c754fcddb54dfb173cc1618a5510d9fb1273f0b2ed3"
Oct 30 13:41:12.585437 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:41:12.585413    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"063041b92ffbf4394ddc2c754fcddb54dfb173cc1618a5510d9fb1273f0b2ed3\": container with ID starting with 063041b92ffbf4394ddc2c754fcddb54dfb173cc1618a5510d9fb1273f0b2ed3 not found: ID does not exist" containerID="063041b92ffbf4394ddc2c754fcddb54dfb173cc1618a5510d9fb1273f0b2ed3"
Oct 30 13:41:12.585506 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:12.585449    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"063041b92ffbf4394ddc2c754fcddb54dfb173cc1618a5510d9fb1273f0b2ed3"} err="failed to get container status \"063041b92ffbf4394ddc2c754fcddb54dfb173cc1618a5510d9fb1273f0b2ed3\": rpc error: code = NotFound desc = could not find container \"063041b92ffbf4394ddc2c754fcddb54dfb173cc1618a5510d9fb1273f0b2ed3\": container with ID starting with 063041b92ffbf4394ddc2c754fcddb54dfb173cc1618a5510d9fb1273f0b2ed3 not found: ID does not exist"
Oct 30 13:41:12.585506 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:12.585475    4364 scope.go:122] "RemoveContainer" containerID="4dee5c972eb3862e27b351da3a3746d5616c2fc60f0d461cf7e7cc0dc3ea87d3"
Oct 30 13:41:12.596285 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:12.596267    4364 scope.go:122] "RemoveContainer" containerID="4dee5c972eb3862e27b351da3a3746d5616c2fc60f0d461cf7e7cc0dc3ea87d3"
Oct 30 13:41:12.596551 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:41:12.596528    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dee5c972eb3862e27b351da3a3746d5616c2fc60f0d461cf7e7cc0dc3ea87d3\": container with ID starting with 4dee5c972eb3862e27b351da3a3746d5616c2fc60f0d461cf7e7cc0dc3ea87d3 not found: ID does not exist" containerID="4dee5c972eb3862e27b351da3a3746d5616c2fc60f0d461cf7e7cc0dc3ea87d3"
Oct 30 13:41:12.596689 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:12.596555    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dee5c972eb3862e27b351da3a3746d5616c2fc60f0d461cf7e7cc0dc3ea87d3"} err="failed to get container status \"4dee5c972eb3862e27b351da3a3746d5616c2fc60f0d461cf7e7cc0dc3ea87d3\": rpc error: code = NotFound desc = could not find container \"4dee5c972eb3862e27b351da3a3746d5616c2fc60f0d461cf7e7cc0dc3ea87d3\": container with ID starting with 4dee5c972eb3862e27b351da3a3746d5616c2fc60f0d461cf7e7cc0dc3ea87d3 not found: ID does not exist"
Oct 30 13:41:12.624950 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:12.624924    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-cb58b8-b449f8b58-r97fs"]
Oct 30 13:41:12.629457 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:12.629399    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-cb58b8-b449f8b58-r97fs"]
Oct 30 13:41:12.630314 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:12.630285    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p25fk\" (UniqueName: \"kubernetes.io/projected/0b45c4c8-a116-43ce-9e06-6b4ceca907e2-kube-api-access-p25fk\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:41:12.661838 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:12.661776    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-cb58b8-78744489c9-lshhg"]
Oct 30 13:41:12.665993 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:12.665973    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-cb58b8-78744489c9-lshhg"]
Oct 30 13:41:12.971450 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:12.971373    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b45c4c8-a116-43ce-9e06-6b4ceca907e2" path="/var/lib/kubelet/pods/0b45c4c8-a116-43ce-9e06-6b4ceca907e2/volumes"
Oct 30 13:41:12.971851 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:12.971837    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f42a48a8-5f5e-459c-8276-dd30d02052a7" path="/var/lib/kubelet/pods/f42a48a8-5f5e-459c-8276-dd30d02052a7/volumes"
Oct 30 13:41:16.554884 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:16.554846    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr" podUID="446c12df-de12-4c7c-8c71-dfed93cc33f1" containerName="main" containerID="cri-o://50462199db7e93b6faff758547a9ab3723af306bc5074114c36b245f146f0f08" gracePeriod=45
Oct 30 13:41:16.671643 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:16.663422    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr" podUID="446c12df-de12-4c7c-8c71-dfed93cc33f1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": dial tcp 10.133.0.91:8000: connect: connection refused"
Oct 30 13:41:17.004342 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:17.004314    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr"
Oct 30 13:41:17.114086 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:17.114052    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/446c12df-de12-4c7c-8c71-dfed93cc33f1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/446c12df-de12-4c7c-8c71-dfed93cc33f1-dshm\") pod \"446c12df-de12-4c7c-8c71-dfed93cc33f1\" (UID: \"446c12df-de12-4c7c-8c71-dfed93cc33f1\") "
Oct 30 13:41:17.114273 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:17.114165    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/446c12df-de12-4c7c-8c71-dfed93cc33f1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/446c12df-de12-4c7c-8c71-dfed93cc33f1-model-cache\") pod \"446c12df-de12-4c7c-8c71-dfed93cc33f1\" (UID: \"446c12df-de12-4c7c-8c71-dfed93cc33f1\") "
Oct 30 13:41:17.114349 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:17.114330    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/446c12df-de12-4c7c-8c71-dfed93cc33f1-kube-api-access-b5qtj\" (UniqueName: \"kubernetes.io/projected/446c12df-de12-4c7c-8c71-dfed93cc33f1-kube-api-access-b5qtj\") pod \"446c12df-de12-4c7c-8c71-dfed93cc33f1\" (UID: \"446c12df-de12-4c7c-8c71-dfed93cc33f1\") "
Oct 30 13:41:17.114443 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:17.114418    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/446c12df-de12-4c7c-8c71-dfed93cc33f1-home\" (UniqueName: \"kubernetes.io/empty-dir/446c12df-de12-4c7c-8c71-dfed93cc33f1-home\") pod \"446c12df-de12-4c7c-8c71-dfed93cc33f1\" (UID: \"446c12df-de12-4c7c-8c71-dfed93cc33f1\") "
Oct 30 13:41:17.114558 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:17.114468    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/446c12df-de12-4c7c-8c71-dfed93cc33f1-tls-certs\" (UniqueName: \"kubernetes.io/secret/446c12df-de12-4c7c-8c71-dfed93cc33f1-tls-certs\") pod \"446c12df-de12-4c7c-8c71-dfed93cc33f1\" (UID: \"446c12df-de12-4c7c-8c71-dfed93cc33f1\") "
Oct 30 13:41:17.114558 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:17.114525    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/446c12df-de12-4c7c-8c71-dfed93cc33f1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/446c12df-de12-4c7c-8c71-dfed93cc33f1-tmp-dir\") pod \"446c12df-de12-4c7c-8c71-dfed93cc33f1\" (UID: \"446c12df-de12-4c7c-8c71-dfed93cc33f1\") "
Oct 30 13:41:17.115148 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:17.115120    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/446c12df-de12-4c7c-8c71-dfed93cc33f1-model-cache" pod "446c12df-de12-4c7c-8c71-dfed93cc33f1" (UID: "446c12df-de12-4c7c-8c71-dfed93cc33f1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:41:17.115768 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:17.115742    4364 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/446c12df-de12-4c7c-8c71-dfed93cc33f1-model-cache\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:41:17.115878 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:17.115765    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/446c12df-de12-4c7c-8c71-dfed93cc33f1-tmp-dir" pod "446c12df-de12-4c7c-8c71-dfed93cc33f1" (UID: "446c12df-de12-4c7c-8c71-dfed93cc33f1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:41:17.115951 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:17.115883    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/446c12df-de12-4c7c-8c71-dfed93cc33f1-home" pod "446c12df-de12-4c7c-8c71-dfed93cc33f1" (UID: "446c12df-de12-4c7c-8c71-dfed93cc33f1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:41:17.118109 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:17.118026    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/446c12df-de12-4c7c-8c71-dfed93cc33f1-kube-api-access-b5qtj" pod "446c12df-de12-4c7c-8c71-dfed93cc33f1" (UID: "446c12df-de12-4c7c-8c71-dfed93cc33f1"). InnerVolumeSpecName "kube-api-access-b5qtj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:41:17.118243 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:17.118182    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/446c12df-de12-4c7c-8c71-dfed93cc33f1-dshm" pod "446c12df-de12-4c7c-8c71-dfed93cc33f1" (UID: "446c12df-de12-4c7c-8c71-dfed93cc33f1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:41:17.118320 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:17.118274    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/446c12df-de12-4c7c-8c71-dfed93cc33f1-tls-certs" pod "446c12df-de12-4c7c-8c71-dfed93cc33f1" (UID: "446c12df-de12-4c7c-8c71-dfed93cc33f1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:41:17.217076 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:17.216996    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5qtj\" (UniqueName: \"kubernetes.io/projected/446c12df-de12-4c7c-8c71-dfed93cc33f1-kube-api-access-b5qtj\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:41:17.217076 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:17.217027    4364 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/446c12df-de12-4c7c-8c71-dfed93cc33f1-home\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:41:17.217076 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:17.217037    4364 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/446c12df-de12-4c7c-8c71-dfed93cc33f1-tls-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:41:17.217076 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:17.217046    4364 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/446c12df-de12-4c7c-8c71-dfed93cc33f1-tmp-dir\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:41:17.217076 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:17.217053    4364 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/446c12df-de12-4c7c-8c71-dfed93cc33f1-dshm\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:41:17.733680 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:17.733643    4364 generic.go:356] "Generic (PLEG): container finished" podID="446c12df-de12-4c7c-8c71-dfed93cc33f1" containerID="50462199db7e93b6faff758547a9ab3723af306bc5074114c36b245f146f0f08" exitCode=0
Oct 30 13:41:17.734113 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:17.733685    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr"
Oct 30 13:41:17.734113 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:17.733737    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr" event={"ID":"446c12df-de12-4c7c-8c71-dfed93cc33f1","Type":"ContainerDied","Data":"50462199db7e93b6faff758547a9ab3723af306bc5074114c36b245f146f0f08"}
Oct 30 13:41:17.734113 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:17.733787    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr" event={"ID":"446c12df-de12-4c7c-8c71-dfed93cc33f1","Type":"ContainerDied","Data":"a64639c9ca43dfafe0f4c3d8421170673c0e87851640016f1d1e2ac1eb9468a9"}
Oct 30 13:41:17.734113 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:17.733811    4364 scope.go:122] "RemoveContainer" containerID="50462199db7e93b6faff758547a9ab3723af306bc5074114c36b245f146f0f08"
Oct 30 13:41:17.746278 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:17.746261    4364 scope.go:122] "RemoveContainer" containerID="50462199db7e93b6faff758547a9ab3723af306bc5074114c36b245f146f0f08"
Oct 30 13:41:17.746544 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:41:17.746524    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50462199db7e93b6faff758547a9ab3723af306bc5074114c36b245f146f0f08\": container with ID starting with 50462199db7e93b6faff758547a9ab3723af306bc5074114c36b245f146f0f08 not found: ID does not exist" containerID="50462199db7e93b6faff758547a9ab3723af306bc5074114c36b245f146f0f08"
Oct 30 13:41:17.746652 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:17.746551    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50462199db7e93b6faff758547a9ab3723af306bc5074114c36b245f146f0f08"} err="failed to get container status \"50462199db7e93b6faff758547a9ab3723af306bc5074114c36b245f146f0f08\": rpc error: code = NotFound desc = could not find container \"50462199db7e93b6faff758547a9ab3723af306bc5074114c36b245f146f0f08\": container with ID starting with 50462199db7e93b6faff758547a9ab3723af306bc5074114c36b245f146f0f08 not found: ID does not exist"
Oct 30 13:41:17.787925 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:17.787906    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr"]
Oct 30 13:41:17.792215 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:17.792197    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-cb58b8/distinct-2-w2-cb58b8-kserve-76d7c9b598-q5mtr"]
Oct 30 13:41:19.060840 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:19.060800    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="446c12df-de12-4c7c-8c71-dfed93cc33f1" path="/var/lib/kubelet/pods/446c12df-de12-4c7c-8c71-dfed93cc33f1/volumes"
Oct 30 13:41:30.343446 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:30.343415    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:41:38.169629 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:38.168633    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-mqpfw_310804ff-17ef-4a28-959c-47995bde35ca/payload-processing/0.log"
Oct 30 13:41:38.374783 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:41:38.374756    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-mqpfw_310804ff-17ef-4a28-959c-47995bde35ca/payload-processing/0.log"
Oct 30 13:41:41.506073 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:41:41.506036    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:42:11.506517 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:42:11.506479    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:42:21.329456 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:21.329418    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c54f0f-5d446cb49d-dhfks"]
Oct 30 13:42:21.330055 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:21.329661    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c54f0f-5d446cb49d-dhfks" podUID="89934eb4-b154-412f-9f02-2f33c0fcb2e3" containerName="maas-api" containerID="cri-o://4010466c718b75c9dc075f672ac8e28eb063aa12a6abdeb108d45a58c049ca56" gracePeriod=30
Oct 30 13:42:21.491266 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:21.491229    4364 generic.go:356] "Generic (PLEG): container finished" podID="89934eb4-b154-412f-9f02-2f33c0fcb2e3" containerID="4010466c718b75c9dc075f672ac8e28eb063aa12a6abdeb108d45a58c049ca56" exitCode=0
Oct 30 13:42:21.491428 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:21.491337    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c54f0f-5d446cb49d-dhfks" event={"ID":"89934eb4-b154-412f-9f02-2f33c0fcb2e3","Type":"ContainerDied","Data":"4010466c718b75c9dc075f672ac8e28eb063aa12a6abdeb108d45a58c049ca56"}
Oct 30 13:42:21.665717 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:21.665694    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c54f0f-5d446cb49d-dhfks"
Oct 30 13:42:21.741266 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:21.741235    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89934eb4-b154-412f-9f02-2f33c0fcb2e3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/89934eb4-b154-412f-9f02-2f33c0fcb2e3-metrics-certs\") pod \"89934eb4-b154-412f-9f02-2f33c0fcb2e3\" (UID: \"89934eb4-b154-412f-9f02-2f33c0fcb2e3\") "
Oct 30 13:42:21.741460 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:21.741280    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89934eb4-b154-412f-9f02-2f33c0fcb2e3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89934eb4-b154-412f-9f02-2f33c0fcb2e3-openshift-service-ca\") pod \"89934eb4-b154-412f-9f02-2f33c0fcb2e3\" (UID: \"89934eb4-b154-412f-9f02-2f33c0fcb2e3\") "
Oct 30 13:42:21.741460 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:21.741315    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89934eb4-b154-412f-9f02-2f33c0fcb2e3-kube-api-access-rgxn8\" (UniqueName: \"kubernetes.io/projected/89934eb4-b154-412f-9f02-2f33c0fcb2e3-kube-api-access-rgxn8\") pod \"89934eb4-b154-412f-9f02-2f33c0fcb2e3\" (UID: \"89934eb4-b154-412f-9f02-2f33c0fcb2e3\") "
Oct 30 13:42:21.741658 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:21.741462    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89934eb4-b154-412f-9f02-2f33c0fcb2e3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89934eb4-b154-412f-9f02-2f33c0fcb2e3-maas-api-tls\") pod \"89934eb4-b154-412f-9f02-2f33c0fcb2e3\" (UID: \"89934eb4-b154-412f-9f02-2f33c0fcb2e3\") "
Oct 30 13:42:21.741744 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:21.741661    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89934eb4-b154-412f-9f02-2f33c0fcb2e3-openshift-service-ca" pod "89934eb4-b154-412f-9f02-2f33c0fcb2e3" (UID: "89934eb4-b154-412f-9f02-2f33c0fcb2e3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:42:21.742293 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:21.742267    4364 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89934eb4-b154-412f-9f02-2f33c0fcb2e3-openshift-service-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:42:21.744646 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:21.744570    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89934eb4-b154-412f-9f02-2f33c0fcb2e3-metrics-certs" pod "89934eb4-b154-412f-9f02-2f33c0fcb2e3" (UID: "89934eb4-b154-412f-9f02-2f33c0fcb2e3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:42:21.744981 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:21.744954    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89934eb4-b154-412f-9f02-2f33c0fcb2e3-kube-api-access-rgxn8" pod "89934eb4-b154-412f-9f02-2f33c0fcb2e3" (UID: "89934eb4-b154-412f-9f02-2f33c0fcb2e3"). InnerVolumeSpecName "kube-api-access-rgxn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:42:21.746777 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:21.746750    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89934eb4-b154-412f-9f02-2f33c0fcb2e3-maas-api-tls" pod "89934eb4-b154-412f-9f02-2f33c0fcb2e3" (UID: "89934eb4-b154-412f-9f02-2f33c0fcb2e3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:42:21.922672 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:21.867341    4364 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89934eb4-b154-412f-9f02-2f33c0fcb2e3-metrics-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:42:21.922672 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:21.867382    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgxn8\" (UniqueName: \"kubernetes.io/projected/89934eb4-b154-412f-9f02-2f33c0fcb2e3-kube-api-access-rgxn8\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:42:21.922672 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:21.867401    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89934eb4-b154-412f-9f02-2f33c0fcb2e3-maas-api-tls\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:42:22.499800 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:22.499771    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c54f0f-5d446cb49d-dhfks"
Oct 30 13:42:22.500300 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:22.499814    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c54f0f-5d446cb49d-dhfks" event={"ID":"89934eb4-b154-412f-9f02-2f33c0fcb2e3","Type":"ContainerDied","Data":"57c1628165ba8e87a415a5f37005bf88d2ee4ffecd6e5959dfadaee2f5228266"}
Oct 30 13:42:22.500300 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:22.499885    4364 scope.go:122] "RemoveContainer" containerID="4010466c718b75c9dc075f672ac8e28eb063aa12a6abdeb108d45a58c049ca56"
Oct 30 13:42:22.553700 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:22.553673    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c54f0f-5d446cb49d-dhfks"]
Oct 30 13:42:22.565373 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:22.565349    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c54f0f-5d446cb49d-dhfks"]
Oct 30 13:42:22.972445 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:22.972416    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89934eb4-b154-412f-9f02-2f33c0fcb2e3" path="/var/lib/kubelet/pods/89934eb4-b154-412f-9f02-2f33c0fcb2e3/volumes"
Oct 30 13:42:26.286221 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:26.286188    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:42:31.057589 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:31.057543    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a8d3b8-7f9bfff7bc-hszb2"]
Oct 30 13:42:31.058092 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:31.057801    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a8d3b8-7f9bfff7bc-hszb2" podUID="213f7500-6291-40df-8a5e-c7008a6ccc47" containerName="maas-api" containerID="cri-o://e3f050944e199cae5734e83970ef006667b3d2e534b15e22321b96acdcd2689a" gracePeriod=30
Oct 30 13:42:31.369085 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:31.369053    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a8d3b8-7f9bfff7bc-hszb2"
Oct 30 13:42:31.451968 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:31.451935    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/213f7500-6291-40df-8a5e-c7008a6ccc47-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/213f7500-6291-40df-8a5e-c7008a6ccc47-maas-api-tls\") pod \"213f7500-6291-40df-8a5e-c7008a6ccc47\" (UID: \"213f7500-6291-40df-8a5e-c7008a6ccc47\") "
Oct 30 13:42:31.452163 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:31.451998    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/213f7500-6291-40df-8a5e-c7008a6ccc47-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/213f7500-6291-40df-8a5e-c7008a6ccc47-openshift-service-ca\") pod \"213f7500-6291-40df-8a5e-c7008a6ccc47\" (UID: \"213f7500-6291-40df-8a5e-c7008a6ccc47\") "
Oct 30 13:42:31.452163 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:31.452139    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/213f7500-6291-40df-8a5e-c7008a6ccc47-metrics-certs\" (UniqueName: \"kubernetes.io/secret/213f7500-6291-40df-8a5e-c7008a6ccc47-metrics-certs\") pod \"213f7500-6291-40df-8a5e-c7008a6ccc47\" (UID: \"213f7500-6291-40df-8a5e-c7008a6ccc47\") "
Oct 30 13:42:31.452298 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:31.452268    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/213f7500-6291-40df-8a5e-c7008a6ccc47-kube-api-access-mwsqq\" (UniqueName: \"kubernetes.io/projected/213f7500-6291-40df-8a5e-c7008a6ccc47-kube-api-access-mwsqq\") pod \"213f7500-6291-40df-8a5e-c7008a6ccc47\" (UID: \"213f7500-6291-40df-8a5e-c7008a6ccc47\") "
Oct 30 13:42:31.452467 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:31.452446    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/213f7500-6291-40df-8a5e-c7008a6ccc47-openshift-service-ca" pod "213f7500-6291-40df-8a5e-c7008a6ccc47" (UID: "213f7500-6291-40df-8a5e-c7008a6ccc47"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:42:31.453085 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:31.453045    4364 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/213f7500-6291-40df-8a5e-c7008a6ccc47-openshift-service-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:42:31.454440 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:31.454409    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/213f7500-6291-40df-8a5e-c7008a6ccc47-metrics-certs" pod "213f7500-6291-40df-8a5e-c7008a6ccc47" (UID: "213f7500-6291-40df-8a5e-c7008a6ccc47"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:42:31.455018 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:31.454993    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/213f7500-6291-40df-8a5e-c7008a6ccc47-maas-api-tls" pod "213f7500-6291-40df-8a5e-c7008a6ccc47" (UID: "213f7500-6291-40df-8a5e-c7008a6ccc47"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:42:31.455240 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:31.455220    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/213f7500-6291-40df-8a5e-c7008a6ccc47-kube-api-access-mwsqq" pod "213f7500-6291-40df-8a5e-c7008a6ccc47" (UID: "213f7500-6291-40df-8a5e-c7008a6ccc47"). InnerVolumeSpecName "kube-api-access-mwsqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:42:31.555786 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:31.555713    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwsqq\" (UniqueName: \"kubernetes.io/projected/213f7500-6291-40df-8a5e-c7008a6ccc47-kube-api-access-mwsqq\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:42:31.555786 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:31.555746    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/213f7500-6291-40df-8a5e-c7008a6ccc47-maas-api-tls\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:42:31.555786 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:31.555763    4364 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/213f7500-6291-40df-8a5e-c7008a6ccc47-metrics-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:42:31.691742 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:31.682390    4364 generic.go:356] "Generic (PLEG): container finished" podID="213f7500-6291-40df-8a5e-c7008a6ccc47" containerID="e3f050944e199cae5734e83970ef006667b3d2e534b15e22321b96acdcd2689a" exitCode=0
Oct 30 13:42:31.691742 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:31.682619    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a8d3b8-7f9bfff7bc-hszb2" event={"ID":"213f7500-6291-40df-8a5e-c7008a6ccc47","Type":"ContainerDied","Data":"e3f050944e199cae5734e83970ef006667b3d2e534b15e22321b96acdcd2689a"}
Oct 30 13:42:31.691742 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:31.682680    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a8d3b8-7f9bfff7bc-hszb2" event={"ID":"213f7500-6291-40df-8a5e-c7008a6ccc47","Type":"ContainerDied","Data":"738e71bc74d9e1c5dece10dc325dc371aeb5052afea71b5e7ab616966d914cec"}
Oct 30 13:42:31.691742 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:31.682710    4364 scope.go:122] "RemoveContainer" containerID="e3f050944e199cae5734e83970ef006667b3d2e534b15e22321b96acdcd2689a"
Oct 30 13:42:31.691742 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:31.682879    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a8d3b8-7f9bfff7bc-hszb2"
Oct 30 13:42:31.754876 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:31.754849    4364 scope.go:122] "RemoveContainer" containerID="e3f050944e199cae5734e83970ef006667b3d2e534b15e22321b96acdcd2689a"
Oct 30 13:42:31.755529 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:42:31.755506    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3f050944e199cae5734e83970ef006667b3d2e534b15e22321b96acdcd2689a\": container with ID starting with e3f050944e199cae5734e83970ef006667b3d2e534b15e22321b96acdcd2689a not found: ID does not exist" containerID="e3f050944e199cae5734e83970ef006667b3d2e534b15e22321b96acdcd2689a"
Oct 30 13:42:31.755726 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:31.755703    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3f050944e199cae5734e83970ef006667b3d2e534b15e22321b96acdcd2689a"} err="failed to get container status \"e3f050944e199cae5734e83970ef006667b3d2e534b15e22321b96acdcd2689a\": rpc error: code = NotFound desc = could not find container \"e3f050944e199cae5734e83970ef006667b3d2e534b15e22321b96acdcd2689a\": container with ID starting with e3f050944e199cae5734e83970ef006667b3d2e534b15e22321b96acdcd2689a not found: ID does not exist"
Oct 30 13:42:31.788249 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:31.788193    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a8d3b8-7f9bfff7bc-hszb2"]
Oct 30 13:42:31.808637 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:31.808577    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a8d3b8-7f9bfff7bc-hszb2"]
Oct 30 13:42:32.971283 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:32.971252    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="213f7500-6291-40df-8a5e-c7008a6ccc47" path="/var/lib/kubelet/pods/213f7500-6291-40df-8a5e-c7008a6ccc47/volumes"
Oct 30 13:42:35.763586 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:35.763539    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf3c79-7846779cd-pmvpz"]
Oct 30 13:42:35.764060 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:35.763795    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf3c79-7846779cd-pmvpz" podUID="966c9a37-1b20-416b-aa28-c9703fca7fbe" containerName="maas-api" containerID="cri-o://3a514d1ecd07faf5726c8706cc5ebec9523625416c362bd58b6b0fc893f761f6" gracePeriod=30
Oct 30 13:42:36.064132 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:36.064106    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf3c79-7846779cd-pmvpz"
Oct 30 13:42:36.260305 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:36.260272    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/966c9a37-1b20-416b-aa28-c9703fca7fbe-metrics-certs\" (UniqueName: \"kubernetes.io/secret/966c9a37-1b20-416b-aa28-c9703fca7fbe-metrics-certs\") pod \"966c9a37-1b20-416b-aa28-c9703fca7fbe\" (UID: \"966c9a37-1b20-416b-aa28-c9703fca7fbe\") "
Oct 30 13:42:36.260525 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:36.260323    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/966c9a37-1b20-416b-aa28-c9703fca7fbe-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/966c9a37-1b20-416b-aa28-c9703fca7fbe-openshift-service-ca\") pod \"966c9a37-1b20-416b-aa28-c9703fca7fbe\" (UID: \"966c9a37-1b20-416b-aa28-c9703fca7fbe\") "
Oct 30 13:42:36.260525 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:36.260351    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/966c9a37-1b20-416b-aa28-c9703fca7fbe-kube-api-access-qmlzz\" (UniqueName: \"kubernetes.io/projected/966c9a37-1b20-416b-aa28-c9703fca7fbe-kube-api-access-qmlzz\") pod \"966c9a37-1b20-416b-aa28-c9703fca7fbe\" (UID: \"966c9a37-1b20-416b-aa28-c9703fca7fbe\") "
Oct 30 13:42:36.260525 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:36.260407    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/966c9a37-1b20-416b-aa28-c9703fca7fbe-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/966c9a37-1b20-416b-aa28-c9703fca7fbe-maas-api-tls\") pod \"966c9a37-1b20-416b-aa28-c9703fca7fbe\" (UID: \"966c9a37-1b20-416b-aa28-c9703fca7fbe\") "
Oct 30 13:42:36.260738 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:36.260709    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/966c9a37-1b20-416b-aa28-c9703fca7fbe-openshift-service-ca" pod "966c9a37-1b20-416b-aa28-c9703fca7fbe" (UID: "966c9a37-1b20-416b-aa28-c9703fca7fbe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:42:36.260982 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:36.260952    4364 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/966c9a37-1b20-416b-aa28-c9703fca7fbe-openshift-service-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:42:36.265753 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:36.263906    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/966c9a37-1b20-416b-aa28-c9703fca7fbe-kube-api-access-qmlzz" pod "966c9a37-1b20-416b-aa28-c9703fca7fbe" (UID: "966c9a37-1b20-416b-aa28-c9703fca7fbe"). InnerVolumeSpecName "kube-api-access-qmlzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:42:36.265884 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:36.265852    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/966c9a37-1b20-416b-aa28-c9703fca7fbe-metrics-certs" pod "966c9a37-1b20-416b-aa28-c9703fca7fbe" (UID: "966c9a37-1b20-416b-aa28-c9703fca7fbe"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:42:36.267470 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:36.266937    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/966c9a37-1b20-416b-aa28-c9703fca7fbe-maas-api-tls" pod "966c9a37-1b20-416b-aa28-c9703fca7fbe" (UID: "966c9a37-1b20-416b-aa28-c9703fca7fbe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:42:36.381619 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:36.381001    4364 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/966c9a37-1b20-416b-aa28-c9703fca7fbe-metrics-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:42:36.381619 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:36.381042    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmlzz\" (UniqueName: \"kubernetes.io/projected/966c9a37-1b20-416b-aa28-c9703fca7fbe-kube-api-access-qmlzz\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:42:36.381619 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:36.381060    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/966c9a37-1b20-416b-aa28-c9703fca7fbe-maas-api-tls\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:42:36.676646 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:36.674519    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8"]
Oct 30 13:42:36.676646 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:36.674753    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8" podUID="fe89b812-7b37-4eeb-8a92-07c436511948" containerName="istio-proxy" containerID="cri-o://3492a594c7962768b036eda766bd56685da25d669c7a51c59d6964a6c22d878b" gracePeriod=30
Oct 30 13:42:36.726573 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:36.726541    4364 generic.go:356] "Generic (PLEG): container finished" podID="966c9a37-1b20-416b-aa28-c9703fca7fbe" containerID="3a514d1ecd07faf5726c8706cc5ebec9523625416c362bd58b6b0fc893f761f6" exitCode=0
Oct 30 13:42:36.726765 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:36.726588    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf3c79-7846779cd-pmvpz"
Oct 30 13:42:36.726765 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:36.726627    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf3c79-7846779cd-pmvpz" event={"ID":"966c9a37-1b20-416b-aa28-c9703fca7fbe","Type":"ContainerDied","Data":"3a514d1ecd07faf5726c8706cc5ebec9523625416c362bd58b6b0fc893f761f6"}
Oct 30 13:42:36.726765 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:36.726688    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf3c79-7846779cd-pmvpz" event={"ID":"966c9a37-1b20-416b-aa28-c9703fca7fbe","Type":"ContainerDied","Data":"82bb8da453afd6703ebb7d6d065c5fcaf52e14ef102230914da847255497e3d9"}
Oct 30 13:42:36.726765 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:36.726715    4364 scope.go:122] "RemoveContainer" containerID="3a514d1ecd07faf5726c8706cc5ebec9523625416c362bd58b6b0fc893f761f6"
Oct 30 13:42:36.739194 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:36.739178    4364 scope.go:122] "RemoveContainer" containerID="3a514d1ecd07faf5726c8706cc5ebec9523625416c362bd58b6b0fc893f761f6"
Oct 30 13:42:36.739425 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:42:36.739401    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a514d1ecd07faf5726c8706cc5ebec9523625416c362bd58b6b0fc893f761f6\": container with ID starting with 3a514d1ecd07faf5726c8706cc5ebec9523625416c362bd58b6b0fc893f761f6 not found: ID does not exist" containerID="3a514d1ecd07faf5726c8706cc5ebec9523625416c362bd58b6b0fc893f761f6"
Oct 30 13:42:36.739531 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:36.739429    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a514d1ecd07faf5726c8706cc5ebec9523625416c362bd58b6b0fc893f761f6"} err="failed to get container status \"3a514d1ecd07faf5726c8706cc5ebec9523625416c362bd58b6b0fc893f761f6\": rpc error: code = NotFound desc = could not find container \"3a514d1ecd07faf5726c8706cc5ebec9523625416c362bd58b6b0fc893f761f6\": container with ID starting with 3a514d1ecd07faf5726c8706cc5ebec9523625416c362bd58b6b0fc893f761f6 not found: ID does not exist"
Oct 30 13:42:36.885625 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:36.881781    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf3c79-7846779cd-pmvpz"]
Oct 30 13:42:36.905621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:36.903222    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf3c79-7846779cd-pmvpz"]
Oct 30 13:42:36.974676 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:36.974178    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="966c9a37-1b20-416b-aa28-c9703fca7fbe" path="/var/lib/kubelet/pods/966c9a37-1b20-416b-aa28-c9703fca7fbe/volumes"
Oct 30 13:42:41.507745 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:42:41.507708    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:42:41.799069 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:41.799033    4364 generic.go:356] "Generic (PLEG): container finished" podID="fe89b812-7b37-4eeb-8a92-07c436511948" containerID="3492a594c7962768b036eda766bd56685da25d669c7a51c59d6964a6c22d878b" exitCode=0
Oct 30 13:42:41.799185 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:41.799099    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8" event={"ID":"fe89b812-7b37-4eeb-8a92-07c436511948","Type":"ContainerDied","Data":"3492a594c7962768b036eda766bd56685da25d669c7a51c59d6964a6c22d878b"}
Oct 30 13:42:42.151490 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:42.151463    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8"
Oct 30 13:42:42.318537 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:42.318501    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe89b812-7b37-4eeb-8a92-07c436511948-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe89b812-7b37-4eeb-8a92-07c436511948-workload-socket\") pod \"fe89b812-7b37-4eeb-8a92-07c436511948\" (UID: \"fe89b812-7b37-4eeb-8a92-07c436511948\") "
Oct 30 13:42:42.318764 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:42.318567    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe89b812-7b37-4eeb-8a92-07c436511948-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe89b812-7b37-4eeb-8a92-07c436511948-istio-envoy\") pod \"fe89b812-7b37-4eeb-8a92-07c436511948\" (UID: \"fe89b812-7b37-4eeb-8a92-07c436511948\") "
Oct 30 13:42:42.318764 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:42.318641    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe89b812-7b37-4eeb-8a92-07c436511948-kube-api-access-sfvss\" (UniqueName: \"kubernetes.io/projected/fe89b812-7b37-4eeb-8a92-07c436511948-kube-api-access-sfvss\") pod \"fe89b812-7b37-4eeb-8a92-07c436511948\" (UID: \"fe89b812-7b37-4eeb-8a92-07c436511948\") "
Oct 30 13:42:42.318764 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:42.318698    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe89b812-7b37-4eeb-8a92-07c436511948-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe89b812-7b37-4eeb-8a92-07c436511948-workload-certs\") pod \"fe89b812-7b37-4eeb-8a92-07c436511948\" (UID: \"fe89b812-7b37-4eeb-8a92-07c436511948\") "
Oct 30 13:42:42.318953 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:42.318767    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fe89b812-7b37-4eeb-8a92-07c436511948-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe89b812-7b37-4eeb-8a92-07c436511948-istio-podinfo\") pod \"fe89b812-7b37-4eeb-8a92-07c436511948\" (UID: \"fe89b812-7b37-4eeb-8a92-07c436511948\") "
Oct 30 13:42:42.318953 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:42.318779    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe89b812-7b37-4eeb-8a92-07c436511948-workload-socket" pod "fe89b812-7b37-4eeb-8a92-07c436511948" (UID: "fe89b812-7b37-4eeb-8a92-07c436511948"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:42:42.318953 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:42.318798    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe89b812-7b37-4eeb-8a92-07c436511948-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe89b812-7b37-4eeb-8a92-07c436511948-credential-socket\") pod \"fe89b812-7b37-4eeb-8a92-07c436511948\" (UID: \"fe89b812-7b37-4eeb-8a92-07c436511948\") "
Oct 30 13:42:42.318953 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:42.318898    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe89b812-7b37-4eeb-8a92-07c436511948-istio-token\" (UniqueName: \"kubernetes.io/projected/fe89b812-7b37-4eeb-8a92-07c436511948-istio-token\") pod \"fe89b812-7b37-4eeb-8a92-07c436511948\" (UID: \"fe89b812-7b37-4eeb-8a92-07c436511948\") "
Oct 30 13:42:42.318953 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:42.318940    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe89b812-7b37-4eeb-8a92-07c436511948-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe89b812-7b37-4eeb-8a92-07c436511948-istio-data\") pod \"fe89b812-7b37-4eeb-8a92-07c436511948\" (UID: \"fe89b812-7b37-4eeb-8a92-07c436511948\") "
Oct 30 13:42:42.319225 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:42.318991    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe89b812-7b37-4eeb-8a92-07c436511948-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe89b812-7b37-4eeb-8a92-07c436511948-istiod-ca-cert\") pod \"fe89b812-7b37-4eeb-8a92-07c436511948\" (UID: \"fe89b812-7b37-4eeb-8a92-07c436511948\") "
Oct 30 13:42:42.319225 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:42.319025    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe89b812-7b37-4eeb-8a92-07c436511948-workload-certs" pod "fe89b812-7b37-4eeb-8a92-07c436511948" (UID: "fe89b812-7b37-4eeb-8a92-07c436511948"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:42:42.319225 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:42.319062    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe89b812-7b37-4eeb-8a92-07c436511948-credential-socket" pod "fe89b812-7b37-4eeb-8a92-07c436511948" (UID: "fe89b812-7b37-4eeb-8a92-07c436511948"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:42:42.319225 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:42.319191    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe89b812-7b37-4eeb-8a92-07c436511948-istio-data" pod "fe89b812-7b37-4eeb-8a92-07c436511948" (UID: "fe89b812-7b37-4eeb-8a92-07c436511948"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:42:42.319430 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:42.319407    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe89b812-7b37-4eeb-8a92-07c436511948-istiod-ca-cert" pod "fe89b812-7b37-4eeb-8a92-07c436511948" (UID: "fe89b812-7b37-4eeb-8a92-07c436511948"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:42:42.319773 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:42.319752    4364 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe89b812-7b37-4eeb-8a92-07c436511948-workload-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:42:42.319860 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:42.319779    4364 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe89b812-7b37-4eeb-8a92-07c436511948-workload-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:42:42.319860 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:42.319794    4364 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe89b812-7b37-4eeb-8a92-07c436511948-credential-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:42:42.319860 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:42.319808    4364 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe89b812-7b37-4eeb-8a92-07c436511948-istio-data\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:42:42.319860 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:42.319822    4364 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe89b812-7b37-4eeb-8a92-07c436511948-istiod-ca-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:42:42.320958 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:42.320939    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe89b812-7b37-4eeb-8a92-07c436511948-istio-podinfo" pod "fe89b812-7b37-4eeb-8a92-07c436511948" (UID: "fe89b812-7b37-4eeb-8a92-07c436511948"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:42:42.321019 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:42.320997    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe89b812-7b37-4eeb-8a92-07c436511948-istio-token" pod "fe89b812-7b37-4eeb-8a92-07c436511948" (UID: "fe89b812-7b37-4eeb-8a92-07c436511948"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:42:42.321214 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:42.321198    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe89b812-7b37-4eeb-8a92-07c436511948-istio-envoy" pod "fe89b812-7b37-4eeb-8a92-07c436511948" (UID: "fe89b812-7b37-4eeb-8a92-07c436511948"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:42:42.321257 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:42.321212    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe89b812-7b37-4eeb-8a92-07c436511948-kube-api-access-sfvss" pod "fe89b812-7b37-4eeb-8a92-07c436511948" (UID: "fe89b812-7b37-4eeb-8a92-07c436511948"). InnerVolumeSpecName "kube-api-access-sfvss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:42:42.421457 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:42.421431    4364 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe89b812-7b37-4eeb-8a92-07c436511948-istio-envoy\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:42:42.421457 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:42.421455    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfvss\" (UniqueName: \"kubernetes.io/projected/fe89b812-7b37-4eeb-8a92-07c436511948-kube-api-access-sfvss\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:42:42.421593 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:42.421466    4364 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe89b812-7b37-4eeb-8a92-07c436511948-istio-podinfo\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:42:42.421593 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:42.421474    4364 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe89b812-7b37-4eeb-8a92-07c436511948-istio-token\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:42:42.808831 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:42.808802    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8"
Oct 30 13:42:42.809263 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:42.808811    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8" event={"ID":"fe89b812-7b37-4eeb-8a92-07c436511948","Type":"ContainerDied","Data":"3fcb14336bda388d96c9382db877e88e3eb3ccd6537a13bc8eb432e06bcfa1b4"}
Oct 30 13:42:42.809263 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:42.808942    4364 scope.go:122] "RemoveContainer" containerID="3492a594c7962768b036eda766bd56685da25d669c7a51c59d6964a6c22d878b"
Oct 30 13:42:42.877283 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:42.877256    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8"]
Oct 30 13:42:42.889700 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:42.889674    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xtpc8"]
Oct 30 13:42:42.970890 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:42.970836    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe89b812-7b37-4eeb-8a92-07c436511948" path="/var/lib/kubelet/pods/fe89b812-7b37-4eeb-8a92-07c436511948/volumes"
Oct 30 13:42:47.300221 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:47.300188    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-78dc89-797f8b64d4-rw7rg"]
Oct 30 13:42:47.300742 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:47.300435    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78dc89-797f8b64d4-rw7rg" podUID="94f6ed0a-3007-4d3a-a569-920348b7da49" containerName="maas-api" containerID="cri-o://e3a020c85c35e381f39b31651d4cec000281abb28698bc1c59c5c6fd925f6372" gracePeriod=30
Oct 30 13:42:47.606309 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:47.606279    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78dc89-797f8b64d4-rw7rg"
Oct 30 13:42:47.698499 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:47.698463    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94f6ed0a-3007-4d3a-a569-920348b7da49-metrics-certs\" (UniqueName: \"kubernetes.io/secret/94f6ed0a-3007-4d3a-a569-920348b7da49-metrics-certs\") pod \"94f6ed0a-3007-4d3a-a569-920348b7da49\" (UID: \"94f6ed0a-3007-4d3a-a569-920348b7da49\") "
Oct 30 13:42:47.699138 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:47.698656    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94f6ed0a-3007-4d3a-a569-920348b7da49-kube-api-access-7smr6\" (UniqueName: \"kubernetes.io/projected/94f6ed0a-3007-4d3a-a569-920348b7da49-kube-api-access-7smr6\") pod \"94f6ed0a-3007-4d3a-a569-920348b7da49\" (UID: \"94f6ed0a-3007-4d3a-a569-920348b7da49\") "
Oct 30 13:42:47.699138 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:47.698787    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94f6ed0a-3007-4d3a-a569-920348b7da49-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94f6ed0a-3007-4d3a-a569-920348b7da49-maas-api-tls\") pod \"94f6ed0a-3007-4d3a-a569-920348b7da49\" (UID: \"94f6ed0a-3007-4d3a-a569-920348b7da49\") "
Oct 30 13:42:47.699138 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:47.698969    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94f6ed0a-3007-4d3a-a569-920348b7da49-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94f6ed0a-3007-4d3a-a569-920348b7da49-openshift-service-ca\") pod \"94f6ed0a-3007-4d3a-a569-920348b7da49\" (UID: \"94f6ed0a-3007-4d3a-a569-920348b7da49\") "
Oct 30 13:42:47.699504 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:47.699481    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94f6ed0a-3007-4d3a-a569-920348b7da49-openshift-service-ca" pod "94f6ed0a-3007-4d3a-a569-920348b7da49" (UID: "94f6ed0a-3007-4d3a-a569-920348b7da49"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:42:47.699807 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:47.699782    4364 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94f6ed0a-3007-4d3a-a569-920348b7da49-openshift-service-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:42:47.701369 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:47.701345    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94f6ed0a-3007-4d3a-a569-920348b7da49-metrics-certs" pod "94f6ed0a-3007-4d3a-a569-920348b7da49" (UID: "94f6ed0a-3007-4d3a-a569-920348b7da49"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:42:47.701591 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:47.701573    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94f6ed0a-3007-4d3a-a569-920348b7da49-maas-api-tls" pod "94f6ed0a-3007-4d3a-a569-920348b7da49" (UID: "94f6ed0a-3007-4d3a-a569-920348b7da49"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:42:47.701898 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:47.701869    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94f6ed0a-3007-4d3a-a569-920348b7da49-kube-api-access-7smr6" pod "94f6ed0a-3007-4d3a-a569-920348b7da49" (UID: "94f6ed0a-3007-4d3a-a569-920348b7da49"). InnerVolumeSpecName "kube-api-access-7smr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:42:47.801335 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:47.801305    4364 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94f6ed0a-3007-4d3a-a569-920348b7da49-metrics-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:42:47.801446 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:47.801337    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7smr6\" (UniqueName: \"kubernetes.io/projected/94f6ed0a-3007-4d3a-a569-920348b7da49-kube-api-access-7smr6\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:42:47.801446 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:47.801364    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94f6ed0a-3007-4d3a-a569-920348b7da49-maas-api-tls\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:42:48.049635 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:48.049582    4364 generic.go:356] "Generic (PLEG): container finished" podID="94f6ed0a-3007-4d3a-a569-920348b7da49" containerID="e3a020c85c35e381f39b31651d4cec000281abb28698bc1c59c5c6fd925f6372" exitCode=0
Oct 30 13:42:48.049939 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:48.049749    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78dc89-797f8b64d4-rw7rg"
Oct 30 13:42:48.050151 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:48.049769    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78dc89-797f8b64d4-rw7rg" event={"ID":"94f6ed0a-3007-4d3a-a569-920348b7da49","Type":"ContainerDied","Data":"e3a020c85c35e381f39b31651d4cec000281abb28698bc1c59c5c6fd925f6372"}
Oct 30 13:42:48.050311 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:48.050293    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-78dc89-797f8b64d4-rw7rg" event={"ID":"94f6ed0a-3007-4d3a-a569-920348b7da49","Type":"ContainerDied","Data":"7e10ff88512ce729f2347d0b962cc9362049a5f07e1edead50c4ad3d192020b3"}
Oct 30 13:42:48.050470 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:48.050433    4364 scope.go:122] "RemoveContainer" containerID="e3a020c85c35e381f39b31651d4cec000281abb28698bc1c59c5c6fd925f6372"
Oct 30 13:42:48.067944 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:48.067924    4364 scope.go:122] "RemoveContainer" containerID="e3a020c85c35e381f39b31651d4cec000281abb28698bc1c59c5c6fd925f6372"
Oct 30 13:42:48.068279 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:42:48.068258    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3a020c85c35e381f39b31651d4cec000281abb28698bc1c59c5c6fd925f6372\": container with ID starting with e3a020c85c35e381f39b31651d4cec000281abb28698bc1c59c5c6fd925f6372 not found: ID does not exist" containerID="e3a020c85c35e381f39b31651d4cec000281abb28698bc1c59c5c6fd925f6372"
Oct 30 13:42:48.068362 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:48.068289    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3a020c85c35e381f39b31651d4cec000281abb28698bc1c59c5c6fd925f6372"} err="failed to get container status \"e3a020c85c35e381f39b31651d4cec000281abb28698bc1c59c5c6fd925f6372\": rpc error: code = NotFound desc = could not find container \"e3a020c85c35e381f39b31651d4cec000281abb28698bc1c59c5c6fd925f6372\": container with ID starting with e3a020c85c35e381f39b31651d4cec000281abb28698bc1c59c5c6fd925f6372 not found: ID does not exist"
Oct 30 13:42:48.107732 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:48.107703    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-78dc89-797f8b64d4-rw7rg"]
Oct 30 13:42:48.115413 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:48.115386    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-78dc89-797f8b64d4-rw7rg"]
Oct 30 13:42:48.971805 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:48.971770    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94f6ed0a-3007-4d3a-a569-920348b7da49" path="/var/lib/kubelet/pods/94f6ed0a-3007-4d3a-a569-920348b7da49/volumes"
Oct 30 13:42:56.531178 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:56.531140    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6"]
Oct 30 13:42:56.531719 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:42:56.531352    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6" podUID="864a2c8d-97ba-4141-957c-0810628ba6ec" containerName="istio-proxy" containerID="cri-o://6a6aefb24e9046bca7c428edea73a3cde0547e75288bd6475cf1ad8eef9f3be9" gracePeriod=30
Oct 30 13:43:00.264759 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:00.264725    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-b4fb25/ipp-route-b4fb25-kserve-5d46888457-25qdx"]
Oct 30 13:43:01.813033 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:01.812965    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6"
Oct 30 13:43:01.858194 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:01.858161    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/864a2c8d-97ba-4141-957c-0810628ba6ec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/864a2c8d-97ba-4141-957c-0810628ba6ec-istio-podinfo\") pod \"864a2c8d-97ba-4141-957c-0810628ba6ec\" (UID: \"864a2c8d-97ba-4141-957c-0810628ba6ec\") "
Oct 30 13:43:01.858385 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:01.858261    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/864a2c8d-97ba-4141-957c-0810628ba6ec-istio-token\" (UniqueName: \"kubernetes.io/projected/864a2c8d-97ba-4141-957c-0810628ba6ec-istio-token\") pod \"864a2c8d-97ba-4141-957c-0810628ba6ec\" (UID: \"864a2c8d-97ba-4141-957c-0810628ba6ec\") "
Oct 30 13:43:01.858385 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:01.858360    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/864a2c8d-97ba-4141-957c-0810628ba6ec-kube-api-access-xdscw\" (UniqueName: \"kubernetes.io/projected/864a2c8d-97ba-4141-957c-0810628ba6ec-kube-api-access-xdscw\") pod \"864a2c8d-97ba-4141-957c-0810628ba6ec\" (UID: \"864a2c8d-97ba-4141-957c-0810628ba6ec\") "
Oct 30 13:43:01.858505 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:01.858492    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/864a2c8d-97ba-4141-957c-0810628ba6ec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/864a2c8d-97ba-4141-957c-0810628ba6ec-istio-envoy\") pod \"864a2c8d-97ba-4141-957c-0810628ba6ec\" (UID: \"864a2c8d-97ba-4141-957c-0810628ba6ec\") "
Oct 30 13:43:01.858614 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:01.858568    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/864a2c8d-97ba-4141-957c-0810628ba6ec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/864a2c8d-97ba-4141-957c-0810628ba6ec-istiod-ca-cert\") pod \"864a2c8d-97ba-4141-957c-0810628ba6ec\" (UID: \"864a2c8d-97ba-4141-957c-0810628ba6ec\") "
Oct 30 13:43:01.858702 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:01.858648    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/864a2c8d-97ba-4141-957c-0810628ba6ec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/864a2c8d-97ba-4141-957c-0810628ba6ec-workload-socket\") pod \"864a2c8d-97ba-4141-957c-0810628ba6ec\" (UID: \"864a2c8d-97ba-4141-957c-0810628ba6ec\") "
Oct 30 13:43:01.858762 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:01.858712    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/864a2c8d-97ba-4141-957c-0810628ba6ec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/864a2c8d-97ba-4141-957c-0810628ba6ec-credential-socket\") pod \"864a2c8d-97ba-4141-957c-0810628ba6ec\" (UID: \"864a2c8d-97ba-4141-957c-0810628ba6ec\") "
Oct 30 13:43:01.858821 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:01.858766    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/864a2c8d-97ba-4141-957c-0810628ba6ec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/864a2c8d-97ba-4141-957c-0810628ba6ec-istio-data\") pod \"864a2c8d-97ba-4141-957c-0810628ba6ec\" (UID: \"864a2c8d-97ba-4141-957c-0810628ba6ec\") "
Oct 30 13:43:01.858821 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:01.858812    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/864a2c8d-97ba-4141-957c-0810628ba6ec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/864a2c8d-97ba-4141-957c-0810628ba6ec-workload-certs\") pod \"864a2c8d-97ba-4141-957c-0810628ba6ec\" (UID: \"864a2c8d-97ba-4141-957c-0810628ba6ec\") "
Oct 30 13:43:01.859184 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:01.859085    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/864a2c8d-97ba-4141-957c-0810628ba6ec-credential-socket" pod "864a2c8d-97ba-4141-957c-0810628ba6ec" (UID: "864a2c8d-97ba-4141-957c-0810628ba6ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:43:01.859329 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:01.859185    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/864a2c8d-97ba-4141-957c-0810628ba6ec-istiod-ca-cert" pod "864a2c8d-97ba-4141-957c-0810628ba6ec" (UID: "864a2c8d-97ba-4141-957c-0810628ba6ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:43:01.859412 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:01.859349    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/864a2c8d-97ba-4141-957c-0810628ba6ec-workload-certs" pod "864a2c8d-97ba-4141-957c-0810628ba6ec" (UID: "864a2c8d-97ba-4141-957c-0810628ba6ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:43:01.859412 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:01.859367    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/864a2c8d-97ba-4141-957c-0810628ba6ec-workload-socket" pod "864a2c8d-97ba-4141-957c-0810628ba6ec" (UID: "864a2c8d-97ba-4141-957c-0810628ba6ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:43:01.859412 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:01.859377    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/864a2c8d-97ba-4141-957c-0810628ba6ec-istio-data" pod "864a2c8d-97ba-4141-957c-0810628ba6ec" (UID: "864a2c8d-97ba-4141-957c-0810628ba6ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:43:01.859883 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:01.859858    4364 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/864a2c8d-97ba-4141-957c-0810628ba6ec-istiod-ca-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:43:01.859992 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:01.859889    4364 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/864a2c8d-97ba-4141-957c-0810628ba6ec-workload-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:43:01.859992 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:01.859903    4364 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/864a2c8d-97ba-4141-957c-0810628ba6ec-credential-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:43:01.859992 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:01.859917    4364 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/864a2c8d-97ba-4141-957c-0810628ba6ec-istio-data\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:43:01.859992 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:01.859930    4364 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/864a2c8d-97ba-4141-957c-0810628ba6ec-workload-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:43:01.860917 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:01.860764    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/864a2c8d-97ba-4141-957c-0810628ba6ec-istio-podinfo" pod "864a2c8d-97ba-4141-957c-0810628ba6ec" (UID: "864a2c8d-97ba-4141-957c-0810628ba6ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:43:01.860917 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:01.860855    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/864a2c8d-97ba-4141-957c-0810628ba6ec-istio-envoy" pod "864a2c8d-97ba-4141-957c-0810628ba6ec" (UID: "864a2c8d-97ba-4141-957c-0810628ba6ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:43:01.860917 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:01.860876    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/864a2c8d-97ba-4141-957c-0810628ba6ec-kube-api-access-xdscw" pod "864a2c8d-97ba-4141-957c-0810628ba6ec" (UID: "864a2c8d-97ba-4141-957c-0810628ba6ec"). InnerVolumeSpecName "kube-api-access-xdscw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:43:01.861261 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:01.861245    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/864a2c8d-97ba-4141-957c-0810628ba6ec-istio-token" pod "864a2c8d-97ba-4141-957c-0810628ba6ec" (UID: "864a2c8d-97ba-4141-957c-0810628ba6ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:43:01.961722 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:01.961682    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdscw\" (UniqueName: \"kubernetes.io/projected/864a2c8d-97ba-4141-957c-0810628ba6ec-kube-api-access-xdscw\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:43:01.961722 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:01.961718    4364 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/864a2c8d-97ba-4141-957c-0810628ba6ec-istio-envoy\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:43:01.961722 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:01.961728    4364 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/864a2c8d-97ba-4141-957c-0810628ba6ec-istio-podinfo\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:43:01.961978 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:01.961735    4364 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/864a2c8d-97ba-4141-957c-0810628ba6ec-istio-token\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:43:02.157519 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:02.157483    4364 generic.go:356] "Generic (PLEG): container finished" podID="864a2c8d-97ba-4141-957c-0810628ba6ec" containerID="6a6aefb24e9046bca7c428edea73a3cde0547e75288bd6475cf1ad8eef9f3be9" exitCode=0
Oct 30 13:43:02.157725 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:02.157526    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6" event={"ID":"864a2c8d-97ba-4141-957c-0810628ba6ec","Type":"ContainerDied","Data":"6a6aefb24e9046bca7c428edea73a3cde0547e75288bd6475cf1ad8eef9f3be9"}
Oct 30 13:43:02.157725 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:02.157544    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6"
Oct 30 13:43:02.157725 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:02.157575    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6" event={"ID":"864a2c8d-97ba-4141-957c-0810628ba6ec","Type":"ContainerDied","Data":"13dd9e002f40ad251dc8ce918b872fb578a03ef0879a1394f6e4c1c337de20ca"}
Oct 30 13:43:02.157725 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:02.157613    4364 scope.go:122] "RemoveContainer" containerID="6a6aefb24e9046bca7c428edea73a3cde0547e75288bd6475cf1ad8eef9f3be9"
Oct 30 13:43:02.170518 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:02.170501    4364 scope.go:122] "RemoveContainer" containerID="385f1fb76fa6f0e60e13865dfc171e43478952e68c76add876ed08e5bbdfe068"
Oct 30 13:43:02.180562 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:02.180544    4364 scope.go:122] "RemoveContainer" containerID="6a6aefb24e9046bca7c428edea73a3cde0547e75288bd6475cf1ad8eef9f3be9"
Oct 30 13:43:02.180838 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:43:02.180815    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a6aefb24e9046bca7c428edea73a3cde0547e75288bd6475cf1ad8eef9f3be9\": container with ID starting with 6a6aefb24e9046bca7c428edea73a3cde0547e75288bd6475cf1ad8eef9f3be9 not found: ID does not exist" containerID="6a6aefb24e9046bca7c428edea73a3cde0547e75288bd6475cf1ad8eef9f3be9"
Oct 30 13:43:02.180896 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:02.180844    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a6aefb24e9046bca7c428edea73a3cde0547e75288bd6475cf1ad8eef9f3be9"} err="failed to get container status \"6a6aefb24e9046bca7c428edea73a3cde0547e75288bd6475cf1ad8eef9f3be9\": rpc error: code = NotFound desc = could not find container \"6a6aefb24e9046bca7c428edea73a3cde0547e75288bd6475cf1ad8eef9f3be9\": container with ID starting with 6a6aefb24e9046bca7c428edea73a3cde0547e75288bd6475cf1ad8eef9f3be9 not found: ID does not exist"
Oct 30 13:43:02.180896 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:02.180859    4364 scope.go:122] "RemoveContainer" containerID="385f1fb76fa6f0e60e13865dfc171e43478952e68c76add876ed08e5bbdfe068"
Oct 30 13:43:02.181062 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:43:02.181047    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"385f1fb76fa6f0e60e13865dfc171e43478952e68c76add876ed08e5bbdfe068\": container with ID starting with 385f1fb76fa6f0e60e13865dfc171e43478952e68c76add876ed08e5bbdfe068 not found: ID does not exist" containerID="385f1fb76fa6f0e60e13865dfc171e43478952e68c76add876ed08e5bbdfe068"
Oct 30 13:43:02.181102 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:02.181066    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"385f1fb76fa6f0e60e13865dfc171e43478952e68c76add876ed08e5bbdfe068"} err="failed to get container status \"385f1fb76fa6f0e60e13865dfc171e43478952e68c76add876ed08e5bbdfe068\": rpc error: code = NotFound desc = could not find container \"385f1fb76fa6f0e60e13865dfc171e43478952e68c76add876ed08e5bbdfe068\": container with ID starting with 385f1fb76fa6f0e60e13865dfc171e43478952e68c76add876ed08e5bbdfe068 not found: ID does not exist"
Oct 30 13:43:02.212611 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:02.212576    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6"]
Oct 30 13:43:02.219826 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:02.219801    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-78dc89-openshift-default-8474445c45-zjsh6"]
Oct 30 13:43:02.980397 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:02.980363    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="864a2c8d-97ba-4141-957c-0810628ba6ec" path="/var/lib/kubelet/pods/864a2c8d-97ba-4141-957c-0810628ba6ec/volumes"
Oct 30 13:43:11.508182 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:43:11.508145    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:43:15.286681 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:15.286635    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-b4fb25/ipp-route-b4fb25-kserve-5d46888457-25qdx" podUID="e77381ab-005b-4b76-81b0-ada11566adca" containerName="main" containerID="cri-o://878db7143ceb0dea950ced2298052a25023c1bb671c22e6967424f5e335a500d" gracePeriod=45
Oct 30 13:43:15.560565 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:15.560539    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-b4fb25/ipp-route-b4fb25-kserve-5d46888457-25qdx"
Oct 30 13:43:15.690960 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:15.690929    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e77381ab-005b-4b76-81b0-ada11566adca-tls-certs\" (UniqueName: \"kubernetes.io/secret/e77381ab-005b-4b76-81b0-ada11566adca-tls-certs\") pod \"e77381ab-005b-4b76-81b0-ada11566adca\" (UID: \"e77381ab-005b-4b76-81b0-ada11566adca\") "
Oct 30 13:43:15.691134 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:15.691040    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e77381ab-005b-4b76-81b0-ada11566adca-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e77381ab-005b-4b76-81b0-ada11566adca-model-cache\") pod \"e77381ab-005b-4b76-81b0-ada11566adca\" (UID: \"e77381ab-005b-4b76-81b0-ada11566adca\") "
Oct 30 13:43:15.691134 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:15.691111    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e77381ab-005b-4b76-81b0-ada11566adca-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e77381ab-005b-4b76-81b0-ada11566adca-dshm\") pod \"e77381ab-005b-4b76-81b0-ada11566adca\" (UID: \"e77381ab-005b-4b76-81b0-ada11566adca\") "
Oct 30 13:43:15.691252 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:15.691169    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e77381ab-005b-4b76-81b0-ada11566adca-kube-api-access-wvnr4\" (UniqueName: \"kubernetes.io/projected/e77381ab-005b-4b76-81b0-ada11566adca-kube-api-access-wvnr4\") pod \"e77381ab-005b-4b76-81b0-ada11566adca\" (UID: \"e77381ab-005b-4b76-81b0-ada11566adca\") "
Oct 30 13:43:15.691252 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:15.691219    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e77381ab-005b-4b76-81b0-ada11566adca-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e77381ab-005b-4b76-81b0-ada11566adca-tmp-dir\") pod \"e77381ab-005b-4b76-81b0-ada11566adca\" (UID: \"e77381ab-005b-4b76-81b0-ada11566adca\") "
Oct 30 13:43:15.691371 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:15.691303    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e77381ab-005b-4b76-81b0-ada11566adca-model-cache" pod "e77381ab-005b-4b76-81b0-ada11566adca" (UID: "e77381ab-005b-4b76-81b0-ada11566adca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:43:15.691371 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:15.691322    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e77381ab-005b-4b76-81b0-ada11566adca-home\" (UniqueName: \"kubernetes.io/empty-dir/e77381ab-005b-4b76-81b0-ada11566adca-home\") pod \"e77381ab-005b-4b76-81b0-ada11566adca\" (UID: \"e77381ab-005b-4b76-81b0-ada11566adca\") "
Oct 30 13:43:15.691630 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:15.691560    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e77381ab-005b-4b76-81b0-ada11566adca-tmp-dir" pod "e77381ab-005b-4b76-81b0-ada11566adca" (UID: "e77381ab-005b-4b76-81b0-ada11566adca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:43:15.691737 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:15.691636    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e77381ab-005b-4b76-81b0-ada11566adca-home" pod "e77381ab-005b-4b76-81b0-ada11566adca" (UID: "e77381ab-005b-4b76-81b0-ada11566adca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:43:15.692086 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:15.692058    4364 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e77381ab-005b-4b76-81b0-ada11566adca-model-cache\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:43:15.692201 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:15.692115    4364 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e77381ab-005b-4b76-81b0-ada11566adca-tmp-dir\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:43:15.692201 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:15.692130    4364 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e77381ab-005b-4b76-81b0-ada11566adca-home\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:43:15.693475 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:15.693437    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e77381ab-005b-4b76-81b0-ada11566adca-tls-certs" pod "e77381ab-005b-4b76-81b0-ada11566adca" (UID: "e77381ab-005b-4b76-81b0-ada11566adca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:43:15.693725 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:15.693694    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e77381ab-005b-4b76-81b0-ada11566adca-dshm" pod "e77381ab-005b-4b76-81b0-ada11566adca" (UID: "e77381ab-005b-4b76-81b0-ada11566adca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:43:15.693819 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:15.693725    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e77381ab-005b-4b76-81b0-ada11566adca-kube-api-access-wvnr4" pod "e77381ab-005b-4b76-81b0-ada11566adca" (UID: "e77381ab-005b-4b76-81b0-ada11566adca"). InnerVolumeSpecName "kube-api-access-wvnr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:43:15.794165 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:15.794084    4364 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e77381ab-005b-4b76-81b0-ada11566adca-tls-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:43:15.794165 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:15.794109    4364 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e77381ab-005b-4b76-81b0-ada11566adca-dshm\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:43:15.794165 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:15.794119    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvnr4\" (UniqueName: \"kubernetes.io/projected/e77381ab-005b-4b76-81b0-ada11566adca-kube-api-access-wvnr4\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:43:16.276256 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:16.276217    4364 generic.go:356] "Generic (PLEG): container finished" podID="e77381ab-005b-4b76-81b0-ada11566adca" containerID="878db7143ceb0dea950ced2298052a25023c1bb671c22e6967424f5e335a500d" exitCode=0
Oct 30 13:43:16.276441 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:16.276286    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-b4fb25/ipp-route-b4fb25-kserve-5d46888457-25qdx"
Oct 30 13:43:16.276441 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:16.276304    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-b4fb25/ipp-route-b4fb25-kserve-5d46888457-25qdx" event={"ID":"e77381ab-005b-4b76-81b0-ada11566adca","Type":"ContainerDied","Data":"878db7143ceb0dea950ced2298052a25023c1bb671c22e6967424f5e335a500d"}
Oct 30 13:43:16.276441 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:16.276356    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-b4fb25/ipp-route-b4fb25-kserve-5d46888457-25qdx" event={"ID":"e77381ab-005b-4b76-81b0-ada11566adca","Type":"ContainerDied","Data":"b4caf1a8e9e7675e69a6aee6da3623afd9a8e84a9f13c4039e8074a8bdadfb16"}
Oct 30 13:43:16.276441 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:16.276380    4364 scope.go:122] "RemoveContainer" containerID="878db7143ceb0dea950ced2298052a25023c1bb671c22e6967424f5e335a500d"
Oct 30 13:43:16.287899 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:16.287678    4364 scope.go:122] "RemoveContainer" containerID="878db7143ceb0dea950ced2298052a25023c1bb671c22e6967424f5e335a500d"
Oct 30 13:43:16.288223 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:43:16.288022    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"878db7143ceb0dea950ced2298052a25023c1bb671c22e6967424f5e335a500d\": container with ID starting with 878db7143ceb0dea950ced2298052a25023c1bb671c22e6967424f5e335a500d not found: ID does not exist" containerID="878db7143ceb0dea950ced2298052a25023c1bb671c22e6967424f5e335a500d"
Oct 30 13:43:16.288223 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:16.288055    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"878db7143ceb0dea950ced2298052a25023c1bb671c22e6967424f5e335a500d"} err="failed to get container status \"878db7143ceb0dea950ced2298052a25023c1bb671c22e6967424f5e335a500d\": rpc error: code = NotFound desc = could not find container \"878db7143ceb0dea950ced2298052a25023c1bb671c22e6967424f5e335a500d\": container with ID starting with 878db7143ceb0dea950ced2298052a25023c1bb671c22e6967424f5e335a500d not found: ID does not exist"
Oct 30 13:43:16.333641 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:16.333608    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-b4fb25/ipp-route-b4fb25-kserve-5d46888457-25qdx"]
Oct 30 13:43:16.343819 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:16.343791    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-b4fb25/ipp-route-b4fb25-kserve-5d46888457-25qdx"]
Oct 30 13:43:16.479512 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:16.479476    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-b4fb25/ipp-route-b4fb25-kserve-5d46888457-25qdx" podUID="e77381ab-005b-4b76-81b0-ada11566adca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.254:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:43:16.971727 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:16.971692    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e77381ab-005b-4b76-81b0-ada11566adca" path="/var/lib/kubelet/pods/e77381ab-005b-4b76-81b0-ada11566adca/volumes"
Oct 30 13:43:19.182952 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:19.182917    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c47903-546d9c8468-kwdcc"]
Oct 30 13:43:19.183433 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:19.183138    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c47903-546d9c8468-kwdcc" podUID="93fc7bda-b92c-41ce-b567-658a571bc5ba" containerName="maas-api" containerID="cri-o://abbd5d9520c8fd5aa9b863c0999bbd36e684215de3a70273421b525e6ce9c7e9" gracePeriod=30
Oct 30 13:43:19.679853 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:19.679813    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c47903-546d9c8468-kwdcc"
Oct 30 13:43:19.805397 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:19.784537    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93fc7bda-b92c-41ce-b567-658a571bc5ba-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93fc7bda-b92c-41ce-b567-658a571bc5ba-openshift-service-ca\") pod \"93fc7bda-b92c-41ce-b567-658a571bc5ba\" (UID: \"93fc7bda-b92c-41ce-b567-658a571bc5ba\") "
Oct 30 13:43:19.805397 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:19.785082    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93fc7bda-b92c-41ce-b567-658a571bc5ba-openshift-service-ca" pod "93fc7bda-b92c-41ce-b567-658a571bc5ba" (UID: "93fc7bda-b92c-41ce-b567-658a571bc5ba"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:43:19.805397 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:19.785177    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93fc7bda-b92c-41ce-b567-658a571bc5ba-metrics-certs\" (UniqueName: \"kubernetes.io/secret/93fc7bda-b92c-41ce-b567-658a571bc5ba-metrics-certs\") pod \"93fc7bda-b92c-41ce-b567-658a571bc5ba\" (UID: \"93fc7bda-b92c-41ce-b567-658a571bc5ba\") "
Oct 30 13:43:19.805397 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:19.785253    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93fc7bda-b92c-41ce-b567-658a571bc5ba-kube-api-access-757mt\" (UniqueName: \"kubernetes.io/projected/93fc7bda-b92c-41ce-b567-658a571bc5ba-kube-api-access-757mt\") pod \"93fc7bda-b92c-41ce-b567-658a571bc5ba\" (UID: \"93fc7bda-b92c-41ce-b567-658a571bc5ba\") "
Oct 30 13:43:19.805397 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:19.786044    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93fc7bda-b92c-41ce-b567-658a571bc5ba-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93fc7bda-b92c-41ce-b567-658a571bc5ba-maas-api-tls\") pod \"93fc7bda-b92c-41ce-b567-658a571bc5ba\" (UID: \"93fc7bda-b92c-41ce-b567-658a571bc5ba\") "
Oct 30 13:43:19.805397 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:19.787335    4364 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93fc7bda-b92c-41ce-b567-658a571bc5ba-openshift-service-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:43:19.823626 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:19.817808    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93fc7bda-b92c-41ce-b567-658a571bc5ba-metrics-certs" pod "93fc7bda-b92c-41ce-b567-658a571bc5ba" (UID: "93fc7bda-b92c-41ce-b567-658a571bc5ba"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:43:19.837886 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:19.836354    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93fc7bda-b92c-41ce-b567-658a571bc5ba-maas-api-tls" pod "93fc7bda-b92c-41ce-b567-658a571bc5ba" (UID: "93fc7bda-b92c-41ce-b567-658a571bc5ba"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:43:19.846006 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:19.845916    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93fc7bda-b92c-41ce-b567-658a571bc5ba-kube-api-access-757mt" pod "93fc7bda-b92c-41ce-b567-658a571bc5ba" (UID: "93fc7bda-b92c-41ce-b567-658a571bc5ba"). InnerVolumeSpecName "kube-api-access-757mt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:43:19.888813 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:19.888780    4364 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93fc7bda-b92c-41ce-b567-658a571bc5ba-metrics-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:43:19.888813 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:19.888811    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-757mt\" (UniqueName: \"kubernetes.io/projected/93fc7bda-b92c-41ce-b567-658a571bc5ba-kube-api-access-757mt\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:43:19.889063 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:19.888823    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93fc7bda-b92c-41ce-b567-658a571bc5ba-maas-api-tls\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:43:20.314325 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:20.314292    4364 generic.go:356] "Generic (PLEG): container finished" podID="93fc7bda-b92c-41ce-b567-658a571bc5ba" containerID="abbd5d9520c8fd5aa9b863c0999bbd36e684215de3a70273421b525e6ce9c7e9" exitCode=0
Oct 30 13:43:20.314770 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:20.314338    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c47903-546d9c8468-kwdcc"
Oct 30 13:43:20.314770 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:20.314381    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c47903-546d9c8468-kwdcc" event={"ID":"93fc7bda-b92c-41ce-b567-658a571bc5ba","Type":"ContainerDied","Data":"abbd5d9520c8fd5aa9b863c0999bbd36e684215de3a70273421b525e6ce9c7e9"}
Oct 30 13:43:20.314770 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:20.314436    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c47903-546d9c8468-kwdcc" event={"ID":"93fc7bda-b92c-41ce-b567-658a571bc5ba","Type":"ContainerDied","Data":"bbd82eb1545fe3c2dff2d57f334da1384b557c52a02fd5f44d4b1e3a9a986ebd"}
Oct 30 13:43:20.314770 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:20.314465    4364 scope.go:122] "RemoveContainer" containerID="abbd5d9520c8fd5aa9b863c0999bbd36e684215de3a70273421b525e6ce9c7e9"
Oct 30 13:43:20.325284 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:20.325260    4364 scope.go:122] "RemoveContainer" containerID="abbd5d9520c8fd5aa9b863c0999bbd36e684215de3a70273421b525e6ce9c7e9"
Oct 30 13:43:20.325642 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:43:20.325585    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abbd5d9520c8fd5aa9b863c0999bbd36e684215de3a70273421b525e6ce9c7e9\": container with ID starting with abbd5d9520c8fd5aa9b863c0999bbd36e684215de3a70273421b525e6ce9c7e9 not found: ID does not exist" containerID="abbd5d9520c8fd5aa9b863c0999bbd36e684215de3a70273421b525e6ce9c7e9"
Oct 30 13:43:20.325701 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:20.325654    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abbd5d9520c8fd5aa9b863c0999bbd36e684215de3a70273421b525e6ce9c7e9"} err="failed to get container status \"abbd5d9520c8fd5aa9b863c0999bbd36e684215de3a70273421b525e6ce9c7e9\": rpc error: code = NotFound desc = could not find container \"abbd5d9520c8fd5aa9b863c0999bbd36e684215de3a70273421b525e6ce9c7e9\": container with ID starting with abbd5d9520c8fd5aa9b863c0999bbd36e684215de3a70273421b525e6ce9c7e9 not found: ID does not exist"
Oct 30 13:43:20.370924 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:20.370887    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c47903-546d9c8468-kwdcc"]
Oct 30 13:43:20.382286 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:20.382250    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c47903-546d9c8468-kwdcc"]
Oct 30 13:43:20.972461 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:20.972428    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93fc7bda-b92c-41ce-b567-658a571bc5ba" path="/var/lib/kubelet/pods/93fc7bda-b92c-41ce-b567-658a571bc5ba/volumes"
Oct 30 13:43:23.755627 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:23.754705    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:43:25.405049 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:25.405016    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e3ebf5-76b5bd46b4-8ll5d"]
Oct 30 13:43:25.405515 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:25.405255    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e3ebf5-76b5bd46b4-8ll5d" podUID="0b5b1e9d-9ea8-430d-9ff0-42b382a5f6a6" containerName="maas-api" containerID="cri-o://24d0b7697755d1209ab721fdca2f6a469fd54fa5d6a3ebe3dcb72d28187584df" gracePeriod=30
Oct 30 13:43:25.723247 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:25.723217    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e3ebf5-76b5bd46b4-8ll5d"
Oct 30 13:43:25.805126 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:25.805084    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b5b1e9d-9ea8-430d-9ff0-42b382a5f6a6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b5b1e9d-9ea8-430d-9ff0-42b382a5f6a6-maas-api-tls\") pod \"0b5b1e9d-9ea8-430d-9ff0-42b382a5f6a6\" (UID: \"0b5b1e9d-9ea8-430d-9ff0-42b382a5f6a6\") "
Oct 30 13:43:25.805314 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:25.805174    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b5b1e9d-9ea8-430d-9ff0-42b382a5f6a6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b5b1e9d-9ea8-430d-9ff0-42b382a5f6a6-metrics-certs\") pod \"0b5b1e9d-9ea8-430d-9ff0-42b382a5f6a6\" (UID: \"0b5b1e9d-9ea8-430d-9ff0-42b382a5f6a6\") "
Oct 30 13:43:25.805314 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:25.805235    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b5b1e9d-9ea8-430d-9ff0-42b382a5f6a6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b5b1e9d-9ea8-430d-9ff0-42b382a5f6a6-openshift-service-ca\") pod \"0b5b1e9d-9ea8-430d-9ff0-42b382a5f6a6\" (UID: \"0b5b1e9d-9ea8-430d-9ff0-42b382a5f6a6\") "
Oct 30 13:43:25.805314 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:25.805281    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b5b1e9d-9ea8-430d-9ff0-42b382a5f6a6-kube-api-access-ssq9z\" (UniqueName: \"kubernetes.io/projected/0b5b1e9d-9ea8-430d-9ff0-42b382a5f6a6-kube-api-access-ssq9z\") pod \"0b5b1e9d-9ea8-430d-9ff0-42b382a5f6a6\" (UID: \"0b5b1e9d-9ea8-430d-9ff0-42b382a5f6a6\") "
Oct 30 13:43:25.805966 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:25.805918    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b5b1e9d-9ea8-430d-9ff0-42b382a5f6a6-openshift-service-ca" pod "0b5b1e9d-9ea8-430d-9ff0-42b382a5f6a6" (UID: "0b5b1e9d-9ea8-430d-9ff0-42b382a5f6a6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:43:25.807871 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:25.807842    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b5b1e9d-9ea8-430d-9ff0-42b382a5f6a6-kube-api-access-ssq9z" pod "0b5b1e9d-9ea8-430d-9ff0-42b382a5f6a6" (UID: "0b5b1e9d-9ea8-430d-9ff0-42b382a5f6a6"). InnerVolumeSpecName "kube-api-access-ssq9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:43:25.808000 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:25.807915    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b5b1e9d-9ea8-430d-9ff0-42b382a5f6a6-metrics-certs" pod "0b5b1e9d-9ea8-430d-9ff0-42b382a5f6a6" (UID: "0b5b1e9d-9ea8-430d-9ff0-42b382a5f6a6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:43:25.808139 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:25.808112    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b5b1e9d-9ea8-430d-9ff0-42b382a5f6a6-maas-api-tls" pod "0b5b1e9d-9ea8-430d-9ff0-42b382a5f6a6" (UID: "0b5b1e9d-9ea8-430d-9ff0-42b382a5f6a6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:43:25.907232 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:25.907203    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b5b1e9d-9ea8-430d-9ff0-42b382a5f6a6-maas-api-tls\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:43:25.907232 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:25.907229    4364 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b5b1e9d-9ea8-430d-9ff0-42b382a5f6a6-metrics-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:43:25.907232 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:25.907239    4364 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b5b1e9d-9ea8-430d-9ff0-42b382a5f6a6-openshift-service-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:43:25.907450 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:25.907247    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssq9z\" (UniqueName: \"kubernetes.io/projected/0b5b1e9d-9ea8-430d-9ff0-42b382a5f6a6-kube-api-access-ssq9z\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:43:26.361354 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:26.361320    4364 generic.go:356] "Generic (PLEG): container finished" podID="0b5b1e9d-9ea8-430d-9ff0-42b382a5f6a6" containerID="24d0b7697755d1209ab721fdca2f6a469fd54fa5d6a3ebe3dcb72d28187584df" exitCode=0
Oct 30 13:43:26.361536 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:26.361374    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e3ebf5-76b5bd46b4-8ll5d"
Oct 30 13:43:26.361536 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:26.361408    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e3ebf5-76b5bd46b4-8ll5d" event={"ID":"0b5b1e9d-9ea8-430d-9ff0-42b382a5f6a6","Type":"ContainerDied","Data":"24d0b7697755d1209ab721fdca2f6a469fd54fa5d6a3ebe3dcb72d28187584df"}
Oct 30 13:43:26.361536 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:26.361471    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e3ebf5-76b5bd46b4-8ll5d" event={"ID":"0b5b1e9d-9ea8-430d-9ff0-42b382a5f6a6","Type":"ContainerDied","Data":"e20a2756b8afa144f3a5c3263e72d9740c44eb3a9e08fc3ca09e052d1d3fc697"}
Oct 30 13:43:26.361536 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:26.361502    4364 scope.go:122] "RemoveContainer" containerID="24d0b7697755d1209ab721fdca2f6a469fd54fa5d6a3ebe3dcb72d28187584df"
Oct 30 13:43:26.372781 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:26.372761    4364 scope.go:122] "RemoveContainer" containerID="24d0b7697755d1209ab721fdca2f6a469fd54fa5d6a3ebe3dcb72d28187584df"
Oct 30 13:43:26.373056 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:43:26.373030    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24d0b7697755d1209ab721fdca2f6a469fd54fa5d6a3ebe3dcb72d28187584df\": container with ID starting with 24d0b7697755d1209ab721fdca2f6a469fd54fa5d6a3ebe3dcb72d28187584df not found: ID does not exist" containerID="24d0b7697755d1209ab721fdca2f6a469fd54fa5d6a3ebe3dcb72d28187584df"
Oct 30 13:43:26.373166 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:26.373063    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24d0b7697755d1209ab721fdca2f6a469fd54fa5d6a3ebe3dcb72d28187584df"} err="failed to get container status \"24d0b7697755d1209ab721fdca2f6a469fd54fa5d6a3ebe3dcb72d28187584df\": rpc error: code = NotFound desc = could not find container \"24d0b7697755d1209ab721fdca2f6a469fd54fa5d6a3ebe3dcb72d28187584df\": container with ID starting with 24d0b7697755d1209ab721fdca2f6a469fd54fa5d6a3ebe3dcb72d28187584df not found: ID does not exist"
Oct 30 13:43:26.427686 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:26.427660    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e3ebf5-76b5bd46b4-8ll5d"]
Oct 30 13:43:26.432210 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:26.432187    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e3ebf5-76b5bd46b4-8ll5d"]
Oct 30 13:43:26.971795 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:26.971763    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b5b1e9d-9ea8-430d-9ff0-42b382a5f6a6" path="/var/lib/kubelet/pods/0b5b1e9d-9ea8-430d-9ff0-42b382a5f6a6/volumes"
Oct 30 13:43:32.282570 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:32.282534    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-30c004-6f9cbc6566-9j5jr"]
Oct 30 13:43:32.283027 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:32.282759    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30c004-6f9cbc6566-9j5jr" podUID="3de36fb7-459f-44d2-ba6a-b2cc20d24516" containerName="maas-api" containerID="cri-o://f565a988e9f8597c8d33b8f2f0d2ba8635fbe3d0df87479e05d0ead60b818fcd" gracePeriod=30
Oct 30 13:43:32.409353 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:32.409323    4364 generic.go:356] "Generic (PLEG): container finished" podID="3de36fb7-459f-44d2-ba6a-b2cc20d24516" containerID="f565a988e9f8597c8d33b8f2f0d2ba8635fbe3d0df87479e05d0ead60b818fcd" exitCode=0
Oct 30 13:43:32.409496 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:32.409402    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30c004-6f9cbc6566-9j5jr" event={"ID":"3de36fb7-459f-44d2-ba6a-b2cc20d24516","Type":"ContainerDied","Data":"f565a988e9f8597c8d33b8f2f0d2ba8635fbe3d0df87479e05d0ead60b818fcd"}
Oct 30 13:43:32.612647 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:32.612593    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30c004-6f9cbc6566-9j5jr"
Oct 30 13:43:32.649609 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:32.649569    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3de36fb7-459f-44d2-ba6a-b2cc20d24516-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3de36fb7-459f-44d2-ba6a-b2cc20d24516-openshift-service-ca\") pod \"3de36fb7-459f-44d2-ba6a-b2cc20d24516\" (UID: \"3de36fb7-459f-44d2-ba6a-b2cc20d24516\") "
Oct 30 13:43:32.649784 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:32.649775    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3de36fb7-459f-44d2-ba6a-b2cc20d24516-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3de36fb7-459f-44d2-ba6a-b2cc20d24516-maas-api-tls\") pod \"3de36fb7-459f-44d2-ba6a-b2cc20d24516\" (UID: \"3de36fb7-459f-44d2-ba6a-b2cc20d24516\") "
Oct 30 13:43:32.649845 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:32.649827    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3de36fb7-459f-44d2-ba6a-b2cc20d24516-kube-api-access-wp7cc\" (UniqueName: \"kubernetes.io/projected/3de36fb7-459f-44d2-ba6a-b2cc20d24516-kube-api-access-wp7cc\") pod \"3de36fb7-459f-44d2-ba6a-b2cc20d24516\" (UID: \"3de36fb7-459f-44d2-ba6a-b2cc20d24516\") "
Oct 30 13:43:32.649886 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:32.649860    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3de36fb7-459f-44d2-ba6a-b2cc20d24516-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3de36fb7-459f-44d2-ba6a-b2cc20d24516-metrics-certs\") pod \"3de36fb7-459f-44d2-ba6a-b2cc20d24516\" (UID: \"3de36fb7-459f-44d2-ba6a-b2cc20d24516\") "
Oct 30 13:43:32.649945 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:32.649898    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3de36fb7-459f-44d2-ba6a-b2cc20d24516-openshift-service-ca" pod "3de36fb7-459f-44d2-ba6a-b2cc20d24516" (UID: "3de36fb7-459f-44d2-ba6a-b2cc20d24516"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:43:32.650538 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:32.650516    4364 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3de36fb7-459f-44d2-ba6a-b2cc20d24516-openshift-service-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:43:32.651962 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:32.651936    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3de36fb7-459f-44d2-ba6a-b2cc20d24516-metrics-certs" pod "3de36fb7-459f-44d2-ba6a-b2cc20d24516" (UID: "3de36fb7-459f-44d2-ba6a-b2cc20d24516"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:43:32.652060 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:32.651973    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3de36fb7-459f-44d2-ba6a-b2cc20d24516-maas-api-tls" pod "3de36fb7-459f-44d2-ba6a-b2cc20d24516" (UID: "3de36fb7-459f-44d2-ba6a-b2cc20d24516"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:43:32.652108 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:32.652088    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3de36fb7-459f-44d2-ba6a-b2cc20d24516-kube-api-access-wp7cc" pod "3de36fb7-459f-44d2-ba6a-b2cc20d24516" (UID: "3de36fb7-459f-44d2-ba6a-b2cc20d24516"). InnerVolumeSpecName "kube-api-access-wp7cc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:43:32.751925 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:32.751889    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3de36fb7-459f-44d2-ba6a-b2cc20d24516-maas-api-tls\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:43:32.751925 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:32.751918    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wp7cc\" (UniqueName: \"kubernetes.io/projected/3de36fb7-459f-44d2-ba6a-b2cc20d24516-kube-api-access-wp7cc\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:43:32.751925 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:32.751929    4364 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3de36fb7-459f-44d2-ba6a-b2cc20d24516-metrics-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:43:33.418845 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:33.418809    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30c004-6f9cbc6566-9j5jr"
Oct 30 13:43:33.419303 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:33.418842    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30c004-6f9cbc6566-9j5jr" event={"ID":"3de36fb7-459f-44d2-ba6a-b2cc20d24516","Type":"ContainerDied","Data":"c7703ef93e5ced2dbfb4f38e8fb48a33b6dd8c8e74f725144f5432f008fe6150"}
Oct 30 13:43:33.419303 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:33.418920    4364 scope.go:122] "RemoveContainer" containerID="f565a988e9f8597c8d33b8f2f0d2ba8635fbe3d0df87479e05d0ead60b818fcd"
Oct 30 13:43:33.509175 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:33.509139    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-30c004-6f9cbc6566-9j5jr"]
Oct 30 13:43:33.519865 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:33.519836    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-30c004-6f9cbc6566-9j5jr"]
Oct 30 13:43:34.975506 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:34.975471    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3de36fb7-459f-44d2-ba6a-b2cc20d24516" path="/var/lib/kubelet/pods/3de36fb7-459f-44d2-ba6a-b2cc20d24516/volumes"
Oct 30 13:43:37.227105 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:37.227068    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8ee35e-6968f44cd7-7xvp8"]
Oct 30 13:43:37.227584 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:37.227261    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8ee35e-6968f44cd7-7xvp8" podUID="d98dc22b-5142-4d00-bf71-27ead6f76d3e" containerName="maas-api" containerID="cri-o://b84de1bf2aa1b743bd937b3b1ae6fa988ca585c67ee55b21dd6e1181cd02db99" gracePeriod=30
Oct 30 13:43:37.536496 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:37.536405    4364 generic.go:356] "Generic (PLEG): container finished" podID="d98dc22b-5142-4d00-bf71-27ead6f76d3e" containerID="b84de1bf2aa1b743bd937b3b1ae6fa988ca585c67ee55b21dd6e1181cd02db99" exitCode=0
Oct 30 13:43:37.536496 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:37.536491    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8ee35e-6968f44cd7-7xvp8" event={"ID":"d98dc22b-5142-4d00-bf71-27ead6f76d3e","Type":"ContainerDied","Data":"b84de1bf2aa1b743bd937b3b1ae6fa988ca585c67ee55b21dd6e1181cd02db99"}
Oct 30 13:43:37.571669 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:37.571641    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8ee35e-6968f44cd7-7xvp8"
Oct 30 13:43:37.704552 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:37.703113    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d98dc22b-5142-4d00-bf71-27ead6f76d3e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d98dc22b-5142-4d00-bf71-27ead6f76d3e-metrics-certs\") pod \"d98dc22b-5142-4d00-bf71-27ead6f76d3e\" (UID: \"d98dc22b-5142-4d00-bf71-27ead6f76d3e\") "
Oct 30 13:43:37.704552 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:37.703333    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d98dc22b-5142-4d00-bf71-27ead6f76d3e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d98dc22b-5142-4d00-bf71-27ead6f76d3e-openshift-service-ca\") pod \"d98dc22b-5142-4d00-bf71-27ead6f76d3e\" (UID: \"d98dc22b-5142-4d00-bf71-27ead6f76d3e\") "
Oct 30 13:43:37.704552 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:37.703415    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d98dc22b-5142-4d00-bf71-27ead6f76d3e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d98dc22b-5142-4d00-bf71-27ead6f76d3e-maas-api-tls\") pod \"d98dc22b-5142-4d00-bf71-27ead6f76d3e\" (UID: \"d98dc22b-5142-4d00-bf71-27ead6f76d3e\") "
Oct 30 13:43:37.704552 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:37.703568    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d98dc22b-5142-4d00-bf71-27ead6f76d3e-kube-api-access-559wt\" (UniqueName: \"kubernetes.io/projected/d98dc22b-5142-4d00-bf71-27ead6f76d3e-kube-api-access-559wt\") pod \"d98dc22b-5142-4d00-bf71-27ead6f76d3e\" (UID: \"d98dc22b-5142-4d00-bf71-27ead6f76d3e\") "
Oct 30 13:43:37.727907 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:37.727864    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d98dc22b-5142-4d00-bf71-27ead6f76d3e-openshift-service-ca" pod "d98dc22b-5142-4d00-bf71-27ead6f76d3e" (UID: "d98dc22b-5142-4d00-bf71-27ead6f76d3e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:43:37.731807 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:37.731745    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d98dc22b-5142-4d00-bf71-27ead6f76d3e-maas-api-tls" pod "d98dc22b-5142-4d00-bf71-27ead6f76d3e" (UID: "d98dc22b-5142-4d00-bf71-27ead6f76d3e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:43:37.740847 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:37.740778    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d98dc22b-5142-4d00-bf71-27ead6f76d3e-kube-api-access-559wt" pod "d98dc22b-5142-4d00-bf71-27ead6f76d3e" (UID: "d98dc22b-5142-4d00-bf71-27ead6f76d3e"). InnerVolumeSpecName "kube-api-access-559wt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:43:37.752398 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:37.751984    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d98dc22b-5142-4d00-bf71-27ead6f76d3e-metrics-certs" pod "d98dc22b-5142-4d00-bf71-27ead6f76d3e" (UID: "d98dc22b-5142-4d00-bf71-27ead6f76d3e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:43:37.806186 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:37.806106    4364 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d98dc22b-5142-4d00-bf71-27ead6f76d3e-metrics-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:43:37.806186 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:37.806137    4364 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d98dc22b-5142-4d00-bf71-27ead6f76d3e-openshift-service-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:43:37.806186 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:37.806149    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d98dc22b-5142-4d00-bf71-27ead6f76d3e-maas-api-tls\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:43:37.806186 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:37.806163    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-559wt\" (UniqueName: \"kubernetes.io/projected/d98dc22b-5142-4d00-bf71-27ead6f76d3e-kube-api-access-559wt\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:43:38.573892 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:38.572464    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8ee35e-6968f44cd7-7xvp8" event={"ID":"d98dc22b-5142-4d00-bf71-27ead6f76d3e","Type":"ContainerDied","Data":"50062c71cbac4238139907a2908756acf8fceffc142d353b60f103959d397bdb"}
Oct 30 13:43:38.573892 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:38.572551    4364 scope.go:122] "RemoveContainer" containerID="b84de1bf2aa1b743bd937b3b1ae6fa988ca585c67ee55b21dd6e1181cd02db99"
Oct 30 13:43:38.573892 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:38.572743    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8ee35e-6968f44cd7-7xvp8"
Oct 30 13:43:38.628687 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:38.628656    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8ee35e-6968f44cd7-7xvp8"]
Oct 30 13:43:38.640315 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:38.640280    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8ee35e-6968f44cd7-7xvp8"]
Oct 30 13:43:38.971169 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:38.971132    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d98dc22b-5142-4d00-bf71-27ead6f76d3e" path="/var/lib/kubelet/pods/d98dc22b-5142-4d00-bf71-27ead6f76d3e/volumes"
Oct 30 13:43:39.328045 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:39.327962    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b0ea9-5f686c7795-7527w"]
Oct 30 13:43:41.705211 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:43:41.705167    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:43:42.735208 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:42.735174    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s"]
Oct 30 13:43:42.735816 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:42.735442    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s" podUID="0a9a3c04-3da8-4efe-8daf-b4821c563a24" containerName="istio-proxy" containerID="cri-o://961fa587390df58fd5669c5aaabc2151bc0a9d4f83dc505576583e277c9b4e17" gracePeriod=30
Oct 30 13:43:44.329072 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:44.329015    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b0ea9-5f686c7795-7527w" podUID="c2597429-b2f7-4263-b715-19410183a9a7" containerName="payload-pre-processing" containerID="cri-o://f36d0c3f3e10c89bf508565ad24228081b8d4ed406030f13d9f89d110bbf498a" gracePeriod=25
Oct 30 13:43:46.465441 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:46.465406    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj"]
Oct 30 13:43:46.466021 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:46.465699    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj" podUID="23116724-6a57-401b-8ba4-a105a71a5e6d" containerName="istio-proxy" containerID="cri-o://6865a7361de91246a238324421ad90d014329021e8a40a8a1b7978c00063bd13" gracePeriod=30
Oct 30 13:43:46.475140 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:46.475110    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq"]
Oct 30 13:43:46.475384 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:46.475344    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq" podUID="09c95c32-f60a-4f60-90ff-90f447247cce" containerName="istio-proxy" containerID="cri-o://a526058e96bf8fe46f5fac380c980be2b9991ed7f86bf8b2825b56a417636a73" gracePeriod=30
Oct 30 13:43:47.663692 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:47.663657    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg"]
Oct 30 13:43:47.664107 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:47.663853    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg" podUID="5ce05a81-71f3-4dab-96e1-44c9ea628a7b" containerName="istio-proxy" containerID="cri-o://b3eb44114eb6eccd60b6eba100561e0a8be2fce1da4d1d51712da581e596d26b" gracePeriod=30
Oct 30 13:43:47.902043 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:47.901956    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b0ea9-5f686c7795-7527w" podUID="c2597429-b2f7-4263-b715-19410183a9a7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:43:48.057265 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:48.057221    4364 patch_prober.go:29] interesting pod/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:43:48.057441 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:48.057293    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg" podUID="5ce05a81-71f3-4dab-96e1-44c9ea628a7b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:43:48.140543 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:48.140519    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s"
Oct 30 13:43:48.230653 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:48.230582    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a9a3c04-3da8-4efe-8daf-b4821c563a24-kube-api-access-2kjzn\" (UniqueName: \"kubernetes.io/projected/0a9a3c04-3da8-4efe-8daf-b4821c563a24-kube-api-access-2kjzn\") pod \"0a9a3c04-3da8-4efe-8daf-b4821c563a24\" (UID: \"0a9a3c04-3da8-4efe-8daf-b4821c563a24\") "
Oct 30 13:43:48.230849 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:48.230776    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a9a3c04-3da8-4efe-8daf-b4821c563a24-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9a3c04-3da8-4efe-8daf-b4821c563a24-credential-socket\") pod \"0a9a3c04-3da8-4efe-8daf-b4821c563a24\" (UID: \"0a9a3c04-3da8-4efe-8daf-b4821c563a24\") "
Oct 30 13:43:48.230849 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:48.230840    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a9a3c04-3da8-4efe-8daf-b4821c563a24-istio-token\" (UniqueName: \"kubernetes.io/projected/0a9a3c04-3da8-4efe-8daf-b4821c563a24-istio-token\") pod \"0a9a3c04-3da8-4efe-8daf-b4821c563a24\" (UID: \"0a9a3c04-3da8-4efe-8daf-b4821c563a24\") "
Oct 30 13:43:48.230962 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:48.230870    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a9a3c04-3da8-4efe-8daf-b4821c563a24-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a9a3c04-3da8-4efe-8daf-b4821c563a24-workload-certs\") pod \"0a9a3c04-3da8-4efe-8daf-b4821c563a24\" (UID: \"0a9a3c04-3da8-4efe-8daf-b4821c563a24\") "
Oct 30 13:43:48.230962 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:48.230925    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a9a3c04-3da8-4efe-8daf-b4821c563a24-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a9a3c04-3da8-4efe-8daf-b4821c563a24-istiod-ca-cert\") pod \"0a9a3c04-3da8-4efe-8daf-b4821c563a24\" (UID: \"0a9a3c04-3da8-4efe-8daf-b4821c563a24\") "
Oct 30 13:43:48.230962 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:48.230955    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a9a3c04-3da8-4efe-8daf-b4821c563a24-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a9a3c04-3da8-4efe-8daf-b4821c563a24-istio-podinfo\") pod \"0a9a3c04-3da8-4efe-8daf-b4821c563a24\" (UID: \"0a9a3c04-3da8-4efe-8daf-b4821c563a24\") "
Oct 30 13:43:48.231126 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:48.231013    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a9a3c04-3da8-4efe-8daf-b4821c563a24-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a9a3c04-3da8-4efe-8daf-b4821c563a24-istio-data\") pod \"0a9a3c04-3da8-4efe-8daf-b4821c563a24\" (UID: \"0a9a3c04-3da8-4efe-8daf-b4821c563a24\") "
Oct 30 13:43:48.231126 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:48.231061    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a9a3c04-3da8-4efe-8daf-b4821c563a24-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9a3c04-3da8-4efe-8daf-b4821c563a24-workload-socket\") pod \"0a9a3c04-3da8-4efe-8daf-b4821c563a24\" (UID: \"0a9a3c04-3da8-4efe-8daf-b4821c563a24\") "
Oct 30 13:43:48.231126 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:48.231108    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a9a3c04-3da8-4efe-8daf-b4821c563a24-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a9a3c04-3da8-4efe-8daf-b4821c563a24-istio-envoy\") pod \"0a9a3c04-3da8-4efe-8daf-b4821c563a24\" (UID: \"0a9a3c04-3da8-4efe-8daf-b4821c563a24\") "
Oct 30 13:43:48.231572 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:48.231541    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a9a3c04-3da8-4efe-8daf-b4821c563a24-workload-certs" pod "0a9a3c04-3da8-4efe-8daf-b4821c563a24" (UID: "0a9a3c04-3da8-4efe-8daf-b4821c563a24"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:43:48.231813 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:48.231786    4364 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a9a3c04-3da8-4efe-8daf-b4821c563a24-workload-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:43:48.232768 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:48.232720    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a9a3c04-3da8-4efe-8daf-b4821c563a24-istio-data" pod "0a9a3c04-3da8-4efe-8daf-b4821c563a24" (UID: "0a9a3c04-3da8-4efe-8daf-b4821c563a24"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:43:48.232963 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:48.232922    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a9a3c04-3da8-4efe-8daf-b4821c563a24-credential-socket" pod "0a9a3c04-3da8-4efe-8daf-b4821c563a24" (UID: "0a9a3c04-3da8-4efe-8daf-b4821c563a24"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:43:48.233183 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:48.233117    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a9a3c04-3da8-4efe-8daf-b4821c563a24-workload-socket" pod "0a9a3c04-3da8-4efe-8daf-b4821c563a24" (UID: "0a9a3c04-3da8-4efe-8daf-b4821c563a24"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:43:48.233740 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:48.233705    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a9a3c04-3da8-4efe-8daf-b4821c563a24-istiod-ca-cert" pod "0a9a3c04-3da8-4efe-8daf-b4821c563a24" (UID: "0a9a3c04-3da8-4efe-8daf-b4821c563a24"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:43:48.234153 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:48.234127    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a9a3c04-3da8-4efe-8daf-b4821c563a24-istio-token" pod "0a9a3c04-3da8-4efe-8daf-b4821c563a24" (UID: "0a9a3c04-3da8-4efe-8daf-b4821c563a24"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:43:48.234263 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:48.234211    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a9a3c04-3da8-4efe-8daf-b4821c563a24-istio-envoy" pod "0a9a3c04-3da8-4efe-8daf-b4821c563a24" (UID: "0a9a3c04-3da8-4efe-8daf-b4821c563a24"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:43:48.234387 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:48.234357    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a9a3c04-3da8-4efe-8daf-b4821c563a24-kube-api-access-2kjzn" pod "0a9a3c04-3da8-4efe-8daf-b4821c563a24" (UID: "0a9a3c04-3da8-4efe-8daf-b4821c563a24"). InnerVolumeSpecName "kube-api-access-2kjzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:43:48.235029 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:48.235011    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a9a3c04-3da8-4efe-8daf-b4821c563a24-istio-podinfo" pod "0a9a3c04-3da8-4efe-8daf-b4821c563a24" (UID: "0a9a3c04-3da8-4efe-8daf-b4821c563a24"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:43:48.333160 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:48.333103    4364 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a9a3c04-3da8-4efe-8daf-b4821c563a24-istio-token\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:43:48.333160 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:48.333125    4364 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a9a3c04-3da8-4efe-8daf-b4821c563a24-istiod-ca-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:43:48.333160 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:48.333134    4364 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a9a3c04-3da8-4efe-8daf-b4821c563a24-istio-podinfo\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:43:48.333160 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:48.333142    4364 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a9a3c04-3da8-4efe-8daf-b4821c563a24-istio-data\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:43:48.333160 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:48.333150    4364 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9a3c04-3da8-4efe-8daf-b4821c563a24-workload-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:43:48.333160 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:48.333158    4364 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a9a3c04-3da8-4efe-8daf-b4821c563a24-istio-envoy\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:43:48.333439 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:48.333168    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kjzn\" (UniqueName: \"kubernetes.io/projected/0a9a3c04-3da8-4efe-8daf-b4821c563a24-kube-api-access-2kjzn\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:43:48.333439 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:48.333176    4364 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9a3c04-3da8-4efe-8daf-b4821c563a24-credential-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:43:48.741730 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:48.741699    4364 generic.go:356] "Generic (PLEG): container finished" podID="0a9a3c04-3da8-4efe-8daf-b4821c563a24" containerID="961fa587390df58fd5669c5aaabc2151bc0a9d4f83dc505576583e277c9b4e17" exitCode=0
Oct 30 13:43:48.742152 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:48.741773    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s"
Oct 30 13:43:48.742152 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:48.741784    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s" event={"ID":"0a9a3c04-3da8-4efe-8daf-b4821c563a24","Type":"ContainerDied","Data":"961fa587390df58fd5669c5aaabc2151bc0a9d4f83dc505576583e277c9b4e17"}
Oct 30 13:43:48.742152 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:48.741859    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s" event={"ID":"0a9a3c04-3da8-4efe-8daf-b4821c563a24","Type":"ContainerDied","Data":"d8ae01058b6db37922479aa9c5b2d466ed2b13b55a1b9f15fc5141fe693c32ff"}
Oct 30 13:43:48.742152 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:48.741884    4364 scope.go:122] "RemoveContainer" containerID="961fa587390df58fd5669c5aaabc2151bc0a9d4f83dc505576583e277c9b4e17"
Oct 30 13:43:48.755178 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:48.755152    4364 scope.go:122] "RemoveContainer" containerID="70f543b1bcf791669112619fbd29585df08d5c8efeae09bb40a437fe62593169"
Oct 30 13:43:48.767538 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:48.767514    4364 scope.go:122] "RemoveContainer" containerID="961fa587390df58fd5669c5aaabc2151bc0a9d4f83dc505576583e277c9b4e17"
Oct 30 13:43:48.767925 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:43:48.767901    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"961fa587390df58fd5669c5aaabc2151bc0a9d4f83dc505576583e277c9b4e17\": container with ID starting with 961fa587390df58fd5669c5aaabc2151bc0a9d4f83dc505576583e277c9b4e17 not found: ID does not exist" containerID="961fa587390df58fd5669c5aaabc2151bc0a9d4f83dc505576583e277c9b4e17"
Oct 30 13:43:48.768022 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:48.767942    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"961fa587390df58fd5669c5aaabc2151bc0a9d4f83dc505576583e277c9b4e17"} err="failed to get container status \"961fa587390df58fd5669c5aaabc2151bc0a9d4f83dc505576583e277c9b4e17\": rpc error: code = NotFound desc = could not find container \"961fa587390df58fd5669c5aaabc2151bc0a9d4f83dc505576583e277c9b4e17\": container with ID starting with 961fa587390df58fd5669c5aaabc2151bc0a9d4f83dc505576583e277c9b4e17 not found: ID does not exist"
Oct 30 13:43:48.768022 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:48.767960    4364 scope.go:122] "RemoveContainer" containerID="70f543b1bcf791669112619fbd29585df08d5c8efeae09bb40a437fe62593169"
Oct 30 13:43:48.768267 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:43:48.768247    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70f543b1bcf791669112619fbd29585df08d5c8efeae09bb40a437fe62593169\": container with ID starting with 70f543b1bcf791669112619fbd29585df08d5c8efeae09bb40a437fe62593169 not found: ID does not exist" containerID="70f543b1bcf791669112619fbd29585df08d5c8efeae09bb40a437fe62593169"
Oct 30 13:43:48.768324 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:48.768271    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70f543b1bcf791669112619fbd29585df08d5c8efeae09bb40a437fe62593169"} err="failed to get container status \"70f543b1bcf791669112619fbd29585df08d5c8efeae09bb40a437fe62593169\": rpc error: code = NotFound desc = could not find container \"70f543b1bcf791669112619fbd29585df08d5c8efeae09bb40a437fe62593169\": container with ID starting with 70f543b1bcf791669112619fbd29585df08d5c8efeae09bb40a437fe62593169 not found: ID does not exist"
Oct 30 13:43:48.821635 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:48.821608    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s"]
Oct 30 13:43:48.826031 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:48.826012    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-30c004-openshift-default-964748c6f-4zp7s"]
Oct 30 13:43:48.976970 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:48.972952    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a9a3c04-3da8-4efe-8daf-b4821c563a24" path="/var/lib/kubelet/pods/0a9a3c04-3da8-4efe-8daf-b4821c563a24/volumes"
Oct 30 13:43:50.987146 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:50.987116    4364 patch_prober.go:29] interesting pod/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:43:50.987544 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:50.987175    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj" podUID="23116724-6a57-401b-8ba4-a105a71a5e6d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:43:51.036413 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:51.036375    4364 patch_prober.go:29] interesting pod/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:43:51.036572 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:51.036432    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq" podUID="09c95c32-f60a-4f60-90ff-90f447247cce" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:43:51.763285 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:51.763251    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj"
Oct 30 13:43:51.769666 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:51.769635    4364 generic.go:356] "Generic (PLEG): container finished" podID="23116724-6a57-401b-8ba4-a105a71a5e6d" containerID="6865a7361de91246a238324421ad90d014329021e8a40a8a1b7978c00063bd13" exitCode=0
Oct 30 13:43:51.769666 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:51.769659    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj"
Oct 30 13:43:51.769866 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:51.769714    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj" event={"ID":"23116724-6a57-401b-8ba4-a105a71a5e6d","Type":"ContainerDied","Data":"6865a7361de91246a238324421ad90d014329021e8a40a8a1b7978c00063bd13"}
Oct 30 13:43:51.769866 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:51.769768    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj" event={"ID":"23116724-6a57-401b-8ba4-a105a71a5e6d","Type":"ContainerDied","Data":"a4b2a8849de9b57834cc3d4d09ce1112258ce9e34da69e77d6871ea30438d87c"}
Oct 30 13:43:51.769866 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:51.769795    4364 scope.go:122] "RemoveContainer" containerID="6865a7361de91246a238324421ad90d014329021e8a40a8a1b7978c00063bd13"
Oct 30 13:43:51.771452 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:51.771431    4364 generic.go:356] "Generic (PLEG): container finished" podID="09c95c32-f60a-4f60-90ff-90f447247cce" containerID="a526058e96bf8fe46f5fac380c980be2b9991ed7f86bf8b2825b56a417636a73" exitCode=0
Oct 30 13:43:51.771590 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:51.771512    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq" event={"ID":"09c95c32-f60a-4f60-90ff-90f447247cce","Type":"ContainerDied","Data":"a526058e96bf8fe46f5fac380c980be2b9991ed7f86bf8b2825b56a417636a73"}
Oct 30 13:43:51.780531 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:51.780510    4364 scope.go:122] "RemoveContainer" containerID="6865a7361de91246a238324421ad90d014329021e8a40a8a1b7978c00063bd13"
Oct 30 13:43:51.780866 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:43:51.780843    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6865a7361de91246a238324421ad90d014329021e8a40a8a1b7978c00063bd13\": container with ID starting with 6865a7361de91246a238324421ad90d014329021e8a40a8a1b7978c00063bd13 not found: ID does not exist" containerID="6865a7361de91246a238324421ad90d014329021e8a40a8a1b7978c00063bd13"
Oct 30 13:43:51.780995 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:51.780873    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6865a7361de91246a238324421ad90d014329021e8a40a8a1b7978c00063bd13"} err="failed to get container status \"6865a7361de91246a238324421ad90d014329021e8a40a8a1b7978c00063bd13\": rpc error: code = NotFound desc = could not find container \"6865a7361de91246a238324421ad90d014329021e8a40a8a1b7978c00063bd13\": container with ID starting with 6865a7361de91246a238324421ad90d014329021e8a40a8a1b7978c00063bd13 not found: ID does not exist"
Oct 30 13:43:51.802186 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:51.802168    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq"
Oct 30 13:43:51.880394 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:51.880350    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23116724-6a57-401b-8ba4-a105a71a5e6d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23116724-6a57-401b-8ba4-a105a71a5e6d-istiod-ca-cert\") pod \"23116724-6a57-401b-8ba4-a105a71a5e6d\" (UID: \"23116724-6a57-401b-8ba4-a105a71a5e6d\") "
Oct 30 13:43:51.880581 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:51.880406    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09c95c32-f60a-4f60-90ff-90f447247cce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c95c32-f60a-4f60-90ff-90f447247cce-credential-socket\") pod \"09c95c32-f60a-4f60-90ff-90f447247cce\" (UID: \"09c95c32-f60a-4f60-90ff-90f447247cce\") "
Oct 30 13:43:51.880581 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:51.880475    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09c95c32-f60a-4f60-90ff-90f447247cce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09c95c32-f60a-4f60-90ff-90f447247cce-istio-data\") pod \"09c95c32-f60a-4f60-90ff-90f447247cce\" (UID: \"09c95c32-f60a-4f60-90ff-90f447247cce\") "
Oct 30 13:43:51.880581 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:51.880503    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23116724-6a57-401b-8ba4-a105a71a5e6d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23116724-6a57-401b-8ba4-a105a71a5e6d-workload-socket\") pod \"23116724-6a57-401b-8ba4-a105a71a5e6d\" (UID: \"23116724-6a57-401b-8ba4-a105a71a5e6d\") "
Oct 30 13:43:51.880581 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:51.880545    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09c95c32-f60a-4f60-90ff-90f447247cce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c95c32-f60a-4f60-90ff-90f447247cce-workload-socket\") pod \"09c95c32-f60a-4f60-90ff-90f447247cce\" (UID: \"09c95c32-f60a-4f60-90ff-90f447247cce\") "
Oct 30 13:43:51.880581 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:51.880571    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09c95c32-f60a-4f60-90ff-90f447247cce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09c95c32-f60a-4f60-90ff-90f447247cce-istiod-ca-cert\") pod \"09c95c32-f60a-4f60-90ff-90f447247cce\" (UID: \"09c95c32-f60a-4f60-90ff-90f447247cce\") "
Oct 30 13:43:51.880917 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:51.880671    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23116724-6a57-401b-8ba4-a105a71a5e6d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23116724-6a57-401b-8ba4-a105a71a5e6d-credential-socket\") pod \"23116724-6a57-401b-8ba4-a105a71a5e6d\" (UID: \"23116724-6a57-401b-8ba4-a105a71a5e6d\") "
Oct 30 13:43:51.880917 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:51.880742    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/09c95c32-f60a-4f60-90ff-90f447247cce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09c95c32-f60a-4f60-90ff-90f447247cce-istio-podinfo\") pod \"09c95c32-f60a-4f60-90ff-90f447247cce\" (UID: \"09c95c32-f60a-4f60-90ff-90f447247cce\") "
Oct 30 13:43:51.880917 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:51.880796    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09c95c32-f60a-4f60-90ff-90f447247cce-istio-token\" (UniqueName: \"kubernetes.io/projected/09c95c32-f60a-4f60-90ff-90f447247cce-istio-token\") pod \"09c95c32-f60a-4f60-90ff-90f447247cce\" (UID: \"09c95c32-f60a-4f60-90ff-90f447247cce\") "
Oct 30 13:43:51.880917 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:51.880827    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09c95c32-f60a-4f60-90ff-90f447247cce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09c95c32-f60a-4f60-90ff-90f447247cce-istio-envoy\") pod \"09c95c32-f60a-4f60-90ff-90f447247cce\" (UID: \"09c95c32-f60a-4f60-90ff-90f447247cce\") "
Oct 30 13:43:51.880917 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:51.880822    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09c95c32-f60a-4f60-90ff-90f447247cce-istio-data" pod "09c95c32-f60a-4f60-90ff-90f447247cce" (UID: "09c95c32-f60a-4f60-90ff-90f447247cce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:43:51.880917 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:51.880860    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23116724-6a57-401b-8ba4-a105a71a5e6d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23116724-6a57-401b-8ba4-a105a71a5e6d-workload-certs\") pod \"23116724-6a57-401b-8ba4-a105a71a5e6d\" (UID: \"23116724-6a57-401b-8ba4-a105a71a5e6d\") "
Oct 30 13:43:51.881264 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:51.880961    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09c95c32-f60a-4f60-90ff-90f447247cce-kube-api-access-pwc5b\" (UniqueName: \"kubernetes.io/projected/09c95c32-f60a-4f60-90ff-90f447247cce-kube-api-access-pwc5b\") pod \"09c95c32-f60a-4f60-90ff-90f447247cce\" (UID: \"09c95c32-f60a-4f60-90ff-90f447247cce\") "
Oct 30 13:43:51.881264 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:51.881010    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23116724-6a57-401b-8ba4-a105a71a5e6d-kube-api-access-srhwh\" (UniqueName: \"kubernetes.io/projected/23116724-6a57-401b-8ba4-a105a71a5e6d-kube-api-access-srhwh\") pod \"23116724-6a57-401b-8ba4-a105a71a5e6d\" (UID: \"23116724-6a57-401b-8ba4-a105a71a5e6d\") "
Oct 30 13:43:51.881264 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:51.881030    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23116724-6a57-401b-8ba4-a105a71a5e6d-credential-socket" pod "23116724-6a57-401b-8ba4-a105a71a5e6d" (UID: "23116724-6a57-401b-8ba4-a105a71a5e6d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:43:51.881264 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:51.881040    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23116724-6a57-401b-8ba4-a105a71a5e6d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23116724-6a57-401b-8ba4-a105a71a5e6d-istio-envoy\") pod \"23116724-6a57-401b-8ba4-a105a71a5e6d\" (UID: \"23116724-6a57-401b-8ba4-a105a71a5e6d\") "
Oct 30 13:43:51.881264 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:51.881068    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09c95c32-f60a-4f60-90ff-90f447247cce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09c95c32-f60a-4f60-90ff-90f447247cce-workload-certs\") pod \"09c95c32-f60a-4f60-90ff-90f447247cce\" (UID: \"09c95c32-f60a-4f60-90ff-90f447247cce\") "
Oct 30 13:43:51.881264 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:51.881100    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23116724-6a57-401b-8ba4-a105a71a5e6d-istio-token\" (UniqueName: \"kubernetes.io/projected/23116724-6a57-401b-8ba4-a105a71a5e6d-istio-token\") pod \"23116724-6a57-401b-8ba4-a105a71a5e6d\" (UID: \"23116724-6a57-401b-8ba4-a105a71a5e6d\") "
Oct 30 13:43:51.881264 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:51.881184    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23116724-6a57-401b-8ba4-a105a71a5e6d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23116724-6a57-401b-8ba4-a105a71a5e6d-istio-data\") pod \"23116724-6a57-401b-8ba4-a105a71a5e6d\" (UID: \"23116724-6a57-401b-8ba4-a105a71a5e6d\") "
Oct 30 13:43:51.881264 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:51.881222    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09c95c32-f60a-4f60-90ff-90f447247cce-workload-socket" pod "09c95c32-f60a-4f60-90ff-90f447247cce" (UID: "09c95c32-f60a-4f60-90ff-90f447247cce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:43:51.881264 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:51.881225    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/23116724-6a57-401b-8ba4-a105a71a5e6d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23116724-6a57-401b-8ba4-a105a71a5e6d-istio-podinfo\") pod \"23116724-6a57-401b-8ba4-a105a71a5e6d\" (UID: \"23116724-6a57-401b-8ba4-a105a71a5e6d\") "
Oct 30 13:43:51.881749 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:51.881543    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09c95c32-f60a-4f60-90ff-90f447247cce-istiod-ca-cert" pod "09c95c32-f60a-4f60-90ff-90f447247cce" (UID: "09c95c32-f60a-4f60-90ff-90f447247cce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:43:51.881749 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:51.881627    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23116724-6a57-401b-8ba4-a105a71a5e6d-workload-socket" pod "23116724-6a57-401b-8ba4-a105a71a5e6d" (UID: "23116724-6a57-401b-8ba4-a105a71a5e6d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:43:51.881874 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:51.881851    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23116724-6a57-401b-8ba4-a105a71a5e6d-workload-certs" pod "23116724-6a57-401b-8ba4-a105a71a5e6d" (UID: "23116724-6a57-401b-8ba4-a105a71a5e6d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:43:51.882081 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:51.882059    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09c95c32-f60a-4f60-90ff-90f447247cce-workload-certs" pod "09c95c32-f60a-4f60-90ff-90f447247cce" (UID: "09c95c32-f60a-4f60-90ff-90f447247cce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:43:51.882260 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:51.882241    4364 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09c95c32-f60a-4f60-90ff-90f447247cce-workload-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:43:51.882348 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:51.882263    4364 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09c95c32-f60a-4f60-90ff-90f447247cce-istio-data\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:43:51.882348 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:51.882279    4364 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23116724-6a57-401b-8ba4-a105a71a5e6d-workload-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:43:51.882348 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:51.882295    4364 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c95c32-f60a-4f60-90ff-90f447247cce-workload-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:43:51.882348 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:51.882308    4364 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09c95c32-f60a-4f60-90ff-90f447247cce-istiod-ca-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:43:51.882348 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:51.882320    4364 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23116724-6a57-401b-8ba4-a105a71a5e6d-credential-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:43:51.882348 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:51.882333    4364 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23116724-6a57-401b-8ba4-a105a71a5e6d-workload-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:43:51.882724 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:51.882416    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23116724-6a57-401b-8ba4-a105a71a5e6d-istiod-ca-cert" pod "23116724-6a57-401b-8ba4-a105a71a5e6d" (UID: "23116724-6a57-401b-8ba4-a105a71a5e6d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:43:51.882841 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:51.882816    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09c95c32-f60a-4f60-90ff-90f447247cce-credential-socket" pod "09c95c32-f60a-4f60-90ff-90f447247cce" (UID: "09c95c32-f60a-4f60-90ff-90f447247cce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:43:51.884105 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:51.884067    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23116724-6a57-401b-8ba4-a105a71a5e6d-istio-data" pod "23116724-6a57-401b-8ba4-a105a71a5e6d" (UID: "23116724-6a57-401b-8ba4-a105a71a5e6d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:43:51.889133 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:51.889109    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23116724-6a57-401b-8ba4-a105a71a5e6d-istio-token" pod "23116724-6a57-401b-8ba4-a105a71a5e6d" (UID: "23116724-6a57-401b-8ba4-a105a71a5e6d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:43:51.889248 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:51.889203    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23116724-6a57-401b-8ba4-a105a71a5e6d-istio-envoy" pod "23116724-6a57-401b-8ba4-a105a71a5e6d" (UID: "23116724-6a57-401b-8ba4-a105a71a5e6d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:43:51.889336 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:51.889281    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09c95c32-f60a-4f60-90ff-90f447247cce-istio-envoy" pod "09c95c32-f60a-4f60-90ff-90f447247cce" (UID: "09c95c32-f60a-4f60-90ff-90f447247cce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:43:51.889449 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:51.889424    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09c95c32-f60a-4f60-90ff-90f447247cce-istio-podinfo" pod "09c95c32-f60a-4f60-90ff-90f447247cce" (UID: "09c95c32-f60a-4f60-90ff-90f447247cce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:43:51.889770 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:51.889426    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09c95c32-f60a-4f60-90ff-90f447247cce-istio-token" pod "09c95c32-f60a-4f60-90ff-90f447247cce" (UID: "09c95c32-f60a-4f60-90ff-90f447247cce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:43:51.889770 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:51.889523    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/23116724-6a57-401b-8ba4-a105a71a5e6d-istio-podinfo" pod "23116724-6a57-401b-8ba4-a105a71a5e6d" (UID: "23116724-6a57-401b-8ba4-a105a71a5e6d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:43:51.889770 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:51.889708    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23116724-6a57-401b-8ba4-a105a71a5e6d-kube-api-access-srhwh" pod "23116724-6a57-401b-8ba4-a105a71a5e6d" (UID: "23116724-6a57-401b-8ba4-a105a71a5e6d"). InnerVolumeSpecName "kube-api-access-srhwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:43:51.891347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:51.891322    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09c95c32-f60a-4f60-90ff-90f447247cce-kube-api-access-pwc5b" pod "09c95c32-f60a-4f60-90ff-90f447247cce" (UID: "09c95c32-f60a-4f60-90ff-90f447247cce"). InnerVolumeSpecName "kube-api-access-pwc5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:43:51.983484 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:51.983394    4364 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23116724-6a57-401b-8ba4-a105a71a5e6d-istio-data\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:43:51.983484 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:51.983428    4364 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23116724-6a57-401b-8ba4-a105a71a5e6d-istio-podinfo\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:43:51.983484 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:51.983441    4364 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23116724-6a57-401b-8ba4-a105a71a5e6d-istiod-ca-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:43:51.983484 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:51.983453    4364 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c95c32-f60a-4f60-90ff-90f447247cce-credential-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:43:51.983484 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:51.983467    4364 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09c95c32-f60a-4f60-90ff-90f447247cce-istio-podinfo\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:43:51.983484 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:51.983481    4364 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09c95c32-f60a-4f60-90ff-90f447247cce-istio-token\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:43:51.983831 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:51.983492    4364 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09c95c32-f60a-4f60-90ff-90f447247cce-istio-envoy\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:43:51.983831 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:51.983504    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwc5b\" (UniqueName: \"kubernetes.io/projected/09c95c32-f60a-4f60-90ff-90f447247cce-kube-api-access-pwc5b\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:43:51.983831 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:51.983517    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srhwh\" (UniqueName: \"kubernetes.io/projected/23116724-6a57-401b-8ba4-a105a71a5e6d-kube-api-access-srhwh\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:43:51.983831 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:51.983528    4364 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23116724-6a57-401b-8ba4-a105a71a5e6d-istio-envoy\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:43:51.983831 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:51.983539    4364 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23116724-6a57-401b-8ba4-a105a71a5e6d-istio-token\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:43:52.121924 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:52.121890    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj"]
Oct 30 13:43:52.125923 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:52.125898    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-8ssbj"]
Oct 30 13:43:52.782107 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:52.782042    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq"
Oct 30 13:43:52.782107 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:52.782071    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq" event={"ID":"09c95c32-f60a-4f60-90ff-90f447247cce","Type":"ContainerDied","Data":"6435d08519d683b06c67039264e3ab9e7860e19b8b4dd1b78c6a17a62b2a5acb"}
Oct 30 13:43:52.782322 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:52.782131    4364 scope.go:122] "RemoveContainer" containerID="a526058e96bf8fe46f5fac380c980be2b9991ed7f86bf8b2825b56a417636a73"
Oct 30 13:43:52.860332 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:52.860303    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq"]
Oct 30 13:43:52.872845 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:52.872746    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-drwcq"]
Oct 30 13:43:52.950424 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:52.950385    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg"
Oct 30 13:43:52.971024 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:52.970997    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09c95c32-f60a-4f60-90ff-90f447247cce" path="/var/lib/kubelet/pods/09c95c32-f60a-4f60-90ff-90f447247cce/volumes"
Oct 30 13:43:52.971501 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:52.971487    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23116724-6a57-401b-8ba4-a105a71a5e6d" path="/var/lib/kubelet/pods/23116724-6a57-401b-8ba4-a105a71a5e6d/volumes"
Oct 30 13:43:52.995560 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:52.995524    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ce05a81-71f3-4dab-96e1-44c9ea628a7b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ce05a81-71f3-4dab-96e1-44c9ea628a7b-istio-podinfo\") pod \"5ce05a81-71f3-4dab-96e1-44c9ea628a7b\" (UID: \"5ce05a81-71f3-4dab-96e1-44c9ea628a7b\") "
Oct 30 13:43:52.995763 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:52.995664    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ce05a81-71f3-4dab-96e1-44c9ea628a7b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ce05a81-71f3-4dab-96e1-44c9ea628a7b-istio-envoy\") pod \"5ce05a81-71f3-4dab-96e1-44c9ea628a7b\" (UID: \"5ce05a81-71f3-4dab-96e1-44c9ea628a7b\") "
Oct 30 13:43:52.999671 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:52.996152    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ce05a81-71f3-4dab-96e1-44c9ea628a7b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ce05a81-71f3-4dab-96e1-44c9ea628a7b-istio-data\") pod \"5ce05a81-71f3-4dab-96e1-44c9ea628a7b\" (UID: \"5ce05a81-71f3-4dab-96e1-44c9ea628a7b\") "
Oct 30 13:43:52.999671 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:52.996431    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ce05a81-71f3-4dab-96e1-44c9ea628a7b-istio-data" pod "5ce05a81-71f3-4dab-96e1-44c9ea628a7b" (UID: "5ce05a81-71f3-4dab-96e1-44c9ea628a7b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:43:52.999671 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:52.996536    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ce05a81-71f3-4dab-96e1-44c9ea628a7b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ce05a81-71f3-4dab-96e1-44c9ea628a7b-workload-certs\") pod \"5ce05a81-71f3-4dab-96e1-44c9ea628a7b\" (UID: \"5ce05a81-71f3-4dab-96e1-44c9ea628a7b\") "
Oct 30 13:43:52.999671 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:52.996805    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ce05a81-71f3-4dab-96e1-44c9ea628a7b-workload-certs" pod "5ce05a81-71f3-4dab-96e1-44c9ea628a7b" (UID: "5ce05a81-71f3-4dab-96e1-44c9ea628a7b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:43:52.999671 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:52.996887    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ce05a81-71f3-4dab-96e1-44c9ea628a7b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ce05a81-71f3-4dab-96e1-44c9ea628a7b-credential-socket\") pod \"5ce05a81-71f3-4dab-96e1-44c9ea628a7b\" (UID: \"5ce05a81-71f3-4dab-96e1-44c9ea628a7b\") "
Oct 30 13:43:52.999671 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:52.998576    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ce05a81-71f3-4dab-96e1-44c9ea628a7b-workload-socket" pod "5ce05a81-71f3-4dab-96e1-44c9ea628a7b" (UID: "5ce05a81-71f3-4dab-96e1-44c9ea628a7b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:43:52.999671 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:52.998845    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ce05a81-71f3-4dab-96e1-44c9ea628a7b-credential-socket" pod "5ce05a81-71f3-4dab-96e1-44c9ea628a7b" (UID: "5ce05a81-71f3-4dab-96e1-44c9ea628a7b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:43:52.999671 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:52.999413    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ce05a81-71f3-4dab-96e1-44c9ea628a7b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ce05a81-71f3-4dab-96e1-44c9ea628a7b-workload-socket\") pod \"5ce05a81-71f3-4dab-96e1-44c9ea628a7b\" (UID: \"5ce05a81-71f3-4dab-96e1-44c9ea628a7b\") "
Oct 30 13:43:52.999671 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:52.999529    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ce05a81-71f3-4dab-96e1-44c9ea628a7b-kube-api-access-gzdpj\" (UniqueName: \"kubernetes.io/projected/5ce05a81-71f3-4dab-96e1-44c9ea628a7b-kube-api-access-gzdpj\") pod \"5ce05a81-71f3-4dab-96e1-44c9ea628a7b\" (UID: \"5ce05a81-71f3-4dab-96e1-44c9ea628a7b\") "
Oct 30 13:43:52.999671 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:52.999621    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ce05a81-71f3-4dab-96e1-44c9ea628a7b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ce05a81-71f3-4dab-96e1-44c9ea628a7b-istiod-ca-cert\") pod \"5ce05a81-71f3-4dab-96e1-44c9ea628a7b\" (UID: \"5ce05a81-71f3-4dab-96e1-44c9ea628a7b\") "
Oct 30 13:43:53.002161 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:52.999714    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ce05a81-71f3-4dab-96e1-44c9ea628a7b-istio-token\" (UniqueName: \"kubernetes.io/projected/5ce05a81-71f3-4dab-96e1-44c9ea628a7b-istio-token\") pod \"5ce05a81-71f3-4dab-96e1-44c9ea628a7b\" (UID: \"5ce05a81-71f3-4dab-96e1-44c9ea628a7b\") "
Oct 30 13:43:53.002161 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:53.000183    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ce05a81-71f3-4dab-96e1-44c9ea628a7b-istiod-ca-cert" pod "5ce05a81-71f3-4dab-96e1-44c9ea628a7b" (UID: "5ce05a81-71f3-4dab-96e1-44c9ea628a7b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:43:53.002800 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:53.002774    4364 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ce05a81-71f3-4dab-96e1-44c9ea628a7b-istio-data\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:43:53.002899 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:53.002804    4364 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ce05a81-71f3-4dab-96e1-44c9ea628a7b-workload-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:43:53.002899 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:53.002818    4364 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ce05a81-71f3-4dab-96e1-44c9ea628a7b-credential-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:43:53.002899 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:53.002831    4364 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ce05a81-71f3-4dab-96e1-44c9ea628a7b-workload-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:43:53.002899 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:53.002844    4364 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ce05a81-71f3-4dab-96e1-44c9ea628a7b-istiod-ca-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:43:53.005384 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:53.005342    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ce05a81-71f3-4dab-96e1-44c9ea628a7b-kube-api-access-gzdpj" pod "5ce05a81-71f3-4dab-96e1-44c9ea628a7b" (UID: "5ce05a81-71f3-4dab-96e1-44c9ea628a7b"). InnerVolumeSpecName "kube-api-access-gzdpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:43:53.007240 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:53.007213    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ce05a81-71f3-4dab-96e1-44c9ea628a7b-istio-envoy" pod "5ce05a81-71f3-4dab-96e1-44c9ea628a7b" (UID: "5ce05a81-71f3-4dab-96e1-44c9ea628a7b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:43:53.007719 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:53.007693    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ce05a81-71f3-4dab-96e1-44c9ea628a7b-istio-token" pod "5ce05a81-71f3-4dab-96e1-44c9ea628a7b" (UID: "5ce05a81-71f3-4dab-96e1-44c9ea628a7b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:43:53.016774 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:53.016738    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ce05a81-71f3-4dab-96e1-44c9ea628a7b-istio-podinfo" pod "5ce05a81-71f3-4dab-96e1-44c9ea628a7b" (UID: "5ce05a81-71f3-4dab-96e1-44c9ea628a7b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:43:53.104113 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:53.104030    4364 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ce05a81-71f3-4dab-96e1-44c9ea628a7b-istio-podinfo\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:43:53.104113 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:53.104062    4364 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ce05a81-71f3-4dab-96e1-44c9ea628a7b-istio-envoy\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:43:53.104113 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:53.104072    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzdpj\" (UniqueName: \"kubernetes.io/projected/5ce05a81-71f3-4dab-96e1-44c9ea628a7b-kube-api-access-gzdpj\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:43:53.104113 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:53.104083    4364 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ce05a81-71f3-4dab-96e1-44c9ea628a7b-istio-token\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:43:53.652631 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:53.652586    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-fdd4d4-78689f8d4-5hq2s"]
Oct 30 13:43:53.794420 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:53.794325    4364 generic.go:356] "Generic (PLEG): container finished" podID="5ce05a81-71f3-4dab-96e1-44c9ea628a7b" containerID="b3eb44114eb6eccd60b6eba100561e0a8be2fce1da4d1d51712da581e596d26b" exitCode=0
Oct 30 13:43:53.794609 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:53.794416    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg"
Oct 30 13:43:53.794609 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:53.794467    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg" event={"ID":"5ce05a81-71f3-4dab-96e1-44c9ea628a7b","Type":"ContainerDied","Data":"b3eb44114eb6eccd60b6eba100561e0a8be2fce1da4d1d51712da581e596d26b"}
Oct 30 13:43:53.794609 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:53.794515    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg" event={"ID":"5ce05a81-71f3-4dab-96e1-44c9ea628a7b","Type":"ContainerDied","Data":"8ac9dbf8245846554a0c15ab00ba290546298ddc79464e7e3e759d1a9ff693c9"}
Oct 30 13:43:53.794609 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:53.794538    4364 scope.go:122] "RemoveContainer" containerID="b3eb44114eb6eccd60b6eba100561e0a8be2fce1da4d1d51712da581e596d26b"
Oct 30 13:43:53.805263 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:53.805239    4364 scope.go:122] "RemoveContainer" containerID="67063f13ec7a249a230bf997909730d9f8e56e8c0bde3ab950063eef0a4ac3b5"
Oct 30 13:43:53.817313 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:53.817272    4364 scope.go:122] "RemoveContainer" containerID="b3eb44114eb6eccd60b6eba100561e0a8be2fce1da4d1d51712da581e596d26b"
Oct 30 13:43:53.818837 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:43:53.818770    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3eb44114eb6eccd60b6eba100561e0a8be2fce1da4d1d51712da581e596d26b\": container with ID starting with b3eb44114eb6eccd60b6eba100561e0a8be2fce1da4d1d51712da581e596d26b not found: ID does not exist" containerID="b3eb44114eb6eccd60b6eba100561e0a8be2fce1da4d1d51712da581e596d26b"
Oct 30 13:43:53.818837 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:53.818812    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3eb44114eb6eccd60b6eba100561e0a8be2fce1da4d1d51712da581e596d26b"} err="failed to get container status \"b3eb44114eb6eccd60b6eba100561e0a8be2fce1da4d1d51712da581e596d26b\": rpc error: code = NotFound desc = could not find container \"b3eb44114eb6eccd60b6eba100561e0a8be2fce1da4d1d51712da581e596d26b\": container with ID starting with b3eb44114eb6eccd60b6eba100561e0a8be2fce1da4d1d51712da581e596d26b not found: ID does not exist"
Oct 30 13:43:53.818837 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:53.818835    4364 scope.go:122] "RemoveContainer" containerID="67063f13ec7a249a230bf997909730d9f8e56e8c0bde3ab950063eef0a4ac3b5"
Oct 30 13:43:53.819151 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:43:53.819133    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67063f13ec7a249a230bf997909730d9f8e56e8c0bde3ab950063eef0a4ac3b5\": container with ID starting with 67063f13ec7a249a230bf997909730d9f8e56e8c0bde3ab950063eef0a4ac3b5 not found: ID does not exist" containerID="67063f13ec7a249a230bf997909730d9f8e56e8c0bde3ab950063eef0a4ac3b5"
Oct 30 13:43:53.819235 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:53.819156    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67063f13ec7a249a230bf997909730d9f8e56e8c0bde3ab950063eef0a4ac3b5"} err="failed to get container status \"67063f13ec7a249a230bf997909730d9f8e56e8c0bde3ab950063eef0a4ac3b5\": rpc error: code = NotFound desc = could not find container \"67063f13ec7a249a230bf997909730d9f8e56e8c0bde3ab950063eef0a4ac3b5\": container with ID starting with 67063f13ec7a249a230bf997909730d9f8e56e8c0bde3ab950063eef0a4ac3b5 not found: ID does not exist"
Oct 30 13:43:53.850363 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:53.850335    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg"]
Oct 30 13:43:53.861993 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:53.861972    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-b4fb25-openshift-default-767944bdfc-lqkpg"]
Oct 30 13:43:54.971360 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:54.971328    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ce05a81-71f3-4dab-96e1-44c9ea628a7b" path="/var/lib/kubelet/pods/5ce05a81-71f3-4dab-96e1-44c9ea628a7b/volumes"
Oct 30 13:43:57.388807 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:57.388769    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-ff4285-6b8554d544-6flwq"]
Oct 30 13:43:57.415419 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:57.415388    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ff4285-59fc768575-tgw8j"]
Oct 30 13:43:57.879452 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:57.879418    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b0ea9-5f686c7795-7527w" podUID="c2597429-b2f7-4263-b715-19410183a9a7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:43:58.653674 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:58.653634    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-fdd4d4-78689f8d4-5hq2s" podUID="43de3339-76dc-44c5-9ce6-9e592b0fceba" containerName="payload-processing" containerID="cri-o://9e6c937eb3fec06045d356b312c08dcf5e7bbed23903774927ef705de445f87d" gracePeriod=25
Oct 30 13:43:58.850908 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:58.841518    4364 generic.go:356] "Generic (PLEG): container finished" podID="43de3339-76dc-44c5-9ce6-9e592b0fceba" containerID="9e6c937eb3fec06045d356b312c08dcf5e7bbed23903774927ef705de445f87d" exitCode=0
Oct 30 13:43:58.850908 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:58.841829    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-fdd4d4-78689f8d4-5hq2s" event={"ID":"43de3339-76dc-44c5-9ce6-9e592b0fceba","Type":"ContainerDied","Data":"9e6c937eb3fec06045d356b312c08dcf5e7bbed23903774927ef705de445f87d"}
Oct 30 13:43:58.933942 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:58.933910    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-fdd4d4-78689f8d4-5hq2s"
Oct 30 13:43:58.994287 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:58.994251    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43de3339-76dc-44c5-9ce6-9e592b0fceba-kube-api-access-v2qxt\" (UniqueName: \"kubernetes.io/projected/43de3339-76dc-44c5-9ce6-9e592b0fceba-kube-api-access-v2qxt\") pod \"43de3339-76dc-44c5-9ce6-9e592b0fceba\" (UID: \"43de3339-76dc-44c5-9ce6-9e592b0fceba\") "
Oct 30 13:43:58.994466 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:58.994302    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43de3339-76dc-44c5-9ce6-9e592b0fceba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43de3339-76dc-44c5-9ce6-9e592b0fceba-plugins-config-volume\") pod \"43de3339-76dc-44c5-9ce6-9e592b0fceba\" (UID: \"43de3339-76dc-44c5-9ce6-9e592b0fceba\") "
Oct 30 13:43:58.994940 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:58.994912    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43de3339-76dc-44c5-9ce6-9e592b0fceba-plugins-config-volume" pod "43de3339-76dc-44c5-9ce6-9e592b0fceba" (UID: "43de3339-76dc-44c5-9ce6-9e592b0fceba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:43:58.996778 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:58.996752    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43de3339-76dc-44c5-9ce6-9e592b0fceba-kube-api-access-v2qxt" pod "43de3339-76dc-44c5-9ce6-9e592b0fceba" (UID: "43de3339-76dc-44c5-9ce6-9e592b0fceba"). InnerVolumeSpecName "kube-api-access-v2qxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:43:59.095908 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:59.095873    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2qxt\" (UniqueName: \"kubernetes.io/projected/43de3339-76dc-44c5-9ce6-9e592b0fceba-kube-api-access-v2qxt\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:43:59.095908 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:59.095902    4364 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43de3339-76dc-44c5-9ce6-9e592b0fceba-plugins-config-volume\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:43:59.850615 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:59.850576    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-fdd4d4-78689f8d4-5hq2s"
Oct 30 13:43:59.851084 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:59.850628    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-fdd4d4-78689f8d4-5hq2s" event={"ID":"43de3339-76dc-44c5-9ce6-9e592b0fceba","Type":"ContainerDied","Data":"b4ad05e820f748ec4613574a358326c37fbc3edee4c8b88d995ce7e7db0fe443"}
Oct 30 13:43:59.851084 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:59.850712    4364 scope.go:122] "RemoveContainer" containerID="9e6c937eb3fec06045d356b312c08dcf5e7bbed23903774927ef705de445f87d"
Oct 30 13:43:59.905681 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:59.905645    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-fdd4d4-78689f8d4-5hq2s"]
Oct 30 13:43:59.917416 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:43:59.917389    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-fdd4d4-78689f8d4-5hq2s"]
Oct 30 13:44:00.976431 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:00.976392    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43de3339-76dc-44c5-9ce6-9e592b0fceba" path="/var/lib/kubelet/pods/43de3339-76dc-44c5-9ce6-9e592b0fceba/volumes"
Oct 30 13:44:02.389565 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:02.389528    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ff4285-6b8554d544-6flwq" podUID="43e6fe99-7d56-417f-b9db-1d41a24340d2" containerName="payload-processing" containerID="cri-o://e7763a4801ab768e54a01a08625bda1aed3ea3e1c6c4323eee61c82fa52e72cd" gracePeriod=25
Oct 30 13:44:02.416322 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:02.416263    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ff4285-59fc768575-tgw8j" podUID="e5c6ed73-3541-4945-8d38-cb289a4e05f8" containerName="payload-pre-processing" containerID="cri-o://40100748bc3f22e0b545a30cdee796a670965ce2fadcc87c0aba5ab0d65622fb" gracePeriod=25
Oct 30 13:44:04.595824 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:04.595767    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b0ea9-5f686c7795-7527w"
Oct 30 13:44:04.715113 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:04.715085    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:44:04.787260 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:04.787231    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2597429-b2f7-4263-b715-19410183a9a7-kube-api-access-l2dwb\" (UniqueName: \"kubernetes.io/projected/c2597429-b2f7-4263-b715-19410183a9a7-kube-api-access-l2dwb\") pod \"c2597429-b2f7-4263-b715-19410183a9a7\" (UID: \"c2597429-b2f7-4263-b715-19410183a9a7\") "
Oct 30 13:44:04.787458 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:04.787416    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2597429-b2f7-4263-b715-19410183a9a7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2597429-b2f7-4263-b715-19410183a9a7-plugins-config-volume\") pod \"c2597429-b2f7-4263-b715-19410183a9a7\" (UID: \"c2597429-b2f7-4263-b715-19410183a9a7\") "
Oct 30 13:44:04.787779 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:04.787756    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2597429-b2f7-4263-b715-19410183a9a7-plugins-config-volume" pod "c2597429-b2f7-4263-b715-19410183a9a7" (UID: "c2597429-b2f7-4263-b715-19410183a9a7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:44:04.787896 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:04.787884    4364 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2597429-b2f7-4263-b715-19410183a9a7-plugins-config-volume\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:04.789284 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:04.789258    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2597429-b2f7-4263-b715-19410183a9a7-kube-api-access-l2dwb" pod "c2597429-b2f7-4263-b715-19410183a9a7" (UID: "c2597429-b2f7-4263-b715-19410183a9a7"). InnerVolumeSpecName "kube-api-access-l2dwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:44:04.887431 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:04.887394    4364 generic.go:356] "Generic (PLEG): container finished" podID="c2597429-b2f7-4263-b715-19410183a9a7" containerID="f36d0c3f3e10c89bf508565ad24228081b8d4ed406030f13d9f89d110bbf498a" exitCode=0
Oct 30 13:44:04.887630 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:04.887435    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b0ea9-5f686c7795-7527w"
Oct 30 13:44:04.887630 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:04.887475    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b0ea9-5f686c7795-7527w" event={"ID":"c2597429-b2f7-4263-b715-19410183a9a7","Type":"ContainerDied","Data":"f36d0c3f3e10c89bf508565ad24228081b8d4ed406030f13d9f89d110bbf498a"}
Oct 30 13:44:04.887630 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:04.887536    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b0ea9-5f686c7795-7527w" event={"ID":"c2597429-b2f7-4263-b715-19410183a9a7","Type":"ContainerDied","Data":"353589a7a05af7db6d7cc40253ccc071f6db919e25a9c9709dd435db9d1befe3"}
Oct 30 13:44:04.887630 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:04.887561    4364 scope.go:122] "RemoveContainer" containerID="f36d0c3f3e10c89bf508565ad24228081b8d4ed406030f13d9f89d110bbf498a"
Oct 30 13:44:04.888826 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:04.888797    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2dwb\" (UniqueName: \"kubernetes.io/projected/c2597429-b2f7-4263-b715-19410183a9a7-kube-api-access-l2dwb\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:04.898416 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:04.898397    4364 scope.go:122] "RemoveContainer" containerID="f36d0c3f3e10c89bf508565ad24228081b8d4ed406030f13d9f89d110bbf498a"
Oct 30 13:44:04.898680 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:44:04.898659    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f36d0c3f3e10c89bf508565ad24228081b8d4ed406030f13d9f89d110bbf498a\": container with ID starting with f36d0c3f3e10c89bf508565ad24228081b8d4ed406030f13d9f89d110bbf498a not found: ID does not exist" containerID="f36d0c3f3e10c89bf508565ad24228081b8d4ed406030f13d9f89d110bbf498a"
Oct 30 13:44:04.898741 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:04.898687    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f36d0c3f3e10c89bf508565ad24228081b8d4ed406030f13d9f89d110bbf498a"} err="failed to get container status \"f36d0c3f3e10c89bf508565ad24228081b8d4ed406030f13d9f89d110bbf498a\": rpc error: code = NotFound desc = could not find container \"f36d0c3f3e10c89bf508565ad24228081b8d4ed406030f13d9f89d110bbf498a\": container with ID starting with f36d0c3f3e10c89bf508565ad24228081b8d4ed406030f13d9f89d110bbf498a not found: ID does not exist"
Oct 30 13:44:04.947085 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:04.947059    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b0ea9-5f686c7795-7527w"]
Oct 30 13:44:04.953015 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:04.952993    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b0ea9-5f686c7795-7527w"]
Oct 30 13:44:04.970263 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:04.970233    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2597429-b2f7-4263-b715-19410183a9a7" path="/var/lib/kubelet/pods/c2597429-b2f7-4263-b715-19410183a9a7/volumes"
Oct 30 13:44:07.431677 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:07.431631    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ff4285-59fc768575-tgw8j" podUID="e5c6ed73-3541-4945-8d38-cb289a4e05f8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:44:07.432085 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:07.431676    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ff4285-6b8554d544-6flwq" podUID="43e6fe99-7d56-417f-b9db-1d41a24340d2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:44:09.555213 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:09.555170    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9"]
Oct 30 13:44:09.555656 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:09.555401    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9" podUID="9a967f02-7c93-484e-9bd4-97e1a4f5fdf5" containerName="istio-proxy" containerID="cri-o://6bd64ed3e7f7b1c54ada326b0f8b22ca74165c53790a0937ddafdccd6d34428e" gracePeriod=30
Oct 30 13:44:11.505125 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:44:11.505085    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:44:12.121560 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:12.121527    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-fca422-767b78c8cb-fzvl2"]
Oct 30 13:44:12.121792 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:12.121769    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fca422-767b78c8cb-fzvl2" podUID="80f1162e-fa26-45b9-8016-c4812edb5bad" containerName="maas-api" containerID="cri-o://3953071e1c98b8fa58c44729b00b66adaa01803c80ab1c2778cc33c873e9ec5c" gracePeriod=30
Oct 30 13:44:12.398686 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:12.398581    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fca422-767b78c8cb-fzvl2"
Oct 30 13:44:12.426164 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:12.426138    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80f1162e-fa26-45b9-8016-c4812edb5bad-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80f1162e-fa26-45b9-8016-c4812edb5bad-maas-api-tls\") pod \"80f1162e-fa26-45b9-8016-c4812edb5bad\" (UID: \"80f1162e-fa26-45b9-8016-c4812edb5bad\") "
Oct 30 13:44:12.426297 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:12.426187    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80f1162e-fa26-45b9-8016-c4812edb5bad-metrics-certs\" (UniqueName: \"kubernetes.io/secret/80f1162e-fa26-45b9-8016-c4812edb5bad-metrics-certs\") pod \"80f1162e-fa26-45b9-8016-c4812edb5bad\" (UID: \"80f1162e-fa26-45b9-8016-c4812edb5bad\") "
Oct 30 13:44:12.426297 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:12.426234    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80f1162e-fa26-45b9-8016-c4812edb5bad-kube-api-access-xvcw7\" (UniqueName: \"kubernetes.io/projected/80f1162e-fa26-45b9-8016-c4812edb5bad-kube-api-access-xvcw7\") pod \"80f1162e-fa26-45b9-8016-c4812edb5bad\" (UID: \"80f1162e-fa26-45b9-8016-c4812edb5bad\") "
Oct 30 13:44:12.426398 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:12.426307    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80f1162e-fa26-45b9-8016-c4812edb5bad-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80f1162e-fa26-45b9-8016-c4812edb5bad-openshift-service-ca\") pod \"80f1162e-fa26-45b9-8016-c4812edb5bad\" (UID: \"80f1162e-fa26-45b9-8016-c4812edb5bad\") "
Oct 30 13:44:12.426782 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:12.426748    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80f1162e-fa26-45b9-8016-c4812edb5bad-openshift-service-ca" pod "80f1162e-fa26-45b9-8016-c4812edb5bad" (UID: "80f1162e-fa26-45b9-8016-c4812edb5bad"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:44:12.427087 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:12.427066    4364 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80f1162e-fa26-45b9-8016-c4812edb5bad-openshift-service-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:12.428173 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:12.428149    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80f1162e-fa26-45b9-8016-c4812edb5bad-metrics-certs" pod "80f1162e-fa26-45b9-8016-c4812edb5bad" (UID: "80f1162e-fa26-45b9-8016-c4812edb5bad"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:44:12.428332 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:12.428316    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80f1162e-fa26-45b9-8016-c4812edb5bad-kube-api-access-xvcw7" pod "80f1162e-fa26-45b9-8016-c4812edb5bad" (UID: "80f1162e-fa26-45b9-8016-c4812edb5bad"). InnerVolumeSpecName "kube-api-access-xvcw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:44:12.429080 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:12.428391    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80f1162e-fa26-45b9-8016-c4812edb5bad-maas-api-tls" pod "80f1162e-fa26-45b9-8016-c4812edb5bad" (UID: "80f1162e-fa26-45b9-8016-c4812edb5bad"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:44:12.528955 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:12.528922    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80f1162e-fa26-45b9-8016-c4812edb5bad-maas-api-tls\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:12.528955 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:12.528951    4364 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80f1162e-fa26-45b9-8016-c4812edb5bad-metrics-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:12.528955 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:12.528963    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvcw7\" (UniqueName: \"kubernetes.io/projected/80f1162e-fa26-45b9-8016-c4812edb5bad-kube-api-access-xvcw7\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:12.991703 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:12.991671    4364 generic.go:356] "Generic (PLEG): container finished" podID="80f1162e-fa26-45b9-8016-c4812edb5bad" containerID="3953071e1c98b8fa58c44729b00b66adaa01803c80ab1c2778cc33c873e9ec5c" exitCode=0
Oct 30 13:44:12.991842 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:12.991713    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fca422-767b78c8cb-fzvl2"
Oct 30 13:44:12.991842 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:12.991749    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fca422-767b78c8cb-fzvl2" event={"ID":"80f1162e-fa26-45b9-8016-c4812edb5bad","Type":"ContainerDied","Data":"3953071e1c98b8fa58c44729b00b66adaa01803c80ab1c2778cc33c873e9ec5c"}
Oct 30 13:44:12.991842 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:12.991798    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fca422-767b78c8cb-fzvl2" event={"ID":"80f1162e-fa26-45b9-8016-c4812edb5bad","Type":"ContainerDied","Data":"5a29604b69f7537da76437079662a747381569e3f6134a95744c79f8afe7bee4"}
Oct 30 13:44:12.991842 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:12.991821    4364 scope.go:122] "RemoveContainer" containerID="3953071e1c98b8fa58c44729b00b66adaa01803c80ab1c2778cc33c873e9ec5c"
Oct 30 13:44:13.002174 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:13.002151    4364 scope.go:122] "RemoveContainer" containerID="3953071e1c98b8fa58c44729b00b66adaa01803c80ab1c2778cc33c873e9ec5c"
Oct 30 13:44:13.002442 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:44:13.002423    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3953071e1c98b8fa58c44729b00b66adaa01803c80ab1c2778cc33c873e9ec5c\": container with ID starting with 3953071e1c98b8fa58c44729b00b66adaa01803c80ab1c2778cc33c873e9ec5c not found: ID does not exist" containerID="3953071e1c98b8fa58c44729b00b66adaa01803c80ab1c2778cc33c873e9ec5c"
Oct 30 13:44:13.002499 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:13.002449    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3953071e1c98b8fa58c44729b00b66adaa01803c80ab1c2778cc33c873e9ec5c"} err="failed to get container status \"3953071e1c98b8fa58c44729b00b66adaa01803c80ab1c2778cc33c873e9ec5c\": rpc error: code = NotFound desc = could not find container \"3953071e1c98b8fa58c44729b00b66adaa01803c80ab1c2778cc33c873e9ec5c\": container with ID starting with 3953071e1c98b8fa58c44729b00b66adaa01803c80ab1c2778cc33c873e9ec5c not found: ID does not exist"
Oct 30 13:44:13.097242 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:13.097217    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-fca422-767b78c8cb-fzvl2"]
Oct 30 13:44:13.110909 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:13.110885    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-fca422-767b78c8cb-fzvl2"]
Oct 30 13:44:13.401721 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:13.401687    4364 patch_prober.go:29] interesting pod/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:44:13.401876 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:13.401742    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9" podUID="9a967f02-7c93-484e-9bd4-97e1a4f5fdf5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:44:14.825938 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:14.825877    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9"
Oct 30 13:44:14.864840 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:14.864811    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a967f02-7c93-484e-9bd4-97e1a4f5fdf5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a967f02-7c93-484e-9bd4-97e1a4f5fdf5-istio-podinfo\") pod \"9a967f02-7c93-484e-9bd4-97e1a4f5fdf5\" (UID: \"9a967f02-7c93-484e-9bd4-97e1a4f5fdf5\") "
Oct 30 13:44:14.865000 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:14.864854    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a967f02-7c93-484e-9bd4-97e1a4f5fdf5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a967f02-7c93-484e-9bd4-97e1a4f5fdf5-istiod-ca-cert\") pod \"9a967f02-7c93-484e-9bd4-97e1a4f5fdf5\" (UID: \"9a967f02-7c93-484e-9bd4-97e1a4f5fdf5\") "
Oct 30 13:44:14.865000 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:14.864918    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a967f02-7c93-484e-9bd4-97e1a4f5fdf5-kube-api-access-xdkjw\" (UniqueName: \"kubernetes.io/projected/9a967f02-7c93-484e-9bd4-97e1a4f5fdf5-kube-api-access-xdkjw\") pod \"9a967f02-7c93-484e-9bd4-97e1a4f5fdf5\" (UID: \"9a967f02-7c93-484e-9bd4-97e1a4f5fdf5\") "
Oct 30 13:44:14.865130 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:14.865037    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a967f02-7c93-484e-9bd4-97e1a4f5fdf5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a967f02-7c93-484e-9bd4-97e1a4f5fdf5-istio-data\") pod \"9a967f02-7c93-484e-9bd4-97e1a4f5fdf5\" (UID: \"9a967f02-7c93-484e-9bd4-97e1a4f5fdf5\") "
Oct 30 13:44:14.865201 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:14.865142    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a967f02-7c93-484e-9bd4-97e1a4f5fdf5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a967f02-7c93-484e-9bd4-97e1a4f5fdf5-workload-certs\") pod \"9a967f02-7c93-484e-9bd4-97e1a4f5fdf5\" (UID: \"9a967f02-7c93-484e-9bd4-97e1a4f5fdf5\") "
Oct 30 13:44:14.865201 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:14.865177    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a967f02-7c93-484e-9bd4-97e1a4f5fdf5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a967f02-7c93-484e-9bd4-97e1a4f5fdf5-istio-envoy\") pod \"9a967f02-7c93-484e-9bd4-97e1a4f5fdf5\" (UID: \"9a967f02-7c93-484e-9bd4-97e1a4f5fdf5\") "
Oct 30 13:44:14.865323 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:14.865299    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a967f02-7c93-484e-9bd4-97e1a4f5fdf5-istiod-ca-cert" pod "9a967f02-7c93-484e-9bd4-97e1a4f5fdf5" (UID: "9a967f02-7c93-484e-9bd4-97e1a4f5fdf5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:44:14.865387 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:14.865368    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a967f02-7c93-484e-9bd4-97e1a4f5fdf5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a967f02-7c93-484e-9bd4-97e1a4f5fdf5-workload-socket\") pod \"9a967f02-7c93-484e-9bd4-97e1a4f5fdf5\" (UID: \"9a967f02-7c93-484e-9bd4-97e1a4f5fdf5\") "
Oct 30 13:44:14.865583 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:14.865538    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a967f02-7c93-484e-9bd4-97e1a4f5fdf5-workload-certs" pod "9a967f02-7c93-484e-9bd4-97e1a4f5fdf5" (UID: "9a967f02-7c93-484e-9bd4-97e1a4f5fdf5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:14.865583 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:14.865556    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a967f02-7c93-484e-9bd4-97e1a4f5fdf5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a967f02-7c93-484e-9bd4-97e1a4f5fdf5-credential-socket\") pod \"9a967f02-7c93-484e-9bd4-97e1a4f5fdf5\" (UID: \"9a967f02-7c93-484e-9bd4-97e1a4f5fdf5\") "
Oct 30 13:44:14.865776 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:14.865664    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a967f02-7c93-484e-9bd4-97e1a4f5fdf5-istio-token\" (UniqueName: \"kubernetes.io/projected/9a967f02-7c93-484e-9bd4-97e1a4f5fdf5-istio-token\") pod \"9a967f02-7c93-484e-9bd4-97e1a4f5fdf5\" (UID: \"9a967f02-7c93-484e-9bd4-97e1a4f5fdf5\") "
Oct 30 13:44:14.865776 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:14.865758    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a967f02-7c93-484e-9bd4-97e1a4f5fdf5-credential-socket" pod "9a967f02-7c93-484e-9bd4-97e1a4f5fdf5" (UID: "9a967f02-7c93-484e-9bd4-97e1a4f5fdf5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:14.866301 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:14.866253    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a967f02-7c93-484e-9bd4-97e1a4f5fdf5-workload-socket" pod "9a967f02-7c93-484e-9bd4-97e1a4f5fdf5" (UID: "9a967f02-7c93-484e-9bd4-97e1a4f5fdf5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:14.866479 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:14.866309    4364 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a967f02-7c93-484e-9bd4-97e1a4f5fdf5-credential-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:14.866650 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:14.866625    4364 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a967f02-7c93-484e-9bd4-97e1a4f5fdf5-istiod-ca-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:14.866773 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:14.866668    4364 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a967f02-7c93-484e-9bd4-97e1a4f5fdf5-workload-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:14.866773 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:14.866500    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a967f02-7c93-484e-9bd4-97e1a4f5fdf5-istio-data" pod "9a967f02-7c93-484e-9bd4-97e1a4f5fdf5" (UID: "9a967f02-7c93-484e-9bd4-97e1a4f5fdf5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:14.867590 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:14.867550    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a967f02-7c93-484e-9bd4-97e1a4f5fdf5-kube-api-access-xdkjw" pod "9a967f02-7c93-484e-9bd4-97e1a4f5fdf5" (UID: "9a967f02-7c93-484e-9bd4-97e1a4f5fdf5"). InnerVolumeSpecName "kube-api-access-xdkjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:44:14.867729 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:14.867656    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a967f02-7c93-484e-9bd4-97e1a4f5fdf5-istio-podinfo" pod "9a967f02-7c93-484e-9bd4-97e1a4f5fdf5" (UID: "9a967f02-7c93-484e-9bd4-97e1a4f5fdf5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:44:14.867729 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:14.867663    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a967f02-7c93-484e-9bd4-97e1a4f5fdf5-istio-envoy" pod "9a967f02-7c93-484e-9bd4-97e1a4f5fdf5" (UID: "9a967f02-7c93-484e-9bd4-97e1a4f5fdf5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:14.868166 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:14.868145    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a967f02-7c93-484e-9bd4-97e1a4f5fdf5-istio-token" pod "9a967f02-7c93-484e-9bd4-97e1a4f5fdf5" (UID: "9a967f02-7c93-484e-9bd4-97e1a4f5fdf5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:44:14.968467 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:14.968431    4364 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a967f02-7c93-484e-9bd4-97e1a4f5fdf5-istio-envoy\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:14.968467 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:14.968468    4364 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a967f02-7c93-484e-9bd4-97e1a4f5fdf5-workload-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:14.968712 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:14.968482    4364 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a967f02-7c93-484e-9bd4-97e1a4f5fdf5-istio-token\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:14.968712 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:14.968496    4364 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a967f02-7c93-484e-9bd4-97e1a4f5fdf5-istio-podinfo\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:14.968712 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:14.968509    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdkjw\" (UniqueName: \"kubernetes.io/projected/9a967f02-7c93-484e-9bd4-97e1a4f5fdf5-kube-api-access-xdkjw\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:14.968712 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:14.968522    4364 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a967f02-7c93-484e-9bd4-97e1a4f5fdf5-istio-data\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:14.972869 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:14.972838    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80f1162e-fa26-45b9-8016-c4812edb5bad" path="/var/lib/kubelet/pods/80f1162e-fa26-45b9-8016-c4812edb5bad/volumes"
Oct 30 13:44:14.973381 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:14.973366    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-d833bd/distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74"]
Oct 30 13:44:14.995018 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:14.994984    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6"]
Oct 30 13:44:15.018586 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:15.018558    4364 generic.go:356] "Generic (PLEG): container finished" podID="9a967f02-7c93-484e-9bd4-97e1a4f5fdf5" containerID="6bd64ed3e7f7b1c54ada326b0f8b22ca74165c53790a0937ddafdccd6d34428e" exitCode=0
Oct 30 13:44:15.018781 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:15.018641    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9"
Oct 30 13:44:15.018781 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:15.018647    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9" event={"ID":"9a967f02-7c93-484e-9bd4-97e1a4f5fdf5","Type":"ContainerDied","Data":"6bd64ed3e7f7b1c54ada326b0f8b22ca74165c53790a0937ddafdccd6d34428e"}
Oct 30 13:44:15.018781 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:15.018705    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9" event={"ID":"9a967f02-7c93-484e-9bd4-97e1a4f5fdf5","Type":"ContainerDied","Data":"862e2b625017ff886ca44397a855b99bae1aa9e7d724de267e88049510e1082a"}
Oct 30 13:44:15.018781 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:15.018735    4364 scope.go:122] "RemoveContainer" containerID="6bd64ed3e7f7b1c54ada326b0f8b22ca74165c53790a0937ddafdccd6d34428e"
Oct 30 13:44:15.029198 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:15.029178    4364 scope.go:122] "RemoveContainer" containerID="95dc1e29203e28a63d9a2c873fa4bd9873cccc99f836275aa57b1dedabca4994"
Oct 30 13:44:15.039038 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:15.039018    4364 scope.go:122] "RemoveContainer" containerID="6bd64ed3e7f7b1c54ada326b0f8b22ca74165c53790a0937ddafdccd6d34428e"
Oct 30 13:44:15.039317 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:44:15.039297    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bd64ed3e7f7b1c54ada326b0f8b22ca74165c53790a0937ddafdccd6d34428e\": container with ID starting with 6bd64ed3e7f7b1c54ada326b0f8b22ca74165c53790a0937ddafdccd6d34428e not found: ID does not exist" containerID="6bd64ed3e7f7b1c54ada326b0f8b22ca74165c53790a0937ddafdccd6d34428e"
Oct 30 13:44:15.039368 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:15.039324    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bd64ed3e7f7b1c54ada326b0f8b22ca74165c53790a0937ddafdccd6d34428e"} err="failed to get container status \"6bd64ed3e7f7b1c54ada326b0f8b22ca74165c53790a0937ddafdccd6d34428e\": rpc error: code = NotFound desc = could not find container \"6bd64ed3e7f7b1c54ada326b0f8b22ca74165c53790a0937ddafdccd6d34428e\": container with ID starting with 6bd64ed3e7f7b1c54ada326b0f8b22ca74165c53790a0937ddafdccd6d34428e not found: ID does not exist"
Oct 30 13:44:15.039368 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:15.039342    4364 scope.go:122] "RemoveContainer" containerID="95dc1e29203e28a63d9a2c873fa4bd9873cccc99f836275aa57b1dedabca4994"
Oct 30 13:44:15.039579 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:44:15.039561    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95dc1e29203e28a63d9a2c873fa4bd9873cccc99f836275aa57b1dedabca4994\": container with ID starting with 95dc1e29203e28a63d9a2c873fa4bd9873cccc99f836275aa57b1dedabca4994 not found: ID does not exist" containerID="95dc1e29203e28a63d9a2c873fa4bd9873cccc99f836275aa57b1dedabca4994"
Oct 30 13:44:15.039637 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:15.039586    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95dc1e29203e28a63d9a2c873fa4bd9873cccc99f836275aa57b1dedabca4994"} err="failed to get container status \"95dc1e29203e28a63d9a2c873fa4bd9873cccc99f836275aa57b1dedabca4994\": rpc error: code = NotFound desc = could not find container \"95dc1e29203e28a63d9a2c873fa4bd9873cccc99f836275aa57b1dedabca4994\": container with ID starting with 95dc1e29203e28a63d9a2c873fa4bd9873cccc99f836275aa57b1dedabca4994 not found: ID does not exist"
Oct 30 13:44:15.078709 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:15.078639    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9"]
Oct 30 13:44:15.099240 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:15.099212    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-qklg9"]
Oct 30 13:44:16.970168 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:16.970136    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a967f02-7c93-484e-9bd4-97e1a4f5fdf5" path="/var/lib/kubelet/pods/9a967f02-7c93-484e-9bd4-97e1a4f5fdf5/volumes"
Oct 30 13:44:17.430681 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:17.430636    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ff4285-59fc768575-tgw8j" podUID="e5c6ed73-3541-4945-8d38-cb289a4e05f8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:44:17.430681 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:17.430668    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ff4285-6b8554d544-6flwq" podUID="43e6fe99-7d56-417f-b9db-1d41a24340d2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:44:18.084692 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:18.084654    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ff4285-76c67bfb9c-ckgmb"]
Oct 30 13:44:18.085192 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:18.084934    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ff4285-76c67bfb9c-ckgmb" podUID="71576777-83eb-4fb6-af0e-33c177bc2dd0" containerName="maas-api" containerID="cri-o://30e4c2140211aab0993d7b43c934efbb8fce752dc0dc96c3c461a44620053334" gracePeriod=30
Oct 30 13:44:18.389340 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:18.389305    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg"]
Oct 30 13:44:18.389570 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:18.389546    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg" podUID="c71d1db6-aeed-434f-9829-a6589831daac" containerName="istio-proxy" containerID="cri-o://09871ad2dad15e8ddfae62ad1934266e457fdabcf163535980f6fd21955c6d0c" gracePeriod=30
Oct 30 13:44:18.390754 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:18.390680    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ff4285-76c67bfb9c-ckgmb"
Oct 30 13:44:18.536048 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:18.535999    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71576777-83eb-4fb6-af0e-33c177bc2dd0-kube-api-access-2frwx\" (UniqueName: \"kubernetes.io/projected/71576777-83eb-4fb6-af0e-33c177bc2dd0-kube-api-access-2frwx\") pod \"71576777-83eb-4fb6-af0e-33c177bc2dd0\" (UID: \"71576777-83eb-4fb6-af0e-33c177bc2dd0\") "
Oct 30 13:44:18.536760 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:18.536732    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71576777-83eb-4fb6-af0e-33c177bc2dd0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71576777-83eb-4fb6-af0e-33c177bc2dd0-maas-api-tls\") pod \"71576777-83eb-4fb6-af0e-33c177bc2dd0\" (UID: \"71576777-83eb-4fb6-af0e-33c177bc2dd0\") "
Oct 30 13:44:18.538782 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:18.538758    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71576777-83eb-4fb6-af0e-33c177bc2dd0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/71576777-83eb-4fb6-af0e-33c177bc2dd0-metrics-certs\") pod \"71576777-83eb-4fb6-af0e-33c177bc2dd0\" (UID: \"71576777-83eb-4fb6-af0e-33c177bc2dd0\") "
Oct 30 13:44:18.541429 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:18.539520    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71576777-83eb-4fb6-af0e-33c177bc2dd0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71576777-83eb-4fb6-af0e-33c177bc2dd0-openshift-service-ca\") pod \"71576777-83eb-4fb6-af0e-33c177bc2dd0\" (UID: \"71576777-83eb-4fb6-af0e-33c177bc2dd0\") "
Oct 30 13:44:18.542716 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:18.541975    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71576777-83eb-4fb6-af0e-33c177bc2dd0-openshift-service-ca" pod "71576777-83eb-4fb6-af0e-33c177bc2dd0" (UID: "71576777-83eb-4fb6-af0e-33c177bc2dd0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:44:18.546943 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:18.546904    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71576777-83eb-4fb6-af0e-33c177bc2dd0-maas-api-tls" pod "71576777-83eb-4fb6-af0e-33c177bc2dd0" (UID: "71576777-83eb-4fb6-af0e-33c177bc2dd0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:44:18.564314 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:18.564274    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71576777-83eb-4fb6-af0e-33c177bc2dd0-kube-api-access-2frwx" pod "71576777-83eb-4fb6-af0e-33c177bc2dd0" (UID: "71576777-83eb-4fb6-af0e-33c177bc2dd0"). InnerVolumeSpecName "kube-api-access-2frwx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:44:18.564755 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:18.564727    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71576777-83eb-4fb6-af0e-33c177bc2dd0-metrics-certs" pod "71576777-83eb-4fb6-af0e-33c177bc2dd0" (UID: "71576777-83eb-4fb6-af0e-33c177bc2dd0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:44:18.648637 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:18.644946    4364 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71576777-83eb-4fb6-af0e-33c177bc2dd0-openshift-service-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:18.656730 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:18.656692    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2frwx\" (UniqueName: \"kubernetes.io/projected/71576777-83eb-4fb6-af0e-33c177bc2dd0-kube-api-access-2frwx\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:18.661717 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:18.661683    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71576777-83eb-4fb6-af0e-33c177bc2dd0-maas-api-tls\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:18.666457 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:18.666417    4364 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71576777-83eb-4fb6-af0e-33c177bc2dd0-metrics-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:19.082538 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:19.082504    4364 generic.go:356] "Generic (PLEG): container finished" podID="71576777-83eb-4fb6-af0e-33c177bc2dd0" containerID="30e4c2140211aab0993d7b43c934efbb8fce752dc0dc96c3c461a44620053334" exitCode=0
Oct 30 13:44:19.082748 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:19.082585    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ff4285-76c67bfb9c-ckgmb"
Oct 30 13:44:19.082748 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:19.082622    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ff4285-76c67bfb9c-ckgmb" event={"ID":"71576777-83eb-4fb6-af0e-33c177bc2dd0","Type":"ContainerDied","Data":"30e4c2140211aab0993d7b43c934efbb8fce752dc0dc96c3c461a44620053334"}
Oct 30 13:44:19.082748 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:19.082708    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ff4285-76c67bfb9c-ckgmb" event={"ID":"71576777-83eb-4fb6-af0e-33c177bc2dd0","Type":"ContainerDied","Data":"531e2707b7fbf1da5b976259b6e501bde6c0b3a79a6b289caa6b68414c51aefe"}
Oct 30 13:44:19.082748 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:19.082735    4364 scope.go:122] "RemoveContainer" containerID="30e4c2140211aab0993d7b43c934efbb8fce752dc0dc96c3c461a44620053334"
Oct 30 13:44:19.093951 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:19.093748    4364 scope.go:122] "RemoveContainer" containerID="30e4c2140211aab0993d7b43c934efbb8fce752dc0dc96c3c461a44620053334"
Oct 30 13:44:19.094216 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:44:19.094040    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30e4c2140211aab0993d7b43c934efbb8fce752dc0dc96c3c461a44620053334\": container with ID starting with 30e4c2140211aab0993d7b43c934efbb8fce752dc0dc96c3c461a44620053334 not found: ID does not exist" containerID="30e4c2140211aab0993d7b43c934efbb8fce752dc0dc96c3c461a44620053334"
Oct 30 13:44:19.094216 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:19.094065    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30e4c2140211aab0993d7b43c934efbb8fce752dc0dc96c3c461a44620053334"} err="failed to get container status \"30e4c2140211aab0993d7b43c934efbb8fce752dc0dc96c3c461a44620053334\": rpc error: code = NotFound desc = could not find container \"30e4c2140211aab0993d7b43c934efbb8fce752dc0dc96c3c461a44620053334\": container with ID starting with 30e4c2140211aab0993d7b43c934efbb8fce752dc0dc96c3c461a44620053334 not found: ID does not exist"
Oct 30 13:44:19.148823 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:19.148785    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ff4285-76c67bfb9c-ckgmb"]
Oct 30 13:44:19.152949 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:19.152924    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ff4285-76c67bfb9c-ckgmb"]
Oct 30 13:44:20.972517 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:20.972482    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71576777-83eb-4fb6-af0e-33c177bc2dd0" path="/var/lib/kubelet/pods/71576777-83eb-4fb6-af0e-33c177bc2dd0/volumes"
Oct 30 13:44:22.698971 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:22.698944    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ff4285-6b8554d544-6flwq"
Oct 30 13:44:22.721173 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:22.721150    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ff4285-59fc768575-tgw8j"
Oct 30 13:44:22.771217 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:22.771189    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5c6ed73-3541-4945-8d38-cb289a4e05f8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5c6ed73-3541-4945-8d38-cb289a4e05f8-plugins-config-volume\") pod \"e5c6ed73-3541-4945-8d38-cb289a4e05f8\" (UID: \"e5c6ed73-3541-4945-8d38-cb289a4e05f8\") "
Oct 30 13:44:22.771384 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:22.771235    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5c6ed73-3541-4945-8d38-cb289a4e05f8-kube-api-access-tbwl8\" (UniqueName: \"kubernetes.io/projected/e5c6ed73-3541-4945-8d38-cb289a4e05f8-kube-api-access-tbwl8\") pod \"e5c6ed73-3541-4945-8d38-cb289a4e05f8\" (UID: \"e5c6ed73-3541-4945-8d38-cb289a4e05f8\") "
Oct 30 13:44:22.771384 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:22.771276    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43e6fe99-7d56-417f-b9db-1d41a24340d2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43e6fe99-7d56-417f-b9db-1d41a24340d2-plugins-config-volume\") pod \"43e6fe99-7d56-417f-b9db-1d41a24340d2\" (UID: \"43e6fe99-7d56-417f-b9db-1d41a24340d2\") "
Oct 30 13:44:22.771384 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:22.771312    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43e6fe99-7d56-417f-b9db-1d41a24340d2-kube-api-access-w7svt\" (UniqueName: \"kubernetes.io/projected/43e6fe99-7d56-417f-b9db-1d41a24340d2-kube-api-access-w7svt\") pod \"43e6fe99-7d56-417f-b9db-1d41a24340d2\" (UID: \"43e6fe99-7d56-417f-b9db-1d41a24340d2\") "
Oct 30 13:44:22.771905 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:22.771876    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5c6ed73-3541-4945-8d38-cb289a4e05f8-plugins-config-volume" pod "e5c6ed73-3541-4945-8d38-cb289a4e05f8" (UID: "e5c6ed73-3541-4945-8d38-cb289a4e05f8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:44:22.772560 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:22.772532    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43e6fe99-7d56-417f-b9db-1d41a24340d2-plugins-config-volume" pod "43e6fe99-7d56-417f-b9db-1d41a24340d2" (UID: "43e6fe99-7d56-417f-b9db-1d41a24340d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:44:22.773804 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:22.773777    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5c6ed73-3541-4945-8d38-cb289a4e05f8-kube-api-access-tbwl8" pod "e5c6ed73-3541-4945-8d38-cb289a4e05f8" (UID: "e5c6ed73-3541-4945-8d38-cb289a4e05f8"). InnerVolumeSpecName "kube-api-access-tbwl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:44:22.773894 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:22.773871    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43e6fe99-7d56-417f-b9db-1d41a24340d2-kube-api-access-w7svt" pod "43e6fe99-7d56-417f-b9db-1d41a24340d2" (UID: "43e6fe99-7d56-417f-b9db-1d41a24340d2"). InnerVolumeSpecName "kube-api-access-w7svt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:44:22.873061 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:22.873025    4364 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5c6ed73-3541-4945-8d38-cb289a4e05f8-plugins-config-volume\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:22.873061 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:22.873059    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbwl8\" (UniqueName: \"kubernetes.io/projected/e5c6ed73-3541-4945-8d38-cb289a4e05f8-kube-api-access-tbwl8\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:22.873259 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:22.873072    4364 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43e6fe99-7d56-417f-b9db-1d41a24340d2-plugins-config-volume\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:22.873259 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:22.873088    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7svt\" (UniqueName: \"kubernetes.io/projected/43e6fe99-7d56-417f-b9db-1d41a24340d2-kube-api-access-w7svt\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:23.112285 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:23.112252    4364 generic.go:356] "Generic (PLEG): container finished" podID="43e6fe99-7d56-417f-b9db-1d41a24340d2" containerID="e7763a4801ab768e54a01a08625bda1aed3ea3e1c6c4323eee61c82fa52e72cd" exitCode=0
Oct 30 13:44:23.112442 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:23.112295    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ff4285-6b8554d544-6flwq"
Oct 30 13:44:23.112442 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:23.112337    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ff4285-6b8554d544-6flwq" event={"ID":"43e6fe99-7d56-417f-b9db-1d41a24340d2","Type":"ContainerDied","Data":"e7763a4801ab768e54a01a08625bda1aed3ea3e1c6c4323eee61c82fa52e72cd"}
Oct 30 13:44:23.112442 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:23.112389    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ff4285-6b8554d544-6flwq" event={"ID":"43e6fe99-7d56-417f-b9db-1d41a24340d2","Type":"ContainerDied","Data":"3c78831d67fc34bf0467a6bb68667f7b3d9cb2e077080860dbd960bc76a1c505"}
Oct 30 13:44:23.112442 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:23.112414    4364 scope.go:122] "RemoveContainer" containerID="e7763a4801ab768e54a01a08625bda1aed3ea3e1c6c4323eee61c82fa52e72cd"
Oct 30 13:44:23.113958 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:23.113933    4364 generic.go:356] "Generic (PLEG): container finished" podID="e5c6ed73-3541-4945-8d38-cb289a4e05f8" containerID="40100748bc3f22e0b545a30cdee796a670965ce2fadcc87c0aba5ab0d65622fb" exitCode=0
Oct 30 13:44:23.114128 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:23.113971    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ff4285-59fc768575-tgw8j"
Oct 30 13:44:23.114128 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:23.114034    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ff4285-59fc768575-tgw8j" event={"ID":"e5c6ed73-3541-4945-8d38-cb289a4e05f8","Type":"ContainerDied","Data":"40100748bc3f22e0b545a30cdee796a670965ce2fadcc87c0aba5ab0d65622fb"}
Oct 30 13:44:23.114128 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:23.114078    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ff4285-59fc768575-tgw8j" event={"ID":"e5c6ed73-3541-4945-8d38-cb289a4e05f8","Type":"ContainerDied","Data":"f765a5f6a49fb0e97d299eeb4f1dd9bbc6329deb46183818370f35270249196d"}
Oct 30 13:44:23.122680 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:23.122664    4364 scope.go:122] "RemoveContainer" containerID="e7763a4801ab768e54a01a08625bda1aed3ea3e1c6c4323eee61c82fa52e72cd"
Oct 30 13:44:23.122962 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:44:23.122939    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7763a4801ab768e54a01a08625bda1aed3ea3e1c6c4323eee61c82fa52e72cd\": container with ID starting with e7763a4801ab768e54a01a08625bda1aed3ea3e1c6c4323eee61c82fa52e72cd not found: ID does not exist" containerID="e7763a4801ab768e54a01a08625bda1aed3ea3e1c6c4323eee61c82fa52e72cd"
Oct 30 13:44:23.123063 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:23.122971    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7763a4801ab768e54a01a08625bda1aed3ea3e1c6c4323eee61c82fa52e72cd"} err="failed to get container status \"e7763a4801ab768e54a01a08625bda1aed3ea3e1c6c4323eee61c82fa52e72cd\": rpc error: code = NotFound desc = could not find container \"e7763a4801ab768e54a01a08625bda1aed3ea3e1c6c4323eee61c82fa52e72cd\": container with ID starting with e7763a4801ab768e54a01a08625bda1aed3ea3e1c6c4323eee61c82fa52e72cd not found: ID does not exist"
Oct 30 13:44:23.123063 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:23.122993    4364 scope.go:122] "RemoveContainer" containerID="40100748bc3f22e0b545a30cdee796a670965ce2fadcc87c0aba5ab0d65622fb"
Oct 30 13:44:23.132126 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:23.132106    4364 scope.go:122] "RemoveContainer" containerID="40100748bc3f22e0b545a30cdee796a670965ce2fadcc87c0aba5ab0d65622fb"
Oct 30 13:44:23.132385 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:44:23.132367    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40100748bc3f22e0b545a30cdee796a670965ce2fadcc87c0aba5ab0d65622fb\": container with ID starting with 40100748bc3f22e0b545a30cdee796a670965ce2fadcc87c0aba5ab0d65622fb not found: ID does not exist" containerID="40100748bc3f22e0b545a30cdee796a670965ce2fadcc87c0aba5ab0d65622fb"
Oct 30 13:44:23.132433 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:23.132391    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40100748bc3f22e0b545a30cdee796a670965ce2fadcc87c0aba5ab0d65622fb"} err="failed to get container status \"40100748bc3f22e0b545a30cdee796a670965ce2fadcc87c0aba5ab0d65622fb\": rpc error: code = NotFound desc = could not find container \"40100748bc3f22e0b545a30cdee796a670965ce2fadcc87c0aba5ab0d65622fb\": container with ID starting with 40100748bc3f22e0b545a30cdee796a670965ce2fadcc87c0aba5ab0d65622fb not found: ID does not exist"
Oct 30 13:44:23.176282 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:23.176253    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-ff4285-6b8554d544-6flwq"]
Oct 30 13:44:23.189090 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:23.189064    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-ff4285-6b8554d544-6flwq"]
Oct 30 13:44:23.228314 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:23.228283    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ff4285-59fc768575-tgw8j"]
Oct 30 13:44:23.236727 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:23.236660    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ff4285-59fc768575-tgw8j"]
Oct 30 13:44:23.692864 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:23.692841    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg"
Oct 30 13:44:23.784269 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:23.784189    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c71d1db6-aeed-434f-9829-a6589831daac-istio-token\" (UniqueName: \"kubernetes.io/projected/c71d1db6-aeed-434f-9829-a6589831daac-istio-token\") pod \"c71d1db6-aeed-434f-9829-a6589831daac\" (UID: \"c71d1db6-aeed-434f-9829-a6589831daac\") "
Oct 30 13:44:23.784269 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:23.784245    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c71d1db6-aeed-434f-9829-a6589831daac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c71d1db6-aeed-434f-9829-a6589831daac-istio-podinfo\") pod \"c71d1db6-aeed-434f-9829-a6589831daac\" (UID: \"c71d1db6-aeed-434f-9829-a6589831daac\") "
Oct 30 13:44:23.784847 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:23.784310    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c71d1db6-aeed-434f-9829-a6589831daac-kube-api-access-svr28\" (UniqueName: \"kubernetes.io/projected/c71d1db6-aeed-434f-9829-a6589831daac-kube-api-access-svr28\") pod \"c71d1db6-aeed-434f-9829-a6589831daac\" (UID: \"c71d1db6-aeed-434f-9829-a6589831daac\") "
Oct 30 13:44:23.784847 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:23.784352    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c71d1db6-aeed-434f-9829-a6589831daac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c71d1db6-aeed-434f-9829-a6589831daac-istio-data\") pod \"c71d1db6-aeed-434f-9829-a6589831daac\" (UID: \"c71d1db6-aeed-434f-9829-a6589831daac\") "
Oct 30 13:44:23.784847 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:23.784385    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c71d1db6-aeed-434f-9829-a6589831daac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c71d1db6-aeed-434f-9829-a6589831daac-istiod-ca-cert\") pod \"c71d1db6-aeed-434f-9829-a6589831daac\" (UID: \"c71d1db6-aeed-434f-9829-a6589831daac\") "
Oct 30 13:44:23.784847 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:23.784484    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c71d1db6-aeed-434f-9829-a6589831daac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c71d1db6-aeed-434f-9829-a6589831daac-workload-certs\") pod \"c71d1db6-aeed-434f-9829-a6589831daac\" (UID: \"c71d1db6-aeed-434f-9829-a6589831daac\") "
Oct 30 13:44:23.784847 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:23.784515    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c71d1db6-aeed-434f-9829-a6589831daac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c71d1db6-aeed-434f-9829-a6589831daac-workload-socket\") pod \"c71d1db6-aeed-434f-9829-a6589831daac\" (UID: \"c71d1db6-aeed-434f-9829-a6589831daac\") "
Oct 30 13:44:23.784847 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:23.784566    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c71d1db6-aeed-434f-9829-a6589831daac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c71d1db6-aeed-434f-9829-a6589831daac-istio-envoy\") pod \"c71d1db6-aeed-434f-9829-a6589831daac\" (UID: \"c71d1db6-aeed-434f-9829-a6589831daac\") "
Oct 30 13:44:23.784847 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:23.784671    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c71d1db6-aeed-434f-9829-a6589831daac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c71d1db6-aeed-434f-9829-a6589831daac-credential-socket\") pod \"c71d1db6-aeed-434f-9829-a6589831daac\" (UID: \"c71d1db6-aeed-434f-9829-a6589831daac\") "
Oct 30 13:44:23.785259 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:23.784923    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c71d1db6-aeed-434f-9829-a6589831daac-workload-socket" pod "c71d1db6-aeed-434f-9829-a6589831daac" (UID: "c71d1db6-aeed-434f-9829-a6589831daac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:23.785259 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:23.785104    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c71d1db6-aeed-434f-9829-a6589831daac-workload-certs" pod "c71d1db6-aeed-434f-9829-a6589831daac" (UID: "c71d1db6-aeed-434f-9829-a6589831daac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:23.785259 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:23.785123    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c71d1db6-aeed-434f-9829-a6589831daac-istiod-ca-cert" pod "c71d1db6-aeed-434f-9829-a6589831daac" (UID: "c71d1db6-aeed-434f-9829-a6589831daac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:44:23.785259 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:23.785147    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c71d1db6-aeed-434f-9829-a6589831daac-istio-data" pod "c71d1db6-aeed-434f-9829-a6589831daac" (UID: "c71d1db6-aeed-434f-9829-a6589831daac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:23.785485 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:23.785417    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c71d1db6-aeed-434f-9829-a6589831daac-credential-socket" pod "c71d1db6-aeed-434f-9829-a6589831daac" (UID: "c71d1db6-aeed-434f-9829-a6589831daac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:23.785706 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:23.785683    4364 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c71d1db6-aeed-434f-9829-a6589831daac-credential-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:23.785813 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:23.785710    4364 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c71d1db6-aeed-434f-9829-a6589831daac-istio-data\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:23.785813 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:23.785727    4364 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c71d1db6-aeed-434f-9829-a6589831daac-istiod-ca-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:23.785813 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:23.785740    4364 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c71d1db6-aeed-434f-9829-a6589831daac-workload-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:23.785813 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:23.785753    4364 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c71d1db6-aeed-434f-9829-a6589831daac-workload-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:23.787863 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:23.787838    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c71d1db6-aeed-434f-9829-a6589831daac-istio-token" pod "c71d1db6-aeed-434f-9829-a6589831daac" (UID: "c71d1db6-aeed-434f-9829-a6589831daac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:44:23.788006 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:23.787980    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c71d1db6-aeed-434f-9829-a6589831daac-istio-podinfo" pod "c71d1db6-aeed-434f-9829-a6589831daac" (UID: "c71d1db6-aeed-434f-9829-a6589831daac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:44:23.788093 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:23.788063    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c71d1db6-aeed-434f-9829-a6589831daac-istio-envoy" pod "c71d1db6-aeed-434f-9829-a6589831daac" (UID: "c71d1db6-aeed-434f-9829-a6589831daac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:23.789986 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:23.789952    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c71d1db6-aeed-434f-9829-a6589831daac-kube-api-access-svr28" pod "c71d1db6-aeed-434f-9829-a6589831daac" (UID: "c71d1db6-aeed-434f-9829-a6589831daac"). InnerVolumeSpecName "kube-api-access-svr28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:44:23.887215 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:23.887185    4364 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c71d1db6-aeed-434f-9829-a6589831daac-istio-envoy\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:23.887215 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:23.887213    4364 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c71d1db6-aeed-434f-9829-a6589831daac-istio-token\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:23.887215 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:23.887222    4364 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c71d1db6-aeed-434f-9829-a6589831daac-istio-podinfo\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:23.887432 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:23.887231    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svr28\" (UniqueName: \"kubernetes.io/projected/c71d1db6-aeed-434f-9829-a6589831daac-kube-api-access-svr28\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:24.122460 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:24.122429    4364 generic.go:356] "Generic (PLEG): container finished" podID="c71d1db6-aeed-434f-9829-a6589831daac" containerID="09871ad2dad15e8ddfae62ad1934266e457fdabcf163535980f6fd21955c6d0c" exitCode=0
Oct 30 13:44:24.122643 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:24.122479    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg"
Oct 30 13:44:24.122643 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:24.122502    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg" event={"ID":"c71d1db6-aeed-434f-9829-a6589831daac","Type":"ContainerDied","Data":"09871ad2dad15e8ddfae62ad1934266e457fdabcf163535980f6fd21955c6d0c"}
Oct 30 13:44:24.122643 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:24.122553    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg" event={"ID":"c71d1db6-aeed-434f-9829-a6589831daac","Type":"ContainerDied","Data":"30d94a83baefa04b256170c0adbe2608882d7ce5e6fade46735b4fc6bc5040d1"}
Oct 30 13:44:24.122643 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:24.122579    4364 scope.go:122] "RemoveContainer" containerID="09871ad2dad15e8ddfae62ad1934266e457fdabcf163535980f6fd21955c6d0c"
Oct 30 13:44:24.133692 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:24.133669    4364 scope.go:122] "RemoveContainer" containerID="09871ad2dad15e8ddfae62ad1934266e457fdabcf163535980f6fd21955c6d0c"
Oct 30 13:44:24.133971 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:44:24.133950    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09871ad2dad15e8ddfae62ad1934266e457fdabcf163535980f6fd21955c6d0c\": container with ID starting with 09871ad2dad15e8ddfae62ad1934266e457fdabcf163535980f6fd21955c6d0c not found: ID does not exist" containerID="09871ad2dad15e8ddfae62ad1934266e457fdabcf163535980f6fd21955c6d0c"
Oct 30 13:44:24.134026 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:24.133979    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09871ad2dad15e8ddfae62ad1934266e457fdabcf163535980f6fd21955c6d0c"} err="failed to get container status \"09871ad2dad15e8ddfae62ad1934266e457fdabcf163535980f6fd21955c6d0c\": rpc error: code = NotFound desc = could not find container \"09871ad2dad15e8ddfae62ad1934266e457fdabcf163535980f6fd21955c6d0c\": container with ID starting with 09871ad2dad15e8ddfae62ad1934266e457fdabcf163535980f6fd21955c6d0c not found: ID does not exist"
Oct 30 13:44:24.187366 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:24.187336    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg"]
Oct 30 13:44:24.201678 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:24.201649    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-49pcg"]
Oct 30 13:44:24.970737 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:24.970699    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43e6fe99-7d56-417f-b9db-1d41a24340d2" path="/var/lib/kubelet/pods/43e6fe99-7d56-417f-b9db-1d41a24340d2/volumes"
Oct 30 13:44:24.971496 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:24.971478    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c71d1db6-aeed-434f-9829-a6589831daac" path="/var/lib/kubelet/pods/c71d1db6-aeed-434f-9829-a6589831daac/volumes"
Oct 30 13:44:24.972124 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:24.972102    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5c6ed73-3541-4945-8d38-cb289a4e05f8" path="/var/lib/kubelet/pods/e5c6ed73-3541-4945-8d38-cb289a4e05f8/volumes"
Oct 30 13:44:25.061308 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:25.061272    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l"]
Oct 30 13:44:25.061581 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:25.061556    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l" podUID="297ed651-f2f1-40f4-bbe9-bf93abf2ac8d" containerName="istio-proxy" containerID="cri-o://1070c42cc863836cf2980d60febfcb8d13f8bb15303a6085e9eb1a2bb304041d" gracePeriod=30
Oct 30 13:44:27.415219 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:27.415173    4364 patch_prober.go:29] interesting pod/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:44:27.415663 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:27.415240    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l" podUID="297ed651-f2f1-40f4-bbe9-bf93abf2ac8d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:44:27.841688 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:27.841577    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2"]
Oct 30 13:44:27.841892 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:27.841848    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2" podUID="fd17aa36-e0ae-4652-a85f-20945233ccc0" containerName="istio-proxy" containerID="cri-o://dde43c13d515a38bb07a29af73448f8a91f8d16989b8908d1f16060414f5b88d" gracePeriod=30
Oct 30 13:44:27.850564 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:27.850536    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv"]
Oct 30 13:44:27.850754 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:27.850733    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv" podUID="7090457f-3bbf-4295-aa35-9dedc02d1b61" containerName="istio-proxy" containerID="cri-o://d37963fcb3d6d226584f31b5f028e9f4bd77fc3f6b346e1f40eafd201535a154" gracePeriod=30
Oct 30 13:44:29.042585 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:29.042545    4364 patch_prober.go:29] interesting pod/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:44:29.042981 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:29.042631    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv" podUID="7090457f-3bbf-4295-aa35-9dedc02d1b61" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:44:30.027622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:30.027261    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-d833bd/distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74" podUID="4f2b06ac-9bf8-4b57-bd22-85532576f9c1" containerName="main" containerID="cri-o://87b706902f5786f2505a8bb78db3558d561529d5e0d8ce0476a0bee04e885419" gracePeriod=45
Oct 30 13:44:30.036210 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:30.036173    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6" podUID="b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1" containerName="main" containerID="cri-o://b5f58c075b9206df95f27d39002a830a5366277a39068e8bdac039676de7a398" gracePeriod=45
Oct 30 13:44:30.169080 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:30.169044    4364 generic.go:356] "Generic (PLEG): container finished" podID="b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1" containerID="b5f58c075b9206df95f27d39002a830a5366277a39068e8bdac039676de7a398" exitCode=0
Oct 30 13:44:30.169478 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:30.169228    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6" event={"ID":"b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1","Type":"ContainerDied","Data":"b5f58c075b9206df95f27d39002a830a5366277a39068e8bdac039676de7a398"}
Oct 30 13:44:30.171568 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:30.171538    4364 generic.go:356] "Generic (PLEG): container finished" podID="4f2b06ac-9bf8-4b57-bd22-85532576f9c1" containerID="87b706902f5786f2505a8bb78db3558d561529d5e0d8ce0476a0bee04e885419" exitCode=0
Oct 30 13:44:30.171729 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:30.171670    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d833bd/distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74" event={"ID":"4f2b06ac-9bf8-4b57-bd22-85532576f9c1","Type":"ContainerDied","Data":"87b706902f5786f2505a8bb78db3558d561529d5e0d8ce0476a0bee04e885419"}
Oct 30 13:44:30.352549 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:30.352417    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6"
Oct 30 13:44:30.362690 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:30.362667    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d833bd/distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74"
Oct 30 13:44:30.369631 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:30.369568    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l"
Oct 30 13:44:30.514468 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:30.513310    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1-model-cache\") pod \"b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1\" (UID: \"b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1\") "
Oct 30 13:44:30.514468 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:30.513388    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/297ed651-f2f1-40f4-bbe9-bf93abf2ac8d-istio-token\" (UniqueName: \"kubernetes.io/projected/297ed651-f2f1-40f4-bbe9-bf93abf2ac8d-istio-token\") pod \"297ed651-f2f1-40f4-bbe9-bf93abf2ac8d\" (UID: \"297ed651-f2f1-40f4-bbe9-bf93abf2ac8d\") "
Oct 30 13:44:30.514468 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:30.513422    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f2b06ac-9bf8-4b57-bd22-85532576f9c1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f2b06ac-9bf8-4b57-bd22-85532576f9c1-tmp-dir\") pod \"4f2b06ac-9bf8-4b57-bd22-85532576f9c1\" (UID: \"4f2b06ac-9bf8-4b57-bd22-85532576f9c1\") "
Oct 30 13:44:30.514468 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:30.513461    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f2b06ac-9bf8-4b57-bd22-85532576f9c1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f2b06ac-9bf8-4b57-bd22-85532576f9c1-dshm\") pod \"4f2b06ac-9bf8-4b57-bd22-85532576f9c1\" (UID: \"4f2b06ac-9bf8-4b57-bd22-85532576f9c1\") "
Oct 30 13:44:30.514468 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:30.513498    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1-dshm\") pod \"b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1\" (UID: \"b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1\") "
Oct 30 13:44:30.514468 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:30.513533    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f2b06ac-9bf8-4b57-bd22-85532576f9c1-home\" (UniqueName: \"kubernetes.io/empty-dir/4f2b06ac-9bf8-4b57-bd22-85532576f9c1-home\") pod \"4f2b06ac-9bf8-4b57-bd22-85532576f9c1\" (UID: \"4f2b06ac-9bf8-4b57-bd22-85532576f9c1\") "
Oct 30 13:44:30.514468 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:30.513640    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/297ed651-f2f1-40f4-bbe9-bf93abf2ac8d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/297ed651-f2f1-40f4-bbe9-bf93abf2ac8d-workload-certs\") pod \"297ed651-f2f1-40f4-bbe9-bf93abf2ac8d\" (UID: \"297ed651-f2f1-40f4-bbe9-bf93abf2ac8d\") "
Oct 30 13:44:30.514468 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:30.513681    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f2b06ac-9bf8-4b57-bd22-85532576f9c1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f2b06ac-9bf8-4b57-bd22-85532576f9c1-model-cache\") pod \"4f2b06ac-9bf8-4b57-bd22-85532576f9c1\" (UID: \"4f2b06ac-9bf8-4b57-bd22-85532576f9c1\") "
Oct 30 13:44:30.514468 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:30.513711    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/297ed651-f2f1-40f4-bbe9-bf93abf2ac8d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/297ed651-f2f1-40f4-bbe9-bf93abf2ac8d-istio-data\") pod \"297ed651-f2f1-40f4-bbe9-bf93abf2ac8d\" (UID: \"297ed651-f2f1-40f4-bbe9-bf93abf2ac8d\") "
Oct 30 13:44:30.514468 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:30.513747    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/297ed651-f2f1-40f4-bbe9-bf93abf2ac8d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/297ed651-f2f1-40f4-bbe9-bf93abf2ac8d-istio-envoy\") pod \"297ed651-f2f1-40f4-bbe9-bf93abf2ac8d\" (UID: \"297ed651-f2f1-40f4-bbe9-bf93abf2ac8d\") "
Oct 30 13:44:30.514468 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:30.513777    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/297ed651-f2f1-40f4-bbe9-bf93abf2ac8d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/297ed651-f2f1-40f4-bbe9-bf93abf2ac8d-workload-socket\") pod \"297ed651-f2f1-40f4-bbe9-bf93abf2ac8d\" (UID: \"297ed651-f2f1-40f4-bbe9-bf93abf2ac8d\") "
Oct 30 13:44:30.514468 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:30.513841    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1-tls-certs\" (UniqueName: \"kubernetes.io/secret/b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1-tls-certs\") pod \"b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1\" (UID: \"b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1\") "
Oct 30 13:44:30.514468 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:30.513880    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1-home\" (UniqueName: \"kubernetes.io/empty-dir/b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1-home\") pod \"b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1\" (UID: \"b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1\") "
Oct 30 13:44:30.514468 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:30.513956    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/297ed651-f2f1-40f4-bbe9-bf93abf2ac8d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/297ed651-f2f1-40f4-bbe9-bf93abf2ac8d-credential-socket\") pod \"297ed651-f2f1-40f4-bbe9-bf93abf2ac8d\" (UID: \"297ed651-f2f1-40f4-bbe9-bf93abf2ac8d\") "
Oct 30 13:44:30.514468 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:30.514001    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f2b06ac-9bf8-4b57-bd22-85532576f9c1-tls-certs\" (UniqueName: \"kubernetes.io/secret/4f2b06ac-9bf8-4b57-bd22-85532576f9c1-tls-certs\") pod \"4f2b06ac-9bf8-4b57-bd22-85532576f9c1\" (UID: \"4f2b06ac-9bf8-4b57-bd22-85532576f9c1\") "
Oct 30 13:44:30.514468 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:30.514035    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1-kube-api-access-mwmvd\" (UniqueName: \"kubernetes.io/projected/b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1-kube-api-access-mwmvd\") pod \"b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1\" (UID: \"b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1\") "
Oct 30 13:44:30.514468 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:30.514120    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/297ed651-f2f1-40f4-bbe9-bf93abf2ac8d-kube-api-access-rvjlh\" (UniqueName: \"kubernetes.io/projected/297ed651-f2f1-40f4-bbe9-bf93abf2ac8d-kube-api-access-rvjlh\") pod \"297ed651-f2f1-40f4-bbe9-bf93abf2ac8d\" (UID: \"297ed651-f2f1-40f4-bbe9-bf93abf2ac8d\") "
Oct 30 13:44:30.514468 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:30.514191    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1-tmp-dir\") pod \"b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1\" (UID: \"b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1\") "
Oct 30 13:44:30.514468 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:30.514227    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f2b06ac-9bf8-4b57-bd22-85532576f9c1-kube-api-access-dx8wb\" (UniqueName: \"kubernetes.io/projected/4f2b06ac-9bf8-4b57-bd22-85532576f9c1-kube-api-access-dx8wb\") pod \"4f2b06ac-9bf8-4b57-bd22-85532576f9c1\" (UID: \"4f2b06ac-9bf8-4b57-bd22-85532576f9c1\") "
Oct 30 13:44:30.514468 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:30.514261    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/297ed651-f2f1-40f4-bbe9-bf93abf2ac8d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/297ed651-f2f1-40f4-bbe9-bf93abf2ac8d-istio-podinfo\") pod \"297ed651-f2f1-40f4-bbe9-bf93abf2ac8d\" (UID: \"297ed651-f2f1-40f4-bbe9-bf93abf2ac8d\") "
Oct 30 13:44:30.514468 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:30.514308    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/297ed651-f2f1-40f4-bbe9-bf93abf2ac8d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/297ed651-f2f1-40f4-bbe9-bf93abf2ac8d-istiod-ca-cert\") pod \"297ed651-f2f1-40f4-bbe9-bf93abf2ac8d\" (UID: \"297ed651-f2f1-40f4-bbe9-bf93abf2ac8d\") "
Oct 30 13:44:30.518400 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:30.518131    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/297ed651-f2f1-40f4-bbe9-bf93abf2ac8d-istiod-ca-cert" pod "297ed651-f2f1-40f4-bbe9-bf93abf2ac8d" (UID: "297ed651-f2f1-40f4-bbe9-bf93abf2ac8d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:44:30.518637 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:30.518554    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/297ed651-f2f1-40f4-bbe9-bf93abf2ac8d-workload-socket" pod "297ed651-f2f1-40f4-bbe9-bf93abf2ac8d" (UID: "297ed651-f2f1-40f4-bbe9-bf93abf2ac8d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:30.519782 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:30.519213    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/297ed651-f2f1-40f4-bbe9-bf93abf2ac8d-istio-envoy" pod "297ed651-f2f1-40f4-bbe9-bf93abf2ac8d" (UID: "297ed651-f2f1-40f4-bbe9-bf93abf2ac8d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:30.519782 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:30.519444    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1-model-cache" pod "b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1" (UID: "b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:30.520366 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:30.520100    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1-tmp-dir" pod "b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1" (UID: "b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:30.520831 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:30.520529    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1-home" pod "b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1" (UID: "b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:30.521002 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:30.520981    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/297ed651-f2f1-40f4-bbe9-bf93abf2ac8d-credential-socket" pod "297ed651-f2f1-40f4-bbe9-bf93abf2ac8d" (UID: "297ed651-f2f1-40f4-bbe9-bf93abf2ac8d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:30.521397 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:30.521376    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f2b06ac-9bf8-4b57-bd22-85532576f9c1-home" pod "4f2b06ac-9bf8-4b57-bd22-85532576f9c1" (UID: "4f2b06ac-9bf8-4b57-bd22-85532576f9c1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:30.521715 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:30.521690    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f2b06ac-9bf8-4b57-bd22-85532576f9c1-tmp-dir" pod "4f2b06ac-9bf8-4b57-bd22-85532576f9c1" (UID: "4f2b06ac-9bf8-4b57-bd22-85532576f9c1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:30.522305 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:30.522052    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/297ed651-f2f1-40f4-bbe9-bf93abf2ac8d-workload-certs" pod "297ed651-f2f1-40f4-bbe9-bf93abf2ac8d" (UID: "297ed651-f2f1-40f4-bbe9-bf93abf2ac8d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:30.522305 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:30.522066    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f2b06ac-9bf8-4b57-bd22-85532576f9c1-model-cache" pod "4f2b06ac-9bf8-4b57-bd22-85532576f9c1" (UID: "4f2b06ac-9bf8-4b57-bd22-85532576f9c1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:30.522305 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:30.522273    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/297ed651-f2f1-40f4-bbe9-bf93abf2ac8d-istio-data" pod "297ed651-f2f1-40f4-bbe9-bf93abf2ac8d" (UID: "297ed651-f2f1-40f4-bbe9-bf93abf2ac8d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:30.523351 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:30.523313    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1-tls-certs" pod "b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1" (UID: "b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:44:30.524419 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:30.524384    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f2b06ac-9bf8-4b57-bd22-85532576f9c1-tls-certs" pod "4f2b06ac-9bf8-4b57-bd22-85532576f9c1" (UID: "4f2b06ac-9bf8-4b57-bd22-85532576f9c1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:44:30.524881 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:30.524857    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1-dshm" pod "b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1" (UID: "b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:30.526043 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:30.526017    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/297ed651-f2f1-40f4-bbe9-bf93abf2ac8d-istio-token" pod "297ed651-f2f1-40f4-bbe9-bf93abf2ac8d" (UID: "297ed651-f2f1-40f4-bbe9-bf93abf2ac8d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:44:30.526263 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:30.526056    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f2b06ac-9bf8-4b57-bd22-85532576f9c1-kube-api-access-dx8wb" pod "4f2b06ac-9bf8-4b57-bd22-85532576f9c1" (UID: "4f2b06ac-9bf8-4b57-bd22-85532576f9c1"). InnerVolumeSpecName "kube-api-access-dx8wb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:44:30.526263 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:30.526168    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1-kube-api-access-mwmvd" pod "b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1" (UID: "b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1"). InnerVolumeSpecName "kube-api-access-mwmvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:44:30.526722 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:30.526686    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/297ed651-f2f1-40f4-bbe9-bf93abf2ac8d-kube-api-access-rvjlh" pod "297ed651-f2f1-40f4-bbe9-bf93abf2ac8d" (UID: "297ed651-f2f1-40f4-bbe9-bf93abf2ac8d"). InnerVolumeSpecName "kube-api-access-rvjlh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:44:30.527459 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:30.527436    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/297ed651-f2f1-40f4-bbe9-bf93abf2ac8d-istio-podinfo" pod "297ed651-f2f1-40f4-bbe9-bf93abf2ac8d" (UID: "297ed651-f2f1-40f4-bbe9-bf93abf2ac8d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:44:30.528192 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:30.528164    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f2b06ac-9bf8-4b57-bd22-85532576f9c1-dshm" pod "4f2b06ac-9bf8-4b57-bd22-85532576f9c1" (UID: "4f2b06ac-9bf8-4b57-bd22-85532576f9c1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:30.622863 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:30.622828    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwmvd\" (UniqueName: \"kubernetes.io/projected/b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1-kube-api-access-mwmvd\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:30.622863 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:30.622863    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvjlh\" (UniqueName: \"kubernetes.io/projected/297ed651-f2f1-40f4-bbe9-bf93abf2ac8d-kube-api-access-rvjlh\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:30.623125 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:30.622877    4364 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1-tmp-dir\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:30.623125 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:30.622890    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dx8wb\" (UniqueName: \"kubernetes.io/projected/4f2b06ac-9bf8-4b57-bd22-85532576f9c1-kube-api-access-dx8wb\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:30.623125 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:30.622903    4364 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/297ed651-f2f1-40f4-bbe9-bf93abf2ac8d-istio-podinfo\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:30.623125 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:30.622915    4364 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/297ed651-f2f1-40f4-bbe9-bf93abf2ac8d-istiod-ca-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:30.623125 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:30.622926    4364 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1-model-cache\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:30.623125 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:30.622937    4364 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/297ed651-f2f1-40f4-bbe9-bf93abf2ac8d-istio-token\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:30.623125 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:30.622947    4364 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f2b06ac-9bf8-4b57-bd22-85532576f9c1-tmp-dir\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:30.623125 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:30.622958    4364 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f2b06ac-9bf8-4b57-bd22-85532576f9c1-dshm\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:30.623125 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:30.622968    4364 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1-dshm\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:30.623125 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:30.622979    4364 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f2b06ac-9bf8-4b57-bd22-85532576f9c1-home\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:30.623125 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:30.622990    4364 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/297ed651-f2f1-40f4-bbe9-bf93abf2ac8d-workload-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:30.623125 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:30.623002    4364 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f2b06ac-9bf8-4b57-bd22-85532576f9c1-model-cache\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:30.623125 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:30.623016    4364 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/297ed651-f2f1-40f4-bbe9-bf93abf2ac8d-istio-data\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:30.623125 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:30.623027    4364 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/297ed651-f2f1-40f4-bbe9-bf93abf2ac8d-istio-envoy\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:30.623125 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:30.623039    4364 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/297ed651-f2f1-40f4-bbe9-bf93abf2ac8d-workload-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:30.623125 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:30.623050    4364 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1-tls-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:30.623125 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:30.623060    4364 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1-home\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:30.623125 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:30.623071    4364 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/297ed651-f2f1-40f4-bbe9-bf93abf2ac8d-credential-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:30.623125 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:30.623084    4364 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f2b06ac-9bf8-4b57-bd22-85532576f9c1-tls-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:31.180151 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:31.180116    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6"
Oct 30 13:44:31.180656 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:31.180144    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6" event={"ID":"b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1","Type":"ContainerDied","Data":"361b120a705036f644c9a4c0a479b132c2f22482dfb5459f0a55f6a82f668558"}
Oct 30 13:44:31.180656 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:31.180208    4364 scope.go:122] "RemoveContainer" containerID="b5f58c075b9206df95f27d39002a830a5366277a39068e8bdac039676de7a398"
Oct 30 13:44:31.181757 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:31.181735    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d833bd/distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74"
Oct 30 13:44:31.181897 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:31.181753    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d833bd/distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74" event={"ID":"4f2b06ac-9bf8-4b57-bd22-85532576f9c1","Type":"ContainerDied","Data":"42075a37c9bf8bd2f71fd89de09f99af8d7df03e7ca8a33cfc4e1172ee8c7596"}
Oct 30 13:44:31.183452 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:31.183429    4364 generic.go:356] "Generic (PLEG): container finished" podID="297ed651-f2f1-40f4-bbe9-bf93abf2ac8d" containerID="1070c42cc863836cf2980d60febfcb8d13f8bb15303a6085e9eb1a2bb304041d" exitCode=0
Oct 30 13:44:31.183556 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:31.183460    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l" event={"ID":"297ed651-f2f1-40f4-bbe9-bf93abf2ac8d","Type":"ContainerDied","Data":"1070c42cc863836cf2980d60febfcb8d13f8bb15303a6085e9eb1a2bb304041d"}
Oct 30 13:44:31.183556 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:31.183491    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l"
Oct 30 13:44:31.183556 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:31.183517    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l" event={"ID":"297ed651-f2f1-40f4-bbe9-bf93abf2ac8d","Type":"ContainerDied","Data":"7d3b61a8829a6b646eb402fb586112904776d680d1cafc8ce1ec4f03bca65770"}
Oct 30 13:44:31.191629 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:31.191571    4364 scope.go:122] "RemoveContainer" containerID="87b706902f5786f2505a8bb78db3558d561529d5e0d8ce0476a0bee04e885419"
Oct 30 13:44:31.200512 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:31.200490    4364 scope.go:122] "RemoveContainer" containerID="1070c42cc863836cf2980d60febfcb8d13f8bb15303a6085e9eb1a2bb304041d"
Oct 30 13:44:31.213044 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:31.212949    4364 scope.go:122] "RemoveContainer" containerID="1070c42cc863836cf2980d60febfcb8d13f8bb15303a6085e9eb1a2bb304041d"
Oct 30 13:44:31.213519 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:44:31.213496    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1070c42cc863836cf2980d60febfcb8d13f8bb15303a6085e9eb1a2bb304041d\": container with ID starting with 1070c42cc863836cf2980d60febfcb8d13f8bb15303a6085e9eb1a2bb304041d not found: ID does not exist" containerID="1070c42cc863836cf2980d60febfcb8d13f8bb15303a6085e9eb1a2bb304041d"
Oct 30 13:44:31.213661 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:31.213534    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1070c42cc863836cf2980d60febfcb8d13f8bb15303a6085e9eb1a2bb304041d"} err="failed to get container status \"1070c42cc863836cf2980d60febfcb8d13f8bb15303a6085e9eb1a2bb304041d\": rpc error: code = NotFound desc = could not find container \"1070c42cc863836cf2980d60febfcb8d13f8bb15303a6085e9eb1a2bb304041d\": container with ID starting with 1070c42cc863836cf2980d60febfcb8d13f8bb15303a6085e9eb1a2bb304041d not found: ID does not exist"
Oct 30 13:44:31.247355 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:31.247285    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l"]
Oct 30 13:44:31.261229 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:31.261204    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-4xg9l"]
Oct 30 13:44:31.293276 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:31.293250    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-d833bd/distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74"]
Oct 30 13:44:31.296967 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:31.296944    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-d833bd/distinct-2-w3-d833bd-kserve-568fd95dfc-5qn74"]
Oct 30 13:44:31.335360 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:31.335333    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6"]
Oct 30 13:44:31.339520 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:31.339496    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-d833bd/premium-w3-d833bd-kserve-67ff8fcbc5-mqjl6"]
Oct 30 13:44:32.972563 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:32.972529    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="297ed651-f2f1-40f4-bbe9-bf93abf2ac8d" path="/var/lib/kubelet/pods/297ed651-f2f1-40f4-bbe9-bf93abf2ac8d/volumes"
Oct 30 13:44:32.973255 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:32.973234    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f2b06ac-9bf8-4b57-bd22-85532576f9c1" path="/var/lib/kubelet/pods/4f2b06ac-9bf8-4b57-bd22-85532576f9c1/volumes"
Oct 30 13:44:32.973841 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:32.973823    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1" path="/var/lib/kubelet/pods/b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1/volumes"
Oct 30 13:44:33.029348 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.027900    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-9c6033-855647cb84-pkjp9"]
Oct 30 13:44:33.094703 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.094671    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9c6033-6546465fff-z4rrd"]
Oct 30 13:44:33.133532 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.133504    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2"
Oct 30 13:44:33.164692 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.164669    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv"
Oct 30 13:44:33.202405 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.202372    4364 generic.go:356] "Generic (PLEG): container finished" podID="7090457f-3bbf-4295-aa35-9dedc02d1b61" containerID="d37963fcb3d6d226584f31b5f028e9f4bd77fc3f6b346e1f40eafd201535a154" exitCode=0
Oct 30 13:44:33.202548 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.202408    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv"
Oct 30 13:44:33.202548 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.202446    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv" event={"ID":"7090457f-3bbf-4295-aa35-9dedc02d1b61","Type":"ContainerDied","Data":"d37963fcb3d6d226584f31b5f028e9f4bd77fc3f6b346e1f40eafd201535a154"}
Oct 30 13:44:33.202548 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.202498    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv" event={"ID":"7090457f-3bbf-4295-aa35-9dedc02d1b61","Type":"ContainerDied","Data":"185cba57705b65253a8f05e0d6a7b90b0bcc35314e703edacfe7893680a11e48"}
Oct 30 13:44:33.202548 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.202529    4364 scope.go:122] "RemoveContainer" containerID="d37963fcb3d6d226584f31b5f028e9f4bd77fc3f6b346e1f40eafd201535a154"
Oct 30 13:44:33.204448 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.204427    4364 generic.go:356] "Generic (PLEG): container finished" podID="fd17aa36-e0ae-4652-a85f-20945233ccc0" containerID="dde43c13d515a38bb07a29af73448f8a91f8d16989b8908d1f16060414f5b88d" exitCode=0
Oct 30 13:44:33.204560 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.204476    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2"
Oct 30 13:44:33.204560 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.204502    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2" event={"ID":"fd17aa36-e0ae-4652-a85f-20945233ccc0","Type":"ContainerDied","Data":"dde43c13d515a38bb07a29af73448f8a91f8d16989b8908d1f16060414f5b88d"}
Oct 30 13:44:33.204716 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.204554    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2" event={"ID":"fd17aa36-e0ae-4652-a85f-20945233ccc0","Type":"ContainerDied","Data":"774d1b268fdcd67b576c6dc19fde9c2285bc1cd45daa7c439283ee7ca3529c7a"}
Oct 30 13:44:33.212928 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.212910    4364 scope.go:122] "RemoveContainer" containerID="d37963fcb3d6d226584f31b5f028e9f4bd77fc3f6b346e1f40eafd201535a154"
Oct 30 13:44:33.213185 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:44:33.213166    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d37963fcb3d6d226584f31b5f028e9f4bd77fc3f6b346e1f40eafd201535a154\": container with ID starting with d37963fcb3d6d226584f31b5f028e9f4bd77fc3f6b346e1f40eafd201535a154 not found: ID does not exist" containerID="d37963fcb3d6d226584f31b5f028e9f4bd77fc3f6b346e1f40eafd201535a154"
Oct 30 13:44:33.213226 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.213193    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d37963fcb3d6d226584f31b5f028e9f4bd77fc3f6b346e1f40eafd201535a154"} err="failed to get container status \"d37963fcb3d6d226584f31b5f028e9f4bd77fc3f6b346e1f40eafd201535a154\": rpc error: code = NotFound desc = could not find container \"d37963fcb3d6d226584f31b5f028e9f4bd77fc3f6b346e1f40eafd201535a154\": container with ID starting with d37963fcb3d6d226584f31b5f028e9f4bd77fc3f6b346e1f40eafd201535a154 not found: ID does not exist"
Oct 30 13:44:33.213226 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.213209    4364 scope.go:122] "RemoveContainer" containerID="dde43c13d515a38bb07a29af73448f8a91f8d16989b8908d1f16060414f5b88d"
Oct 30 13:44:33.222754 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.222724    4364 scope.go:122] "RemoveContainer" containerID="b96ac07b11db5c0d57da34eada5d105abdfb0f1c9e05bd72d8f06f3357c2d022"
Oct 30 13:44:33.234536 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.234512    4364 scope.go:122] "RemoveContainer" containerID="dde43c13d515a38bb07a29af73448f8a91f8d16989b8908d1f16060414f5b88d"
Oct 30 13:44:33.234909 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:44:33.234869    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dde43c13d515a38bb07a29af73448f8a91f8d16989b8908d1f16060414f5b88d\": container with ID starting with dde43c13d515a38bb07a29af73448f8a91f8d16989b8908d1f16060414f5b88d not found: ID does not exist" containerID="dde43c13d515a38bb07a29af73448f8a91f8d16989b8908d1f16060414f5b88d"
Oct 30 13:44:33.235041 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.234907    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dde43c13d515a38bb07a29af73448f8a91f8d16989b8908d1f16060414f5b88d"} err="failed to get container status \"dde43c13d515a38bb07a29af73448f8a91f8d16989b8908d1f16060414f5b88d\": rpc error: code = NotFound desc = could not find container \"dde43c13d515a38bb07a29af73448f8a91f8d16989b8908d1f16060414f5b88d\": container with ID starting with dde43c13d515a38bb07a29af73448f8a91f8d16989b8908d1f16060414f5b88d not found: ID does not exist"
Oct 30 13:44:33.235041 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.234930    4364 scope.go:122] "RemoveContainer" containerID="b96ac07b11db5c0d57da34eada5d105abdfb0f1c9e05bd72d8f06f3357c2d022"
Oct 30 13:44:33.235226 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:44:33.235202    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b96ac07b11db5c0d57da34eada5d105abdfb0f1c9e05bd72d8f06f3357c2d022\": container with ID starting with b96ac07b11db5c0d57da34eada5d105abdfb0f1c9e05bd72d8f06f3357c2d022 not found: ID does not exist" containerID="b96ac07b11db5c0d57da34eada5d105abdfb0f1c9e05bd72d8f06f3357c2d022"
Oct 30 13:44:33.235295 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.235236    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b96ac07b11db5c0d57da34eada5d105abdfb0f1c9e05bd72d8f06f3357c2d022"} err="failed to get container status \"b96ac07b11db5c0d57da34eada5d105abdfb0f1c9e05bd72d8f06f3357c2d022\": rpc error: code = NotFound desc = could not find container \"b96ac07b11db5c0d57da34eada5d105abdfb0f1c9e05bd72d8f06f3357c2d022\": container with ID starting with b96ac07b11db5c0d57da34eada5d105abdfb0f1c9e05bd72d8f06f3357c2d022 not found: ID does not exist"
Oct 30 13:44:33.252779 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.252756    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd17aa36-e0ae-4652-a85f-20945233ccc0-kube-api-access-d8bnk\" (UniqueName: \"kubernetes.io/projected/fd17aa36-e0ae-4652-a85f-20945233ccc0-kube-api-access-d8bnk\") pod \"fd17aa36-e0ae-4652-a85f-20945233ccc0\" (UID: \"fd17aa36-e0ae-4652-a85f-20945233ccc0\") "
Oct 30 13:44:33.252904 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.252865    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fd17aa36-e0ae-4652-a85f-20945233ccc0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd17aa36-e0ae-4652-a85f-20945233ccc0-istio-podinfo\") pod \"fd17aa36-e0ae-4652-a85f-20945233ccc0\" (UID: \"fd17aa36-e0ae-4652-a85f-20945233ccc0\") "
Oct 30 13:44:33.252972 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.252906    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd17aa36-e0ae-4652-a85f-20945233ccc0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd17aa36-e0ae-4652-a85f-20945233ccc0-istiod-ca-cert\") pod \"fd17aa36-e0ae-4652-a85f-20945233ccc0\" (UID: \"fd17aa36-e0ae-4652-a85f-20945233ccc0\") "
Oct 30 13:44:33.252972 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.252939    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd17aa36-e0ae-4652-a85f-20945233ccc0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd17aa36-e0ae-4652-a85f-20945233ccc0-istio-data\") pod \"fd17aa36-e0ae-4652-a85f-20945233ccc0\" (UID: \"fd17aa36-e0ae-4652-a85f-20945233ccc0\") "
Oct 30 13:44:33.253105 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.252982    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd17aa36-e0ae-4652-a85f-20945233ccc0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd17aa36-e0ae-4652-a85f-20945233ccc0-istio-envoy\") pod \"fd17aa36-e0ae-4652-a85f-20945233ccc0\" (UID: \"fd17aa36-e0ae-4652-a85f-20945233ccc0\") "
Oct 30 13:44:33.253105 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.253097    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd17aa36-e0ae-4652-a85f-20945233ccc0-istio-token\" (UniqueName: \"kubernetes.io/projected/fd17aa36-e0ae-4652-a85f-20945233ccc0-istio-token\") pod \"fd17aa36-e0ae-4652-a85f-20945233ccc0\" (UID: \"fd17aa36-e0ae-4652-a85f-20945233ccc0\") "
Oct 30 13:44:33.253215 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.253158    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd17aa36-e0ae-4652-a85f-20945233ccc0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd17aa36-e0ae-4652-a85f-20945233ccc0-workload-certs\") pod \"fd17aa36-e0ae-4652-a85f-20945233ccc0\" (UID: \"fd17aa36-e0ae-4652-a85f-20945233ccc0\") "
Oct 30 13:44:33.253215 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.253194    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd17aa36-e0ae-4652-a85f-20945233ccc0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd17aa36-e0ae-4652-a85f-20945233ccc0-credential-socket\") pod \"fd17aa36-e0ae-4652-a85f-20945233ccc0\" (UID: \"fd17aa36-e0ae-4652-a85f-20945233ccc0\") "
Oct 30 13:44:33.253331 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.253239    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd17aa36-e0ae-4652-a85f-20945233ccc0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd17aa36-e0ae-4652-a85f-20945233ccc0-workload-socket\") pod \"fd17aa36-e0ae-4652-a85f-20945233ccc0\" (UID: \"fd17aa36-e0ae-4652-a85f-20945233ccc0\") "
Oct 30 13:44:33.253331 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.253297    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd17aa36-e0ae-4652-a85f-20945233ccc0-istio-data" pod "fd17aa36-e0ae-4652-a85f-20945233ccc0" (UID: "fd17aa36-e0ae-4652-a85f-20945233ccc0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:33.253331 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.253305    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd17aa36-e0ae-4652-a85f-20945233ccc0-istiod-ca-cert" pod "fd17aa36-e0ae-4652-a85f-20945233ccc0" (UID: "fd17aa36-e0ae-4652-a85f-20945233ccc0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:44:33.253504 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.253440    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd17aa36-e0ae-4652-a85f-20945233ccc0-workload-certs" pod "fd17aa36-e0ae-4652-a85f-20945233ccc0" (UID: "fd17aa36-e0ae-4652-a85f-20945233ccc0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:33.253571 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.253535    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd17aa36-e0ae-4652-a85f-20945233ccc0-credential-socket" pod "fd17aa36-e0ae-4652-a85f-20945233ccc0" (UID: "fd17aa36-e0ae-4652-a85f-20945233ccc0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:33.253571 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.253555    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd17aa36-e0ae-4652-a85f-20945233ccc0-workload-socket" pod "fd17aa36-e0ae-4652-a85f-20945233ccc0" (UID: "fd17aa36-e0ae-4652-a85f-20945233ccc0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:33.254029 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.254003    4364 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd17aa36-e0ae-4652-a85f-20945233ccc0-workload-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:33.254114 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.254034    4364 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd17aa36-e0ae-4652-a85f-20945233ccc0-credential-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:33.254114 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.254050    4364 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd17aa36-e0ae-4652-a85f-20945233ccc0-workload-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:33.254114 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.254066    4364 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd17aa36-e0ae-4652-a85f-20945233ccc0-istiod-ca-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:33.254114 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.254080    4364 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd17aa36-e0ae-4652-a85f-20945233ccc0-istio-data\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:33.255366 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.255339    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd17aa36-e0ae-4652-a85f-20945233ccc0-istio-token" pod "fd17aa36-e0ae-4652-a85f-20945233ccc0" (UID: "fd17aa36-e0ae-4652-a85f-20945233ccc0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:44:33.255486 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.255398    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd17aa36-e0ae-4652-a85f-20945233ccc0-kube-api-access-d8bnk" pod "fd17aa36-e0ae-4652-a85f-20945233ccc0" (UID: "fd17aa36-e0ae-4652-a85f-20945233ccc0"). InnerVolumeSpecName "kube-api-access-d8bnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:44:33.255656 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.255631    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd17aa36-e0ae-4652-a85f-20945233ccc0-istio-podinfo" pod "fd17aa36-e0ae-4652-a85f-20945233ccc0" (UID: "fd17aa36-e0ae-4652-a85f-20945233ccc0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:44:33.255748 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.255725    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd17aa36-e0ae-4652-a85f-20945233ccc0-istio-envoy" pod "fd17aa36-e0ae-4652-a85f-20945233ccc0" (UID: "fd17aa36-e0ae-4652-a85f-20945233ccc0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:33.355365 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.355323    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7090457f-3bbf-4295-aa35-9dedc02d1b61-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7090457f-3bbf-4295-aa35-9dedc02d1b61-istio-envoy\") pod \"7090457f-3bbf-4295-aa35-9dedc02d1b61\" (UID: \"7090457f-3bbf-4295-aa35-9dedc02d1b61\") "
Oct 30 13:44:33.355365 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.355381    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7090457f-3bbf-4295-aa35-9dedc02d1b61-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7090457f-3bbf-4295-aa35-9dedc02d1b61-istiod-ca-cert\") pod \"7090457f-3bbf-4295-aa35-9dedc02d1b61\" (UID: \"7090457f-3bbf-4295-aa35-9dedc02d1b61\") "
Oct 30 13:44:33.355654 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.355416    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7090457f-3bbf-4295-aa35-9dedc02d1b61-kube-api-access-p97n9\" (UniqueName: \"kubernetes.io/projected/7090457f-3bbf-4295-aa35-9dedc02d1b61-kube-api-access-p97n9\") pod \"7090457f-3bbf-4295-aa35-9dedc02d1b61\" (UID: \"7090457f-3bbf-4295-aa35-9dedc02d1b61\") "
Oct 30 13:44:33.355654 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.355451    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7090457f-3bbf-4295-aa35-9dedc02d1b61-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7090457f-3bbf-4295-aa35-9dedc02d1b61-istio-podinfo\") pod \"7090457f-3bbf-4295-aa35-9dedc02d1b61\" (UID: \"7090457f-3bbf-4295-aa35-9dedc02d1b61\") "
Oct 30 13:44:33.355654 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.355575    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7090457f-3bbf-4295-aa35-9dedc02d1b61-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7090457f-3bbf-4295-aa35-9dedc02d1b61-workload-certs\") pod \"7090457f-3bbf-4295-aa35-9dedc02d1b61\" (UID: \"7090457f-3bbf-4295-aa35-9dedc02d1b61\") "
Oct 30 13:44:33.355866 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.355687    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7090457f-3bbf-4295-aa35-9dedc02d1b61-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7090457f-3bbf-4295-aa35-9dedc02d1b61-credential-socket\") pod \"7090457f-3bbf-4295-aa35-9dedc02d1b61\" (UID: \"7090457f-3bbf-4295-aa35-9dedc02d1b61\") "
Oct 30 13:44:33.355866 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.355768    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7090457f-3bbf-4295-aa35-9dedc02d1b61-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7090457f-3bbf-4295-aa35-9dedc02d1b61-workload-socket\") pod \"7090457f-3bbf-4295-aa35-9dedc02d1b61\" (UID: \"7090457f-3bbf-4295-aa35-9dedc02d1b61\") "
Oct 30 13:44:33.355866 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.355833    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7090457f-3bbf-4295-aa35-9dedc02d1b61-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7090457f-3bbf-4295-aa35-9dedc02d1b61-istio-data\") pod \"7090457f-3bbf-4295-aa35-9dedc02d1b61\" (UID: \"7090457f-3bbf-4295-aa35-9dedc02d1b61\") "
Oct 30 13:44:33.356029 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.355874    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7090457f-3bbf-4295-aa35-9dedc02d1b61-istio-token\" (UniqueName: \"kubernetes.io/projected/7090457f-3bbf-4295-aa35-9dedc02d1b61-istio-token\") pod \"7090457f-3bbf-4295-aa35-9dedc02d1b61\" (UID: \"7090457f-3bbf-4295-aa35-9dedc02d1b61\") "
Oct 30 13:44:33.356029 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.355957    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7090457f-3bbf-4295-aa35-9dedc02d1b61-credential-socket" pod "7090457f-3bbf-4295-aa35-9dedc02d1b61" (UID: "7090457f-3bbf-4295-aa35-9dedc02d1b61"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:33.356029 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.355976    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7090457f-3bbf-4295-aa35-9dedc02d1b61-istiod-ca-cert" pod "7090457f-3bbf-4295-aa35-9dedc02d1b61" (UID: "7090457f-3bbf-4295-aa35-9dedc02d1b61"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:44:33.356200 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.356174    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7090457f-3bbf-4295-aa35-9dedc02d1b61-workload-socket" pod "7090457f-3bbf-4295-aa35-9dedc02d1b61" (UID: "7090457f-3bbf-4295-aa35-9dedc02d1b61"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:33.356200 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.356187    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7090457f-3bbf-4295-aa35-9dedc02d1b61-istio-data" pod "7090457f-3bbf-4295-aa35-9dedc02d1b61" (UID: "7090457f-3bbf-4295-aa35-9dedc02d1b61"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:33.356200 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.356177    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7090457f-3bbf-4295-aa35-9dedc02d1b61-workload-certs" pod "7090457f-3bbf-4295-aa35-9dedc02d1b61" (UID: "7090457f-3bbf-4295-aa35-9dedc02d1b61"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:33.356445 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.356420    4364 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7090457f-3bbf-4295-aa35-9dedc02d1b61-workload-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:33.356621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.356450    4364 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7090457f-3bbf-4295-aa35-9dedc02d1b61-credential-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:33.356621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.356464    4364 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd17aa36-e0ae-4652-a85f-20945233ccc0-istio-token\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:33.356621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.356479    4364 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7090457f-3bbf-4295-aa35-9dedc02d1b61-workload-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:33.356621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.356492    4364 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7090457f-3bbf-4295-aa35-9dedc02d1b61-istio-data\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:33.356621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.356502    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8bnk\" (UniqueName: \"kubernetes.io/projected/fd17aa36-e0ae-4652-a85f-20945233ccc0-kube-api-access-d8bnk\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:33.356621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.356511    4364 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7090457f-3bbf-4295-aa35-9dedc02d1b61-istiod-ca-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:33.356621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.356519    4364 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd17aa36-e0ae-4652-a85f-20945233ccc0-istio-podinfo\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:33.356621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.356530    4364 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd17aa36-e0ae-4652-a85f-20945233ccc0-istio-envoy\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:33.358501 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.358472    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7090457f-3bbf-4295-aa35-9dedc02d1b61-istio-envoy" pod "7090457f-3bbf-4295-aa35-9dedc02d1b61" (UID: "7090457f-3bbf-4295-aa35-9dedc02d1b61"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:33.358688 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.358480    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7090457f-3bbf-4295-aa35-9dedc02d1b61-istio-token" pod "7090457f-3bbf-4295-aa35-9dedc02d1b61" (UID: "7090457f-3bbf-4295-aa35-9dedc02d1b61"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:44:33.358762 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.358689    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7090457f-3bbf-4295-aa35-9dedc02d1b61-kube-api-access-p97n9" pod "7090457f-3bbf-4295-aa35-9dedc02d1b61" (UID: "7090457f-3bbf-4295-aa35-9dedc02d1b61"). InnerVolumeSpecName "kube-api-access-p97n9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:44:33.358833 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.358816    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7090457f-3bbf-4295-aa35-9dedc02d1b61-istio-podinfo" pod "7090457f-3bbf-4295-aa35-9dedc02d1b61" (UID: "7090457f-3bbf-4295-aa35-9dedc02d1b61"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:44:33.457741 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.457678    4364 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7090457f-3bbf-4295-aa35-9dedc02d1b61-istio-token\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:33.457741 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.457703    4364 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7090457f-3bbf-4295-aa35-9dedc02d1b61-istio-envoy\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:33.457741 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.457713    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p97n9\" (UniqueName: \"kubernetes.io/projected/7090457f-3bbf-4295-aa35-9dedc02d1b61-kube-api-access-p97n9\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:33.457741 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.457722    4364 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7090457f-3bbf-4295-aa35-9dedc02d1b61-istio-podinfo\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:33.554580 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.554553    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv"]
Oct 30 13:44:33.558940 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.558915    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-hk7xv"]
Oct 30 13:44:33.598342 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.598318    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2"]
Oct 30 13:44:33.601696 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:33.601677    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-78pd2"]
Oct 30 13:44:34.972376 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:34.972334    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7090457f-3bbf-4295-aa35-9dedc02d1b61" path="/var/lib/kubelet/pods/7090457f-3bbf-4295-aa35-9dedc02d1b61/volumes"
Oct 30 13:44:34.973284 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:34.973258    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd17aa36-e0ae-4652-a85f-20945233ccc0" path="/var/lib/kubelet/pods/fd17aa36-e0ae-4652-a85f-20945233ccc0/volumes"
Oct 30 13:44:37.002123 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:37.002087    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6da60a-589f9f6ddb-vcldm"]
Oct 30 13:44:37.002664 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:37.002293    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6da60a-589f9f6ddb-vcldm" podUID="29ab696e-02cd-40c3-a170-32545b8407f3" containerName="maas-api" containerID="cri-o://74f1e147dc5b3a66551e6586e81bd3863e4a3e426f84e630d846cd4f11090fed" gracePeriod=30
Oct 30 13:44:37.280917 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:37.280835    4364 generic.go:356] "Generic (PLEG): container finished" podID="29ab696e-02cd-40c3-a170-32545b8407f3" containerID="74f1e147dc5b3a66551e6586e81bd3863e4a3e426f84e630d846cd4f11090fed" exitCode=0
Oct 30 13:44:37.281092 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:37.281019    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6da60a-589f9f6ddb-vcldm" event={"ID":"29ab696e-02cd-40c3-a170-32545b8407f3","Type":"ContainerDied","Data":"74f1e147dc5b3a66551e6586e81bd3863e4a3e426f84e630d846cd4f11090fed"}
Oct 30 13:44:37.316024 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:37.315996    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6da60a-589f9f6ddb-vcldm"
Oct 30 13:44:37.417118 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:37.417089    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29ab696e-02cd-40c3-a170-32545b8407f3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29ab696e-02cd-40c3-a170-32545b8407f3-maas-api-tls\") pod \"29ab696e-02cd-40c3-a170-32545b8407f3\" (UID: \"29ab696e-02cd-40c3-a170-32545b8407f3\") "
Oct 30 13:44:37.417118 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:37.417121    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29ab696e-02cd-40c3-a170-32545b8407f3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/29ab696e-02cd-40c3-a170-32545b8407f3-metrics-certs\") pod \"29ab696e-02cd-40c3-a170-32545b8407f3\" (UID: \"29ab696e-02cd-40c3-a170-32545b8407f3\") "
Oct 30 13:44:37.417328 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:37.417176    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29ab696e-02cd-40c3-a170-32545b8407f3-kube-api-access-7b95g\" (UniqueName: \"kubernetes.io/projected/29ab696e-02cd-40c3-a170-32545b8407f3-kube-api-access-7b95g\") pod \"29ab696e-02cd-40c3-a170-32545b8407f3\" (UID: \"29ab696e-02cd-40c3-a170-32545b8407f3\") "
Oct 30 13:44:37.417328 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:37.417197    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29ab696e-02cd-40c3-a170-32545b8407f3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29ab696e-02cd-40c3-a170-32545b8407f3-openshift-service-ca\") pod \"29ab696e-02cd-40c3-a170-32545b8407f3\" (UID: \"29ab696e-02cd-40c3-a170-32545b8407f3\") "
Oct 30 13:44:37.417687 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:37.417658    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29ab696e-02cd-40c3-a170-32545b8407f3-openshift-service-ca" pod "29ab696e-02cd-40c3-a170-32545b8407f3" (UID: "29ab696e-02cd-40c3-a170-32545b8407f3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:44:37.417838 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:37.417760    4364 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29ab696e-02cd-40c3-a170-32545b8407f3-openshift-service-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:37.419209 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:37.419176    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29ab696e-02cd-40c3-a170-32545b8407f3-maas-api-tls" pod "29ab696e-02cd-40c3-a170-32545b8407f3" (UID: "29ab696e-02cd-40c3-a170-32545b8407f3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:44:37.419331 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:37.419247    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29ab696e-02cd-40c3-a170-32545b8407f3-metrics-certs" pod "29ab696e-02cd-40c3-a170-32545b8407f3" (UID: "29ab696e-02cd-40c3-a170-32545b8407f3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:44:37.419405 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:37.419383    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29ab696e-02cd-40c3-a170-32545b8407f3-kube-api-access-7b95g" pod "29ab696e-02cd-40c3-a170-32545b8407f3" (UID: "29ab696e-02cd-40c3-a170-32545b8407f3"). InnerVolumeSpecName "kube-api-access-7b95g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:44:37.519379 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:37.519343    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29ab696e-02cd-40c3-a170-32545b8407f3-maas-api-tls\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:37.519379 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:37.519378    4364 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29ab696e-02cd-40c3-a170-32545b8407f3-metrics-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:37.519379 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:37.519387    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7b95g\" (UniqueName: \"kubernetes.io/projected/29ab696e-02cd-40c3-a170-32545b8407f3-kube-api-access-7b95g\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:38.031500 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:38.031442    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-9c6033-855647cb84-pkjp9" podUID="3a5999a2-3c46-4554-8b5b-0e1ccf464ec8" containerName="payload-processing" containerID="cri-o://2bf3c9192711fe8aec4966e4f745f5294735a4991f5bd8529fdbf4cd36ab7c95" gracePeriod=25
Oct 30 13:44:38.095900 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:38.095862    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9c6033-6546465fff-z4rrd" podUID="362b87d6-a207-44a8-a247-92bc1e05e554" containerName="payload-pre-processing" containerID="cri-o://56faa51cac10db60007730500063d06a536488b96e31d9fd0ee541fc9a53519e" gracePeriod=25
Oct 30 13:44:38.289778 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:38.289691    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6da60a-589f9f6ddb-vcldm" event={"ID":"29ab696e-02cd-40c3-a170-32545b8407f3","Type":"ContainerDied","Data":"4bab3638f76027a461a3f567d74b891269dab44088098f02bbd1b754c32aba40"}
Oct 30 13:44:38.289778 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:38.289748    4364 scope.go:122] "RemoveContainer" containerID="74f1e147dc5b3a66551e6586e81bd3863e4a3e426f84e630d846cd4f11090fed"
Oct 30 13:44:38.290000 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:38.289706    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6da60a-589f9f6ddb-vcldm"
Oct 30 13:44:38.350307 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:38.350275    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6da60a-589f9f6ddb-vcldm"]
Oct 30 13:44:38.364506 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:38.364475    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6da60a-589f9f6ddb-vcldm"]
Oct 30 13:44:38.975443 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:38.975409    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29ab696e-02cd-40c3-a170-32545b8407f3" path="/var/lib/kubelet/pods/29ab696e-02cd-40c3-a170-32545b8407f3/volumes"
Oct 30 13:44:39.640540 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:39.640504    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-8d82d5-65c88b865b-xt6lk"]
Oct 30 13:44:39.641012 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:39.640785    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-8d82d5-65c88b865b-xt6lk" podUID="4813393d-187f-4a3a-a77b-6151b211d2f8" containerName="maas-api" containerID="cri-o://b652338b553545dd0fc3e24484182bc304b269266e06e732bc56b139550c884f" gracePeriod=30
Oct 30 13:44:39.915960 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:39.915930    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-8d82d5-65c88b865b-xt6lk"
Oct 30 13:44:40.062477 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:40.062440    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4813393d-187f-4a3a-a77b-6151b211d2f8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4813393d-187f-4a3a-a77b-6151b211d2f8-openshift-service-ca\") pod \"4813393d-187f-4a3a-a77b-6151b211d2f8\" (UID: \"4813393d-187f-4a3a-a77b-6151b211d2f8\") "
Oct 30 13:44:40.062692 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:40.062532    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4813393d-187f-4a3a-a77b-6151b211d2f8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4813393d-187f-4a3a-a77b-6151b211d2f8-metrics-certs\") pod \"4813393d-187f-4a3a-a77b-6151b211d2f8\" (UID: \"4813393d-187f-4a3a-a77b-6151b211d2f8\") "
Oct 30 13:44:40.062692 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:40.062664    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4813393d-187f-4a3a-a77b-6151b211d2f8-kube-api-access-tgcxl\" (UniqueName: \"kubernetes.io/projected/4813393d-187f-4a3a-a77b-6151b211d2f8-kube-api-access-tgcxl\") pod \"4813393d-187f-4a3a-a77b-6151b211d2f8\" (UID: \"4813393d-187f-4a3a-a77b-6151b211d2f8\") "
Oct 30 13:44:40.062820 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:40.062701    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4813393d-187f-4a3a-a77b-6151b211d2f8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4813393d-187f-4a3a-a77b-6151b211d2f8-maas-api-tls\") pod \"4813393d-187f-4a3a-a77b-6151b211d2f8\" (UID: \"4813393d-187f-4a3a-a77b-6151b211d2f8\") "
Oct 30 13:44:40.062880 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:40.062825    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4813393d-187f-4a3a-a77b-6151b211d2f8-openshift-service-ca" pod "4813393d-187f-4a3a-a77b-6151b211d2f8" (UID: "4813393d-187f-4a3a-a77b-6151b211d2f8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:44:40.063242 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:40.063223    4364 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4813393d-187f-4a3a-a77b-6151b211d2f8-openshift-service-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:40.064726 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:40.064699    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4813393d-187f-4a3a-a77b-6151b211d2f8-metrics-certs" pod "4813393d-187f-4a3a-a77b-6151b211d2f8" (UID: "4813393d-187f-4a3a-a77b-6151b211d2f8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:44:40.064813 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:40.064736    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4813393d-187f-4a3a-a77b-6151b211d2f8-maas-api-tls" pod "4813393d-187f-4a3a-a77b-6151b211d2f8" (UID: "4813393d-187f-4a3a-a77b-6151b211d2f8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:44:40.064813 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:40.064747    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4813393d-187f-4a3a-a77b-6151b211d2f8-kube-api-access-tgcxl" pod "4813393d-187f-4a3a-a77b-6151b211d2f8" (UID: "4813393d-187f-4a3a-a77b-6151b211d2f8"). InnerVolumeSpecName "kube-api-access-tgcxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:44:40.171218 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:40.171113    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgcxl\" (UniqueName: \"kubernetes.io/projected/4813393d-187f-4a3a-a77b-6151b211d2f8-kube-api-access-tgcxl\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:40.171218 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:40.171151    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4813393d-187f-4a3a-a77b-6151b211d2f8-maas-api-tls\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:40.171218 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:40.171166    4364 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4813393d-187f-4a3a-a77b-6151b211d2f8-metrics-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:40.338776 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:40.338737    4364 generic.go:356] "Generic (PLEG): container finished" podID="4813393d-187f-4a3a-a77b-6151b211d2f8" containerID="b652338b553545dd0fc3e24484182bc304b269266e06e732bc56b139550c884f" exitCode=0
Oct 30 13:44:40.338959 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:40.338800    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-8d82d5-65c88b865b-xt6lk"
Oct 30 13:44:40.338959 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:40.338826    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-8d82d5-65c88b865b-xt6lk" event={"ID":"4813393d-187f-4a3a-a77b-6151b211d2f8","Type":"ContainerDied","Data":"b652338b553545dd0fc3e24484182bc304b269266e06e732bc56b139550c884f"}
Oct 30 13:44:40.338959 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:40.338906    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-8d82d5-65c88b865b-xt6lk" event={"ID":"4813393d-187f-4a3a-a77b-6151b211d2f8","Type":"ContainerDied","Data":"70a5480174238e72e167548405e15adbb23486f4980c61ac5b8a7f106a0c1d4e"}
Oct 30 13:44:40.338959 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:40.338927    4364 scope.go:122] "RemoveContainer" containerID="b652338b553545dd0fc3e24484182bc304b269266e06e732bc56b139550c884f"
Oct 30 13:44:40.349467 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:40.349449    4364 scope.go:122] "RemoveContainer" containerID="b652338b553545dd0fc3e24484182bc304b269266e06e732bc56b139550c884f"
Oct 30 13:44:40.349734 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:44:40.349717    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b652338b553545dd0fc3e24484182bc304b269266e06e732bc56b139550c884f\": container with ID starting with b652338b553545dd0fc3e24484182bc304b269266e06e732bc56b139550c884f not found: ID does not exist" containerID="b652338b553545dd0fc3e24484182bc304b269266e06e732bc56b139550c884f"
Oct 30 13:44:40.349783 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:40.349741    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b652338b553545dd0fc3e24484182bc304b269266e06e732bc56b139550c884f"} err="failed to get container status \"b652338b553545dd0fc3e24484182bc304b269266e06e732bc56b139550c884f\": rpc error: code = NotFound desc = could not find container \"b652338b553545dd0fc3e24484182bc304b269266e06e732bc56b139550c884f\": container with ID starting with b652338b553545dd0fc3e24484182bc304b269266e06e732bc56b139550c884f not found: ID does not exist"
Oct 30 13:44:40.400863 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:40.400833    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-8d82d5-65c88b865b-xt6lk"]
Oct 30 13:44:40.406671 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:40.406642    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-8d82d5-65c88b865b-xt6lk"]
Oct 30 13:44:40.970729 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:40.970695    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4813393d-187f-4a3a-a77b-6151b211d2f8" path="/var/lib/kubelet/pods/4813393d-187f-4a3a-a77b-6151b211d2f8/volumes"
Oct 30 13:44:41.503840 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:44:41.503802    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:44:43.406302 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:43.406269    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc"]
Oct 30 13:44:43.406839 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:43.406540    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc" podUID="d675cfa9-7df1-446a-984d-a4e1b0dda352" containerName="istio-proxy" containerID="cri-o://8e3c0c6a8fc0b33845319cb8efd3eb01abefade605420501182f7c59c42c47ce" gracePeriod=30
Oct 30 13:44:44.341013 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:44.340972    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9c6033-6546465fff-z4rrd" podUID="362b87d6-a207-44a8-a247-92bc1e05e554" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:44:44.341013 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:44.341006    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-9c6033-855647cb84-pkjp9" podUID="3a5999a2-3c46-4554-8b5b-0e1ccf464ec8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:44:45.400276 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:45.400224    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:44:46.386031 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:46.385993    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-d833bd-5cf45d8d99-8jl44"]
Oct 30 13:44:46.386664 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:46.386627    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d833bd-5cf45d8d99-8jl44" podUID="8188711a-ba90-4fdd-9184-1b604692e9bb" containerName="maas-api" containerID="cri-o://b372056a6f75c6ae7ebea664eb874e25338ffd0fbcc9ca07e9362a8d08d549e0" gracePeriod=30
Oct 30 13:44:46.784946 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:46.784919    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d833bd-5cf45d8d99-8jl44"
Oct 30 13:44:46.786973 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:46.786942    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8188711a-ba90-4fdd-9184-1b604692e9bb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8188711a-ba90-4fdd-9184-1b604692e9bb-maas-api-tls\") pod \"8188711a-ba90-4fdd-9184-1b604692e9bb\" (UID: \"8188711a-ba90-4fdd-9184-1b604692e9bb\") "
Oct 30 13:44:46.788136 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:46.787017    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8188711a-ba90-4fdd-9184-1b604692e9bb-kube-api-access-n88j2\" (UniqueName: \"kubernetes.io/projected/8188711a-ba90-4fdd-9184-1b604692e9bb-kube-api-access-n88j2\") pod \"8188711a-ba90-4fdd-9184-1b604692e9bb\" (UID: \"8188711a-ba90-4fdd-9184-1b604692e9bb\") "
Oct 30 13:44:46.788136 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:46.787052    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8188711a-ba90-4fdd-9184-1b604692e9bb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8188711a-ba90-4fdd-9184-1b604692e9bb-openshift-service-ca\") pod \"8188711a-ba90-4fdd-9184-1b604692e9bb\" (UID: \"8188711a-ba90-4fdd-9184-1b604692e9bb\") "
Oct 30 13:44:46.788136 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:46.787256    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8188711a-ba90-4fdd-9184-1b604692e9bb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8188711a-ba90-4fdd-9184-1b604692e9bb-metrics-certs\") pod \"8188711a-ba90-4fdd-9184-1b604692e9bb\" (UID: \"8188711a-ba90-4fdd-9184-1b604692e9bb\") "
Oct 30 13:44:46.788136 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:46.787630    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8188711a-ba90-4fdd-9184-1b604692e9bb-openshift-service-ca" pod "8188711a-ba90-4fdd-9184-1b604692e9bb" (UID: "8188711a-ba90-4fdd-9184-1b604692e9bb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:44:46.788136 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:46.787894    4364 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8188711a-ba90-4fdd-9184-1b604692e9bb-openshift-service-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:46.830778 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:46.830737    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8188711a-ba90-4fdd-9184-1b604692e9bb-maas-api-tls" pod "8188711a-ba90-4fdd-9184-1b604692e9bb" (UID: "8188711a-ba90-4fdd-9184-1b604692e9bb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:44:46.835857 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:46.834822    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8188711a-ba90-4fdd-9184-1b604692e9bb-kube-api-access-n88j2" pod "8188711a-ba90-4fdd-9184-1b604692e9bb" (UID: "8188711a-ba90-4fdd-9184-1b604692e9bb"). InnerVolumeSpecName "kube-api-access-n88j2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:44:46.836293 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:46.836263    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8188711a-ba90-4fdd-9184-1b604692e9bb-metrics-certs" pod "8188711a-ba90-4fdd-9184-1b604692e9bb" (UID: "8188711a-ba90-4fdd-9184-1b604692e9bb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:44:46.890622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:46.890495    4364 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8188711a-ba90-4fdd-9184-1b604692e9bb-metrics-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:46.890622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:46.890528    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8188711a-ba90-4fdd-9184-1b604692e9bb-maas-api-tls\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:46.890622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:46.890539    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n88j2\" (UniqueName: \"kubernetes.io/projected/8188711a-ba90-4fdd-9184-1b604692e9bb-kube-api-access-n88j2\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:47.476580 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:47.476543    4364 generic.go:356] "Generic (PLEG): container finished" podID="8188711a-ba90-4fdd-9184-1b604692e9bb" containerID="b372056a6f75c6ae7ebea664eb874e25338ffd0fbcc9ca07e9362a8d08d549e0" exitCode=0
Oct 30 13:44:47.476801 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:47.476633    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d833bd-5cf45d8d99-8jl44"
Oct 30 13:44:47.476801 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:47.476667    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d833bd-5cf45d8d99-8jl44" event={"ID":"8188711a-ba90-4fdd-9184-1b604692e9bb","Type":"ContainerDied","Data":"b372056a6f75c6ae7ebea664eb874e25338ffd0fbcc9ca07e9362a8d08d549e0"}
Oct 30 13:44:47.476801 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:47.476720    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d833bd-5cf45d8d99-8jl44" event={"ID":"8188711a-ba90-4fdd-9184-1b604692e9bb","Type":"ContainerDied","Data":"a6a90edbd76f034f20d0ede2efa336e3382082f3f2dd96a034e1e5e65b925279"}
Oct 30 13:44:47.476801 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:47.476751    4364 scope.go:122] "RemoveContainer" containerID="b372056a6f75c6ae7ebea664eb874e25338ffd0fbcc9ca07e9362a8d08d549e0"
Oct 30 13:44:47.488867 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:47.488843    4364 scope.go:122] "RemoveContainer" containerID="b372056a6f75c6ae7ebea664eb874e25338ffd0fbcc9ca07e9362a8d08d549e0"
Oct 30 13:44:47.489265 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:44:47.489231    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b372056a6f75c6ae7ebea664eb874e25338ffd0fbcc9ca07e9362a8d08d549e0\": container with ID starting with b372056a6f75c6ae7ebea664eb874e25338ffd0fbcc9ca07e9362a8d08d549e0 not found: ID does not exist" containerID="b372056a6f75c6ae7ebea664eb874e25338ffd0fbcc9ca07e9362a8d08d549e0"
Oct 30 13:44:47.489348 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:47.489278    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b372056a6f75c6ae7ebea664eb874e25338ffd0fbcc9ca07e9362a8d08d549e0"} err="failed to get container status \"b372056a6f75c6ae7ebea664eb874e25338ffd0fbcc9ca07e9362a8d08d549e0\": rpc error: code = NotFound desc = could not find container \"b372056a6f75c6ae7ebea664eb874e25338ffd0fbcc9ca07e9362a8d08d549e0\": container with ID starting with b372056a6f75c6ae7ebea664eb874e25338ffd0fbcc9ca07e9362a8d08d549e0 not found: ID does not exist"
Oct 30 13:44:47.528686 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:47.528661    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk"]
Oct 30 13:44:47.528895 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:47.528874    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk" podUID="5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b" containerName="istio-proxy" containerID="cri-o://fed1ac4064786654b61c4c54e7b0f18094223daf7de251f5114b989759c18992" gracePeriod=30
Oct 30 13:44:47.572790 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:47.572758    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-d833bd-5cf45d8d99-8jl44"]
Oct 30 13:44:47.580931 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:47.580905    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-d833bd-5cf45d8d99-8jl44"]
Oct 30 13:44:48.167285 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.167252    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f07dfe-bbj94"]
Oct 30 13:44:48.167795 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.167781    4364 cpu_manager.go:414] "containerMap: removing container" podUID="3de36fb7-459f-44d2-ba6a-b2cc20d24516" containerName="maas-api"
Oct 30 13:44:48.167845 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.167796    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3de36fb7-459f-44d2-ba6a-b2cc20d24516" containerName="maas-api"
Oct 30 13:44:48.167845 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.167816    4364 cpu_manager.go:414] "containerMap: removing container" podUID="446c12df-de12-4c7c-8c71-dfed93cc33f1" containerName="main"
Oct 30 13:44:48.167845 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.167821    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="446c12df-de12-4c7c-8c71-dfed93cc33f1" containerName="main"
Oct 30 13:44:48.167845 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.167830    4364 cpu_manager.go:414] "containerMap: removing container" podUID="213f7500-6291-40df-8a5e-c7008a6ccc47" containerName="maas-api"
Oct 30 13:44:48.167845 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.167835    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="213f7500-6291-40df-8a5e-c7008a6ccc47" containerName="maas-api"
Oct 30 13:44:48.167845 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.167841    4364 cpu_manager.go:414] "containerMap: removing container" podUID="5ce05a81-71f3-4dab-96e1-44c9ea628a7b" containerName="istio-proxy"
Oct 30 13:44:48.167845 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.167846    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ce05a81-71f3-4dab-96e1-44c9ea628a7b" containerName="istio-proxy"
Oct 30 13:44:48.168072 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.167853    4364 cpu_manager.go:414] "containerMap: removing container" podUID="864a2c8d-97ba-4141-957c-0810628ba6ec" containerName="istio-proxy"
Oct 30 13:44:48.168072 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.167858    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="864a2c8d-97ba-4141-957c-0810628ba6ec" containerName="istio-proxy"
Oct 30 13:44:48.168072 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.167865    4364 cpu_manager.go:414] "containerMap: removing container" podUID="c2597429-b2f7-4263-b715-19410183a9a7" containerName="payload-pre-processing"
Oct 30 13:44:48.168072 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.167869    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2597429-b2f7-4263-b715-19410183a9a7" containerName="payload-pre-processing"
Oct 30 13:44:48.168072 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.167876    4364 cpu_manager.go:414] "containerMap: removing container" podUID="80f1162e-fa26-45b9-8016-c4812edb5bad" containerName="maas-api"
Oct 30 13:44:48.168072 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.167881    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80f1162e-fa26-45b9-8016-c4812edb5bad" containerName="maas-api"
Oct 30 13:44:48.168072 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.167888    4364 cpu_manager.go:414] "containerMap: removing container" podUID="9a967f02-7c93-484e-9bd4-97e1a4f5fdf5" containerName="istio-proxy"
Oct 30 13:44:48.168072 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.167892    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a967f02-7c93-484e-9bd4-97e1a4f5fdf5" containerName="istio-proxy"
Oct 30 13:44:48.168072 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.167897    4364 cpu_manager.go:414] "containerMap: removing container" podUID="0b5b1e9d-9ea8-430d-9ff0-42b382a5f6a6" containerName="maas-api"
Oct 30 13:44:48.168072 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.167902    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b5b1e9d-9ea8-430d-9ff0-42b382a5f6a6" containerName="maas-api"
Oct 30 13:44:48.168072 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.167913    4364 cpu_manager.go:414] "containerMap: removing container" podUID="7090457f-3bbf-4295-aa35-9dedc02d1b61" containerName="istio-proxy"
Oct 30 13:44:48.168072 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.167919    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7090457f-3bbf-4295-aa35-9dedc02d1b61" containerName="istio-proxy"
Oct 30 13:44:48.168072 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.167926    4364 cpu_manager.go:414] "containerMap: removing container" podUID="c71d1db6-aeed-434f-9829-a6589831daac" containerName="istio-proxy"
Oct 30 13:44:48.168072 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.167931    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c71d1db6-aeed-434f-9829-a6589831daac" containerName="istio-proxy"
Oct 30 13:44:48.168072 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.167937    4364 cpu_manager.go:414] "containerMap: removing container" podUID="89934eb4-b154-412f-9f02-2f33c0fcb2e3" containerName="maas-api"
Oct 30 13:44:48.168072 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.167941    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89934eb4-b154-412f-9f02-2f33c0fcb2e3" containerName="maas-api"
Oct 30 13:44:48.168072 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.167948    4364 cpu_manager.go:414] "containerMap: removing container" podUID="0a9a3c04-3da8-4efe-8daf-b4821c563a24" containerName="istio-proxy"
Oct 30 13:44:48.168072 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.167952    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a9a3c04-3da8-4efe-8daf-b4821c563a24" containerName="istio-proxy"
Oct 30 13:44:48.168072 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.167958    4364 cpu_manager.go:414] "containerMap: removing container" podUID="8188711a-ba90-4fdd-9184-1b604692e9bb" containerName="maas-api"
Oct 30 13:44:48.168072 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.167962    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8188711a-ba90-4fdd-9184-1b604692e9bb" containerName="maas-api"
Oct 30 13:44:48.168072 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.167969    4364 cpu_manager.go:414] "containerMap: removing container" podUID="e77381ab-005b-4b76-81b0-ada11566adca" containerName="main"
Oct 30 13:44:48.168072 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.167973    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e77381ab-005b-4b76-81b0-ada11566adca" containerName="main"
Oct 30 13:44:48.168072 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.167979    4364 cpu_manager.go:414] "containerMap: removing container" podUID="d98dc22b-5142-4d00-bf71-27ead6f76d3e" containerName="maas-api"
Oct 30 13:44:48.168072 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.167983    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d98dc22b-5142-4d00-bf71-27ead6f76d3e" containerName="maas-api"
Oct 30 13:44:48.168072 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.167988    4364 cpu_manager.go:414] "containerMap: removing container" podUID="e5c6ed73-3541-4945-8d38-cb289a4e05f8" containerName="payload-pre-processing"
Oct 30 13:44:48.168072 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.167994    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5c6ed73-3541-4945-8d38-cb289a4e05f8" containerName="payload-pre-processing"
Oct 30 13:44:48.168072 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168004    4364 cpu_manager.go:414] "containerMap: removing container" podUID="864a2c8d-97ba-4141-957c-0810628ba6ec" containerName="istio-proxy"
Oct 30 13:44:48.168072 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168009    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="864a2c8d-97ba-4141-957c-0810628ba6ec" containerName="istio-proxy"
Oct 30 13:44:48.168072 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168017    4364 cpu_manager.go:414] "containerMap: removing container" podUID="23116724-6a57-401b-8ba4-a105a71a5e6d" containerName="istio-proxy"
Oct 30 13:44:48.168072 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168022    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23116724-6a57-401b-8ba4-a105a71a5e6d" containerName="istio-proxy"
Oct 30 13:44:48.168072 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168029    4364 cpu_manager.go:414] "containerMap: removing container" podUID="f42a48a8-5f5e-459c-8276-dd30d02052a7" containerName="payload-processing"
Oct 30 13:44:48.168072 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168034    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f42a48a8-5f5e-459c-8276-dd30d02052a7" containerName="payload-processing"
Oct 30 13:44:48.168072 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168043    4364 cpu_manager.go:414] "containerMap: removing container" podUID="fe89b812-7b37-4eeb-8a92-07c436511948" containerName="istio-proxy"
Oct 30 13:44:48.168072 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168047    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe89b812-7b37-4eeb-8a92-07c436511948" containerName="istio-proxy"
Oct 30 13:44:48.168072 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168058    4364 cpu_manager.go:414] "containerMap: removing container" podUID="4813393d-187f-4a3a-a77b-6151b211d2f8" containerName="maas-api"
Oct 30 13:44:48.168072 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168062    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4813393d-187f-4a3a-a77b-6151b211d2f8" containerName="maas-api"
Oct 30 13:44:48.168072 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168068    4364 cpu_manager.go:414] "containerMap: removing container" podUID="71576777-83eb-4fb6-af0e-33c177bc2dd0" containerName="maas-api"
Oct 30 13:44:48.168072 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168073    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71576777-83eb-4fb6-af0e-33c177bc2dd0" containerName="maas-api"
Oct 30 13:44:48.168072 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168080    4364 cpu_manager.go:414] "containerMap: removing container" podUID="94f6ed0a-3007-4d3a-a569-920348b7da49" containerName="maas-api"
Oct 30 13:44:48.168072 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168084    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94f6ed0a-3007-4d3a-a569-920348b7da49" containerName="maas-api"
Oct 30 13:44:48.168072 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168090    4364 cpu_manager.go:414] "containerMap: removing container" podUID="43de3339-76dc-44c5-9ce6-9e592b0fceba" containerName="payload-processing"
Oct 30 13:44:48.169347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168095    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43de3339-76dc-44c5-9ce6-9e592b0fceba" containerName="payload-processing"
Oct 30 13:44:48.169347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168103    4364 cpu_manager.go:414] "containerMap: removing container" podUID="297ed651-f2f1-40f4-bbe9-bf93abf2ac8d" containerName="istio-proxy"
Oct 30 13:44:48.169347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168108    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="297ed651-f2f1-40f4-bbe9-bf93abf2ac8d" containerName="istio-proxy"
Oct 30 13:44:48.169347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168114    4364 cpu_manager.go:414] "containerMap: removing container" podUID="29ab696e-02cd-40c3-a170-32545b8407f3" containerName="maas-api"
Oct 30 13:44:48.169347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168119    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29ab696e-02cd-40c3-a170-32545b8407f3" containerName="maas-api"
Oct 30 13:44:48.169347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168129    4364 cpu_manager.go:414] "containerMap: removing container" podUID="fd17aa36-e0ae-4652-a85f-20945233ccc0" containerName="istio-proxy"
Oct 30 13:44:48.169347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168134    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd17aa36-e0ae-4652-a85f-20945233ccc0" containerName="istio-proxy"
Oct 30 13:44:48.169347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168140    4364 cpu_manager.go:414] "containerMap: removing container" podUID="0a9a3c04-3da8-4efe-8daf-b4821c563a24" containerName="istio-proxy"
Oct 30 13:44:48.169347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168144    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a9a3c04-3da8-4efe-8daf-b4821c563a24" containerName="istio-proxy"
Oct 30 13:44:48.169347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168151    4364 cpu_manager.go:414] "containerMap: removing container" podUID="0b45c4c8-a116-43ce-9e06-6b4ceca907e2" containerName="payload-pre-processing"
Oct 30 13:44:48.169347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168155    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b45c4c8-a116-43ce-9e06-6b4ceca907e2" containerName="payload-pre-processing"
Oct 30 13:44:48.169347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168162    4364 cpu_manager.go:414] "containerMap: removing container" podUID="93fc7bda-b92c-41ce-b567-658a571bc5ba" containerName="maas-api"
Oct 30 13:44:48.169347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168166    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93fc7bda-b92c-41ce-b567-658a571bc5ba" containerName="maas-api"
Oct 30 13:44:48.169347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168175    4364 cpu_manager.go:414] "containerMap: removing container" podUID="b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1" containerName="main"
Oct 30 13:44:48.169347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168179    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1" containerName="main"
Oct 30 13:44:48.169347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168188    4364 cpu_manager.go:414] "containerMap: removing container" podUID="966c9a37-1b20-416b-aa28-c9703fca7fbe" containerName="maas-api"
Oct 30 13:44:48.169347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168193    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="966c9a37-1b20-416b-aa28-c9703fca7fbe" containerName="maas-api"
Oct 30 13:44:48.169347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168198    4364 cpu_manager.go:414] "containerMap: removing container" podUID="fd17aa36-e0ae-4652-a85f-20945233ccc0" containerName="istio-proxy"
Oct 30 13:44:48.169347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168203    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd17aa36-e0ae-4652-a85f-20945233ccc0" containerName="istio-proxy"
Oct 30 13:44:48.169347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168210    4364 cpu_manager.go:414] "containerMap: removing container" podUID="4f2b06ac-9bf8-4b57-bd22-85532576f9c1" containerName="main"
Oct 30 13:44:48.169347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168214    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f2b06ac-9bf8-4b57-bd22-85532576f9c1" containerName="main"
Oct 30 13:44:48.169347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168219    4364 cpu_manager.go:414] "containerMap: removing container" podUID="43e6fe99-7d56-417f-b9db-1d41a24340d2" containerName="payload-processing"
Oct 30 13:44:48.169347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168223    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43e6fe99-7d56-417f-b9db-1d41a24340d2" containerName="payload-processing"
Oct 30 13:44:48.169347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168231    4364 cpu_manager.go:414] "containerMap: removing container" podUID="09c95c32-f60a-4f60-90ff-90f447247cce" containerName="istio-proxy"
Oct 30 13:44:48.169347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168235    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09c95c32-f60a-4f60-90ff-90f447247cce" containerName="istio-proxy"
Oct 30 13:44:48.169347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168245    4364 cpu_manager.go:414] "containerMap: removing container" podUID="09bcc5ff-a33b-4699-a797-58b18a5f63bd" containerName="istio-proxy"
Oct 30 13:44:48.169347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168249    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09bcc5ff-a33b-4699-a797-58b18a5f63bd" containerName="istio-proxy"
Oct 30 13:44:48.169347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168320    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd17aa36-e0ae-4652-a85f-20945233ccc0" containerName="istio-proxy"
Oct 30 13:44:48.169347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168328    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a9a3c04-3da8-4efe-8daf-b4821c563a24" containerName="istio-proxy"
Oct 30 13:44:48.169347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168334    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b45c4c8-a116-43ce-9e06-6b4ceca907e2" containerName="payload-pre-processing"
Oct 30 13:44:48.169347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168340    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="93fc7bda-b92c-41ce-b567-658a571bc5ba" containerName="maas-api"
Oct 30 13:44:48.169347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168346    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ce05a81-71f3-4dab-96e1-44c9ea628a7b" containerName="istio-proxy"
Oct 30 13:44:48.169347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168353    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2ec47d4-90ed-4752-88b9-f5db4d4ee7e1" containerName="main"
Oct 30 13:44:48.169347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168359    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="966c9a37-1b20-416b-aa28-c9703fca7fbe" containerName="maas-api"
Oct 30 13:44:48.169347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168363    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd17aa36-e0ae-4652-a85f-20945233ccc0" containerName="istio-proxy"
Oct 30 13:44:48.169347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168368    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f2b06ac-9bf8-4b57-bd22-85532576f9c1" containerName="main"
Oct 30 13:44:48.169347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168373    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="43e6fe99-7d56-417f-b9db-1d41a24340d2" containerName="payload-processing"
Oct 30 13:44:48.169347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168379    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="09c95c32-f60a-4f60-90ff-90f447247cce" containerName="istio-proxy"
Oct 30 13:44:48.169347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168384    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="09bcc5ff-a33b-4699-a797-58b18a5f63bd" containerName="istio-proxy"
Oct 30 13:44:48.169347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168389    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="3de36fb7-459f-44d2-ba6a-b2cc20d24516" containerName="maas-api"
Oct 30 13:44:48.169347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168395    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="446c12df-de12-4c7c-8c71-dfed93cc33f1" containerName="main"
Oct 30 13:44:48.169347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168400    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="213f7500-6291-40df-8a5e-c7008a6ccc47" containerName="maas-api"
Oct 30 13:44:48.169347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168404    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ce05a81-71f3-4dab-96e1-44c9ea628a7b" containerName="istio-proxy"
Oct 30 13:44:48.169347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168409    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="864a2c8d-97ba-4141-957c-0810628ba6ec" containerName="istio-proxy"
Oct 30 13:44:48.169347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168416    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2597429-b2f7-4263-b715-19410183a9a7" containerName="payload-pre-processing"
Oct 30 13:44:48.169347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168421    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="80f1162e-fa26-45b9-8016-c4812edb5bad" containerName="maas-api"
Oct 30 13:44:48.169347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168426    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a967f02-7c93-484e-9bd4-97e1a4f5fdf5" containerName="istio-proxy"
Oct 30 13:44:48.169347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168431    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b5b1e9d-9ea8-430d-9ff0-42b382a5f6a6" containerName="maas-api"
Oct 30 13:44:48.169347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168437    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="7090457f-3bbf-4295-aa35-9dedc02d1b61" containerName="istio-proxy"
Oct 30 13:44:48.169347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168441    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="c71d1db6-aeed-434f-9829-a6589831daac" containerName="istio-proxy"
Oct 30 13:44:48.169347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168446    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="89934eb4-b154-412f-9f02-2f33c0fcb2e3" containerName="maas-api"
Oct 30 13:44:48.169347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168452    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a9a3c04-3da8-4efe-8daf-b4821c563a24" containerName="istio-proxy"
Oct 30 13:44:48.169347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168456    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="8188711a-ba90-4fdd-9184-1b604692e9bb" containerName="maas-api"
Oct 30 13:44:48.169347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168460    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="e77381ab-005b-4b76-81b0-ada11566adca" containerName="main"
Oct 30 13:44:48.169347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168465    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="d98dc22b-5142-4d00-bf71-27ead6f76d3e" containerName="maas-api"
Oct 30 13:44:48.169347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168471    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5c6ed73-3541-4945-8d38-cb289a4e05f8" containerName="payload-pre-processing"
Oct 30 13:44:48.169347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168477    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="864a2c8d-97ba-4141-957c-0810628ba6ec" containerName="istio-proxy"
Oct 30 13:44:48.169347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168483    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="23116724-6a57-401b-8ba4-a105a71a5e6d" containerName="istio-proxy"
Oct 30 13:44:48.169347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168488    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="f42a48a8-5f5e-459c-8276-dd30d02052a7" containerName="payload-processing"
Oct 30 13:44:48.169347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168493    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe89b812-7b37-4eeb-8a92-07c436511948" containerName="istio-proxy"
Oct 30 13:44:48.169347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168499    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a967f02-7c93-484e-9bd4-97e1a4f5fdf5" containerName="istio-proxy"
Oct 30 13:44:48.169347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168504    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="4813393d-187f-4a3a-a77b-6151b211d2f8" containerName="maas-api"
Oct 30 13:44:48.169347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168509    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="71576777-83eb-4fb6-af0e-33c177bc2dd0" containerName="maas-api"
Oct 30 13:44:48.169347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168514    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="94f6ed0a-3007-4d3a-a569-920348b7da49" containerName="maas-api"
Oct 30 13:44:48.169347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168520    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="43de3339-76dc-44c5-9ce6-9e592b0fceba" containerName="payload-processing"
Oct 30 13:44:48.169347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168525    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="297ed651-f2f1-40f4-bbe9-bf93abf2ac8d" containerName="istio-proxy"
Oct 30 13:44:48.169347 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.168530    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="29ab696e-02cd-40c3-a170-32545b8407f3" containerName="maas-api"
Oct 30 13:44:48.173302 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.173286    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f07dfe-bbj94"
Oct 30 13:44:48.194130 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.194103    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f07dfe-bbj94"]
Oct 30 13:44:48.204368 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.204342    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:44:48.204501 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.204388    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\""
Oct 30 13:44:48.204557 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.204526    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9frpl\" (UniqueName: \"kubernetes.io/projected/f1ebdbad-f9d4-4758-a7a8-e2513b6d837d-kube-api-access-9frpl\") pod \"maas-api-revoke-keys-e2e-worker-w0-f07dfe-bbj94\" (UID: \"f1ebdbad-f9d4-4758-a7a8-e2513b6d837d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f07dfe-bbj94"
Oct 30 13:44:48.204617 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.204571    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1ebdbad-f9d4-4758-a7a8-e2513b6d837d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f07dfe-bbj94\" (UID: \"f1ebdbad-f9d4-4758-a7a8-e2513b6d837d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f07dfe-bbj94"
Oct 30 13:44:48.204766 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.204750    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1ebdbad-f9d4-4758-a7a8-e2513b6d837d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f07dfe-bbj94\" (UID: \"f1ebdbad-f9d4-4758-a7a8-e2513b6d837d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f07dfe-bbj94"
Oct 30 13:44:48.305767 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.305731    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1ebdbad-f9d4-4758-a7a8-e2513b6d837d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f07dfe-bbj94\" (UID: \"f1ebdbad-f9d4-4758-a7a8-e2513b6d837d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f07dfe-bbj94"
Oct 30 13:44:48.305940 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.305845    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1ebdbad-f9d4-4758-a7a8-e2513b6d837d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f07dfe-bbj94\" (UID: \"f1ebdbad-f9d4-4758-a7a8-e2513b6d837d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f07dfe-bbj94"
Oct 30 13:44:48.305995 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.305945    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9frpl\" (UniqueName: \"kubernetes.io/projected/f1ebdbad-f9d4-4758-a7a8-e2513b6d837d-kube-api-access-9frpl\") pod \"maas-api-revoke-keys-e2e-worker-w0-f07dfe-bbj94\" (UID: \"f1ebdbad-f9d4-4758-a7a8-e2513b6d837d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f07dfe-bbj94"
Oct 30 13:44:48.306413 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.306389    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1ebdbad-f9d4-4758-a7a8-e2513b6d837d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f07dfe-bbj94\" (UID: \"f1ebdbad-f9d4-4758-a7a8-e2513b6d837d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f07dfe-bbj94"
Oct 30 13:44:48.308278 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.308254    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1ebdbad-f9d4-4758-a7a8-e2513b6d837d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f07dfe-bbj94\" (UID: \"f1ebdbad-f9d4-4758-a7a8-e2513b6d837d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f07dfe-bbj94"
Oct 30 13:44:48.331355 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.331331    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9frpl\" (UniqueName: \"kubernetes.io/projected/f1ebdbad-f9d4-4758-a7a8-e2513b6d837d-kube-api-access-9frpl\") pod \"maas-api-revoke-keys-e2e-worker-w0-f07dfe-bbj94\" (UID: \"f1ebdbad-f9d4-4758-a7a8-e2513b6d837d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f07dfe-bbj94"
Oct 30 13:44:48.487382 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.487356    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f07dfe-bbj94"
Oct 30 13:44:48.690427 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.690370    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc"
Oct 30 13:44:48.709680 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.709655    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d675cfa9-7df1-446a-984d-a4e1b0dda352-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d675cfa9-7df1-446a-984d-a4e1b0dda352-istio-podinfo\") pod \"d675cfa9-7df1-446a-984d-a4e1b0dda352\" (UID: \"d675cfa9-7df1-446a-984d-a4e1b0dda352\") "
Oct 30 13:44:48.709834 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.709736    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d675cfa9-7df1-446a-984d-a4e1b0dda352-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d675cfa9-7df1-446a-984d-a4e1b0dda352-istio-data\") pod \"d675cfa9-7df1-446a-984d-a4e1b0dda352\" (UID: \"d675cfa9-7df1-446a-984d-a4e1b0dda352\") "
Oct 30 13:44:48.709834 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.709797    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d675cfa9-7df1-446a-984d-a4e1b0dda352-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d675cfa9-7df1-446a-984d-a4e1b0dda352-istio-envoy\") pod \"d675cfa9-7df1-446a-984d-a4e1b0dda352\" (UID: \"d675cfa9-7df1-446a-984d-a4e1b0dda352\") "
Oct 30 13:44:48.709939 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.709882    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d675cfa9-7df1-446a-984d-a4e1b0dda352-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d675cfa9-7df1-446a-984d-a4e1b0dda352-credential-socket\") pod \"d675cfa9-7df1-446a-984d-a4e1b0dda352\" (UID: \"d675cfa9-7df1-446a-984d-a4e1b0dda352\") "
Oct 30 13:44:48.710019 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.709934    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d675cfa9-7df1-446a-984d-a4e1b0dda352-istio-token\" (UniqueName: \"kubernetes.io/projected/d675cfa9-7df1-446a-984d-a4e1b0dda352-istio-token\") pod \"d675cfa9-7df1-446a-984d-a4e1b0dda352\" (UID: \"d675cfa9-7df1-446a-984d-a4e1b0dda352\") "
Oct 30 13:44:48.710019 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.709963    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d675cfa9-7df1-446a-984d-a4e1b0dda352-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d675cfa9-7df1-446a-984d-a4e1b0dda352-workload-certs\") pod \"d675cfa9-7df1-446a-984d-a4e1b0dda352\" (UID: \"d675cfa9-7df1-446a-984d-a4e1b0dda352\") "
Oct 30 13:44:48.710019 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.710012    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d675cfa9-7df1-446a-984d-a4e1b0dda352-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d675cfa9-7df1-446a-984d-a4e1b0dda352-istiod-ca-cert\") pod \"d675cfa9-7df1-446a-984d-a4e1b0dda352\" (UID: \"d675cfa9-7df1-446a-984d-a4e1b0dda352\") "
Oct 30 13:44:48.710215 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.710026    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d675cfa9-7df1-446a-984d-a4e1b0dda352-istio-data" pod "d675cfa9-7df1-446a-984d-a4e1b0dda352" (UID: "d675cfa9-7df1-446a-984d-a4e1b0dda352"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:48.710215 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.710049    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d675cfa9-7df1-446a-984d-a4e1b0dda352-kube-api-access-z8v6l\" (UniqueName: \"kubernetes.io/projected/d675cfa9-7df1-446a-984d-a4e1b0dda352-kube-api-access-z8v6l\") pod \"d675cfa9-7df1-446a-984d-a4e1b0dda352\" (UID: \"d675cfa9-7df1-446a-984d-a4e1b0dda352\") "
Oct 30 13:44:48.710215 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.710145    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d675cfa9-7df1-446a-984d-a4e1b0dda352-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d675cfa9-7df1-446a-984d-a4e1b0dda352-workload-socket\") pod \"d675cfa9-7df1-446a-984d-a4e1b0dda352\" (UID: \"d675cfa9-7df1-446a-984d-a4e1b0dda352\") "
Oct 30 13:44:48.710396 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.710244    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d675cfa9-7df1-446a-984d-a4e1b0dda352-credential-socket" pod "d675cfa9-7df1-446a-984d-a4e1b0dda352" (UID: "d675cfa9-7df1-446a-984d-a4e1b0dda352"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:48.710396 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.710263    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d675cfa9-7df1-446a-984d-a4e1b0dda352-workload-certs" pod "d675cfa9-7df1-446a-984d-a4e1b0dda352" (UID: "d675cfa9-7df1-446a-984d-a4e1b0dda352"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:48.710587 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.710566    4364 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d675cfa9-7df1-446a-984d-a4e1b0dda352-credential-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:48.710709 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.710586    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d675cfa9-7df1-446a-984d-a4e1b0dda352-istiod-ca-cert" pod "d675cfa9-7df1-446a-984d-a4e1b0dda352" (UID: "d675cfa9-7df1-446a-984d-a4e1b0dda352"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:44:48.710709 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.710592    4364 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d675cfa9-7df1-446a-984d-a4e1b0dda352-workload-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:48.710709 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.710650    4364 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d675cfa9-7df1-446a-984d-a4e1b0dda352-istio-data\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:48.710949 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.710798    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d675cfa9-7df1-446a-984d-a4e1b0dda352-workload-socket" pod "d675cfa9-7df1-446a-984d-a4e1b0dda352" (UID: "d675cfa9-7df1-446a-984d-a4e1b0dda352"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:48.712060 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.712036    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d675cfa9-7df1-446a-984d-a4e1b0dda352-istio-podinfo" pod "d675cfa9-7df1-446a-984d-a4e1b0dda352" (UID: "d675cfa9-7df1-446a-984d-a4e1b0dda352"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:44:48.712325 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.712305    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d675cfa9-7df1-446a-984d-a4e1b0dda352-istio-token" pod "d675cfa9-7df1-446a-984d-a4e1b0dda352" (UID: "d675cfa9-7df1-446a-984d-a4e1b0dda352"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:44:48.712420 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.712381    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d675cfa9-7df1-446a-984d-a4e1b0dda352-istio-envoy" pod "d675cfa9-7df1-446a-984d-a4e1b0dda352" (UID: "d675cfa9-7df1-446a-984d-a4e1b0dda352"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:48.712639 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.712594    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d675cfa9-7df1-446a-984d-a4e1b0dda352-kube-api-access-z8v6l" pod "d675cfa9-7df1-446a-984d-a4e1b0dda352" (UID: "d675cfa9-7df1-446a-984d-a4e1b0dda352"). InnerVolumeSpecName "kube-api-access-z8v6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:44:48.796346 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:44:48.796313    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1ebdbad_f9d4_4758_a7a8_e2513b6d837d.slice/crio-15110e4633425b3f6f9596d31570a00940289e47d5305dcc4ee166da1660f181 WatchSource:0}: Error finding container 15110e4633425b3f6f9596d31570a00940289e47d5305dcc4ee166da1660f181: Status 404 returned error can't find the container with id 15110e4633425b3f6f9596d31570a00940289e47d5305dcc4ee166da1660f181
Oct 30 13:44:48.796507 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.796484    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f07dfe-bbj94"]
Oct 30 13:44:48.811533 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.811501    4364 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d675cfa9-7df1-446a-984d-a4e1b0dda352-istio-envoy\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:48.811670 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.811538    4364 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d675cfa9-7df1-446a-984d-a4e1b0dda352-istio-token\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:48.811670 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.811553    4364 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d675cfa9-7df1-446a-984d-a4e1b0dda352-istiod-ca-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:48.811670 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.811567    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8v6l\" (UniqueName: \"kubernetes.io/projected/d675cfa9-7df1-446a-984d-a4e1b0dda352-kube-api-access-z8v6l\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:48.811670 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.811583    4364 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d675cfa9-7df1-446a-984d-a4e1b0dda352-workload-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:48.811670 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.811629    4364 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d675cfa9-7df1-446a-984d-a4e1b0dda352-istio-podinfo\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:48.913105 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.913071    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc"]
Oct 30 13:44:48.970737 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:48.970653    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8188711a-ba90-4fdd-9184-1b604692e9bb" path="/var/lib/kubelet/pods/8188711a-ba90-4fdd-9184-1b604692e9bb/volumes"
Oct 30 13:44:49.450841 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:49.450802    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942"]
Oct 30 13:44:49.451245 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:49.451081    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942" podUID="654161bd-9cf8-4f63-9a86-b312763e75cc" containerName="istio-proxy" containerID="cri-o://1f172dad62aacefaf343b3e23df93faaa6b45b4430a568d93053d8210fb265cc" gracePeriod=30
Oct 30 13:44:49.494858 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:49.494810    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k"]
Oct 30 13:44:49.495072 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:49.495045    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k" podUID="9bed178a-380c-456c-ac60-ed0020bc32f5" containerName="istio-proxy" containerID="cri-o://c0eaf1f1ede3e5d2e012b256f720c4b52985b3ffc4058fcb70a0e6ff72e18a3d" gracePeriod=30
Oct 30 13:44:49.496302 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:49.496266    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f07dfe-bbj94" event={"ID":"f1ebdbad-f9d4-4758-a7a8-e2513b6d837d","Type":"ContainerStarted","Data":"5dcfcba21d591edafd1f1aa0b6dd31105dd8af3311840cba91ec7ad9c6744150"}
Oct 30 13:44:49.496444 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:49.496320    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f07dfe-bbj94" event={"ID":"f1ebdbad-f9d4-4758-a7a8-e2513b6d837d","Type":"ContainerStarted","Data":"15110e4633425b3f6f9596d31570a00940289e47d5305dcc4ee166da1660f181"}
Oct 30 13:44:49.499010 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:49.498983    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc"]
Oct 30 13:44:49.499371 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:49.499178    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc" podUID="0bfddc61-f2f3-4f1a-bb6e-83830a2a058d" containerName="istio-proxy" containerID="cri-o://d1085ae44d933ca483e2acb81446c148d704921311f4910228c917ed6dadb2cd" gracePeriod=30
Oct 30 13:44:49.500478 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:49.500452    4364 generic.go:356] "Generic (PLEG): container finished" podID="d675cfa9-7df1-446a-984d-a4e1b0dda352" containerID="8e3c0c6a8fc0b33845319cb8efd3eb01abefade605420501182f7c59c42c47ce" exitCode=0
Oct 30 13:44:49.500614 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:49.500516    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc" event={"ID":"d675cfa9-7df1-446a-984d-a4e1b0dda352","Type":"ContainerDied","Data":"8e3c0c6a8fc0b33845319cb8efd3eb01abefade605420501182f7c59c42c47ce"}
Oct 30 13:44:49.500614 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:49.500565    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc" event={"ID":"d675cfa9-7df1-446a-984d-a4e1b0dda352","Type":"ContainerDied","Data":"a83f574dc10025974ea934d10547ed491ee0da27a2e3ce8875054659f7184099"}
Oct 30 13:44:49.500614 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:49.500608    4364 scope.go:122] "RemoveContainer" containerID="8e3c0c6a8fc0b33845319cb8efd3eb01abefade605420501182f7c59c42c47ce"
Oct 30 13:44:49.500877 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:49.500859    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc"
Oct 30 13:44:49.541957 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:49.541928    4364 scope.go:122] "RemoveContainer" containerID="42c7c544365e5d4fb53bee70de1e7283d4ce4c0a13e7c83c2f88f1c1ce4c966f"
Oct 30 13:44:49.568190 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:49.568160    4364 scope.go:122] "RemoveContainer" containerID="8e3c0c6a8fc0b33845319cb8efd3eb01abefade605420501182f7c59c42c47ce"
Oct 30 13:44:49.568578 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:44:49.568551    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e3c0c6a8fc0b33845319cb8efd3eb01abefade605420501182f7c59c42c47ce\": container with ID starting with 8e3c0c6a8fc0b33845319cb8efd3eb01abefade605420501182f7c59c42c47ce not found: ID does not exist" containerID="8e3c0c6a8fc0b33845319cb8efd3eb01abefade605420501182f7c59c42c47ce"
Oct 30 13:44:49.568731 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:49.568589    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e3c0c6a8fc0b33845319cb8efd3eb01abefade605420501182f7c59c42c47ce"} err="failed to get container status \"8e3c0c6a8fc0b33845319cb8efd3eb01abefade605420501182f7c59c42c47ce\": rpc error: code = NotFound desc = could not find container \"8e3c0c6a8fc0b33845319cb8efd3eb01abefade605420501182f7c59c42c47ce\": container with ID starting with 8e3c0c6a8fc0b33845319cb8efd3eb01abefade605420501182f7c59c42c47ce not found: ID does not exist"
Oct 30 13:44:49.568731 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:49.568629    4364 scope.go:122] "RemoveContainer" containerID="42c7c544365e5d4fb53bee70de1e7283d4ce4c0a13e7c83c2f88f1c1ce4c966f"
Oct 30 13:44:49.569048 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:44:49.568944    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42c7c544365e5d4fb53bee70de1e7283d4ce4c0a13e7c83c2f88f1c1ce4c966f\": container with ID starting with 42c7c544365e5d4fb53bee70de1e7283d4ce4c0a13e7c83c2f88f1c1ce4c966f not found: ID does not exist" containerID="42c7c544365e5d4fb53bee70de1e7283d4ce4c0a13e7c83c2f88f1c1ce4c966f"
Oct 30 13:44:49.569048 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:49.568976    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42c7c544365e5d4fb53bee70de1e7283d4ce4c0a13e7c83c2f88f1c1ce4c966f"} err="failed to get container status \"42c7c544365e5d4fb53bee70de1e7283d4ce4c0a13e7c83c2f88f1c1ce4c966f\": rpc error: code = NotFound desc = could not find container \"42c7c544365e5d4fb53bee70de1e7283d4ce4c0a13e7c83c2f88f1c1ce4c966f\": container with ID starting with 42c7c544365e5d4fb53bee70de1e7283d4ce4c0a13e7c83c2f88f1c1ce4c966f not found: ID does not exist"
Oct 30 13:44:49.608502 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:49.608449    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f07dfe-bbj94" podStartSLOduration=1.6084355399999999 podStartE2EDuration="1.60843554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:44:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:44:49.604543817 +0000 UTC m=+3188.770908464" watchObservedRunningTime="2026-10-30 13:44:49.60843554 +0000 UTC m=+3188.774800187"
Oct 30 13:44:49.651868 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:49.651830    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc"]
Oct 30 13:44:49.654910 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:49.654883    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-q87xc"]
Oct 30 13:44:50.509756 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:50.509710    4364 generic.go:356] "Generic (PLEG): container finished" podID="f1ebdbad-f9d4-4758-a7a8-e2513b6d837d" containerID="5dcfcba21d591edafd1f1aa0b6dd31105dd8af3311840cba91ec7ad9c6744150" exitCode=0
Oct 30 13:44:50.510149 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:50.509801    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f07dfe-bbj94" event={"ID":"f1ebdbad-f9d4-4758-a7a8-e2513b6d837d","Type":"ContainerDied","Data":"5dcfcba21d591edafd1f1aa0b6dd31105dd8af3311840cba91ec7ad9c6744150"}
Oct 30 13:44:50.970857 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:50.970825    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d675cfa9-7df1-446a-984d-a4e1b0dda352" path="/var/lib/kubelet/pods/d675cfa9-7df1-446a-984d-a4e1b0dda352/volumes"
Oct 30 13:44:51.671062 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:51.671037    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f07dfe-bbj94"
Oct 30 13:44:51.748197 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:51.748160    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1ebdbad-f9d4-4758-a7a8-e2513b6d837d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1ebdbad-f9d4-4758-a7a8-e2513b6d837d-maas-api-xks-ca\") pod \"f1ebdbad-f9d4-4758-a7a8-e2513b6d837d\" (UID: \"f1ebdbad-f9d4-4758-a7a8-e2513b6d837d\") "
Oct 30 13:44:51.748383 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:51.748278    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1ebdbad-f9d4-4758-a7a8-e2513b6d837d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1ebdbad-f9d4-4758-a7a8-e2513b6d837d-maas-api-openshift-ca\") pod \"f1ebdbad-f9d4-4758-a7a8-e2513b6d837d\" (UID: \"f1ebdbad-f9d4-4758-a7a8-e2513b6d837d\") "
Oct 30 13:44:51.748383 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:51.748343    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1ebdbad-f9d4-4758-a7a8-e2513b6d837d-kube-api-access-9frpl\" (UniqueName: \"kubernetes.io/projected/f1ebdbad-f9d4-4758-a7a8-e2513b6d837d-kube-api-access-9frpl\") pod \"f1ebdbad-f9d4-4758-a7a8-e2513b6d837d\" (UID: \"f1ebdbad-f9d4-4758-a7a8-e2513b6d837d\") "
Oct 30 13:44:51.748727 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:51.748698    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1ebdbad-f9d4-4758-a7a8-e2513b6d837d-maas-api-openshift-ca" pod "f1ebdbad-f9d4-4758-a7a8-e2513b6d837d" (UID: "f1ebdbad-f9d4-4758-a7a8-e2513b6d837d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:44:51.748970 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:51.748947    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1ebdbad-f9d4-4758-a7a8-e2513b6d837d-maas-api-openshift-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:51.750520 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:51.750496    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1ebdbad-f9d4-4758-a7a8-e2513b6d837d-kube-api-access-9frpl" pod "f1ebdbad-f9d4-4758-a7a8-e2513b6d837d" (UID: "f1ebdbad-f9d4-4758-a7a8-e2513b6d837d"). InnerVolumeSpecName "kube-api-access-9frpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:44:51.750827 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:51.750810    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1ebdbad-f9d4-4758-a7a8-e2513b6d837d-maas-api-xks-ca" pod "f1ebdbad-f9d4-4758-a7a8-e2513b6d837d" (UID: "f1ebdbad-f9d4-4758-a7a8-e2513b6d837d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:44:51.850127 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:51.850065    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1ebdbad-f9d4-4758-a7a8-e2513b6d837d-maas-api-xks-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:51.850127 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:51.850089    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9frpl\" (UniqueName: \"kubernetes.io/projected/f1ebdbad-f9d4-4758-a7a8-e2513b6d837d-kube-api-access-9frpl\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:52.522983 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:52.522950    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f07dfe-bbj94"
Oct 30 13:44:52.523146 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:52.522983    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f07dfe-bbj94" event={"ID":"f1ebdbad-f9d4-4758-a7a8-e2513b6d837d","Type":"ContainerDied","Data":"15110e4633425b3f6f9596d31570a00940289e47d5305dcc4ee166da1660f181"}
Oct 30 13:44:52.523146 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:52.523033    4364 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15110e4633425b3f6f9596d31570a00940289e47d5305dcc4ee166da1660f181"
Oct 30 13:44:52.798988 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:52.798964    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk"
Oct 30 13:44:52.860484 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:52.860454    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b-workload-socket\") pod \"5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b\" (UID: \"5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b\") "
Oct 30 13:44:52.860670 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:52.860516    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b-kube-api-access-9fkcn\" (UniqueName: \"kubernetes.io/projected/5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b-kube-api-access-9fkcn\") pod \"5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b\" (UID: \"5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b\") "
Oct 30 13:44:52.860670 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:52.860582    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b-istiod-ca-cert\") pod \"5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b\" (UID: \"5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b\") "
Oct 30 13:44:52.860670 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:52.860623    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b-istio-data\") pod \"5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b\" (UID: \"5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b\") "
Oct 30 13:44:52.860834 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:52.860673    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b-istio-podinfo\") pod \"5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b\" (UID: \"5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b\") "
Oct 30 13:44:52.860834 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:52.860795    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b-workload-socket" pod "5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b" (UID: "5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:52.860915 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:52.860810    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b-istio-token\" (UniqueName: \"kubernetes.io/projected/5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b-istio-token\") pod \"5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b\" (UID: \"5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b\") "
Oct 30 13:44:52.861000 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:52.860982    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b-workload-certs\") pod \"5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b\" (UID: \"5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b\") "
Oct 30 13:44:52.861054 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:52.860985    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b-istio-data" pod "5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b" (UID: "5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:52.861054 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:52.861032    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b-credential-socket\") pod \"5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b\" (UID: \"5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b\") "
Oct 30 13:44:52.861175 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:52.861083    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b-istio-envoy\") pod \"5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b\" (UID: \"5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b\") "
Oct 30 13:44:52.861175 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:52.861097    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b-istiod-ca-cert" pod "5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b" (UID: "5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:44:52.861294 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:52.861230    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b-workload-certs" pod "5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b" (UID: "5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:52.861294 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:52.861267    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b-credential-socket" pod "5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b" (UID: "5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:52.861645 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:52.861625    4364 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b-istio-data\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:52.861747 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:52.861649    4364 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b-workload-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:52.861747 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:52.861664    4364 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b-credential-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:52.861747 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:52.861677    4364 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b-workload-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:52.861747 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:52.861690    4364 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b-istiod-ca-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:52.863253 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:52.863232    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b-kube-api-access-9fkcn" pod "5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b" (UID: "5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b"). InnerVolumeSpecName "kube-api-access-9fkcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:44:52.863384 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:52.863355    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b-istio-podinfo" pod "5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b" (UID: "5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:44:52.863500 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:52.863483    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b-istio-envoy" pod "5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b" (UID: "5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:52.863737 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:52.863716    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b-istio-token" pod "5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b" (UID: "5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:44:52.962971 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:52.962927    4364 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b-istio-podinfo\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:52.962971 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:52.962972    4364 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b-istio-token\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:52.963183 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:52.962986    4364 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b-istio-envoy\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:52.963183 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:52.963000    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fkcn\" (UniqueName: \"kubernetes.io/projected/5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b-kube-api-access-9fkcn\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:53.091903 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:53.091828    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz"]
Oct 30 13:44:53.102412 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:53.102383    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz"]
Oct 30 13:44:53.533506 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:53.533469    4364 generic.go:356] "Generic (PLEG): container finished" podID="5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b" containerID="fed1ac4064786654b61c4c54e7b0f18094223daf7de251f5114b989759c18992" exitCode=0
Oct 30 13:44:53.533697 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:53.533552    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk"
Oct 30 13:44:53.533697 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:53.533560    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk" event={"ID":"5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b","Type":"ContainerDied","Data":"fed1ac4064786654b61c4c54e7b0f18094223daf7de251f5114b989759c18992"}
Oct 30 13:44:53.533697 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:53.533654    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk" event={"ID":"5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b","Type":"ContainerDied","Data":"898a68892aeaa7fffeac71fa7e7297b03ceafe9618ea2bc0dd2796886f63dc36"}
Oct 30 13:44:53.533697 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:53.533675    4364 scope.go:122] "RemoveContainer" containerID="fed1ac4064786654b61c4c54e7b0f18094223daf7de251f5114b989759c18992"
Oct 30 13:44:53.549654 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:53.546227    4364 scope.go:122] "RemoveContainer" containerID="15629b602c0d92a0beebeec30ea9d71e7a69fceabc26bf40d512995430aadba2"
Oct 30 13:44:53.557649 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:53.557581    4364 scope.go:122] "RemoveContainer" containerID="fed1ac4064786654b61c4c54e7b0f18094223daf7de251f5114b989759c18992"
Oct 30 13:44:53.558018 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:44:53.557991    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fed1ac4064786654b61c4c54e7b0f18094223daf7de251f5114b989759c18992\": container with ID starting with fed1ac4064786654b61c4c54e7b0f18094223daf7de251f5114b989759c18992 not found: ID does not exist" containerID="fed1ac4064786654b61c4c54e7b0f18094223daf7de251f5114b989759c18992"
Oct 30 13:44:53.558099 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:53.558034    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fed1ac4064786654b61c4c54e7b0f18094223daf7de251f5114b989759c18992"} err="failed to get container status \"fed1ac4064786654b61c4c54e7b0f18094223daf7de251f5114b989759c18992\": rpc error: code = NotFound desc = could not find container \"fed1ac4064786654b61c4c54e7b0f18094223daf7de251f5114b989759c18992\": container with ID starting with fed1ac4064786654b61c4c54e7b0f18094223daf7de251f5114b989759c18992 not found: ID does not exist"
Oct 30 13:44:53.558099 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:53.558056    4364 scope.go:122] "RemoveContainer" containerID="15629b602c0d92a0beebeec30ea9d71e7a69fceabc26bf40d512995430aadba2"
Oct 30 13:44:53.558635 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:44:53.558333    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15629b602c0d92a0beebeec30ea9d71e7a69fceabc26bf40d512995430aadba2\": container with ID starting with 15629b602c0d92a0beebeec30ea9d71e7a69fceabc26bf40d512995430aadba2 not found: ID does not exist" containerID="15629b602c0d92a0beebeec30ea9d71e7a69fceabc26bf40d512995430aadba2"
Oct 30 13:44:53.558635 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:53.558368    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15629b602c0d92a0beebeec30ea9d71e7a69fceabc26bf40d512995430aadba2"} err="failed to get container status \"15629b602c0d92a0beebeec30ea9d71e7a69fceabc26bf40d512995430aadba2\": rpc error: code = NotFound desc = could not find container \"15629b602c0d92a0beebeec30ea9d71e7a69fceabc26bf40d512995430aadba2\": container with ID starting with 15629b602c0d92a0beebeec30ea9d71e7a69fceabc26bf40d512995430aadba2 not found: ID does not exist"
Oct 30 13:44:53.598069 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:53.598031    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk"]
Oct 30 13:44:53.602270 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:53.601149    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-6da60a-openshift-default-755b8899d9-xxdqk"]
Oct 30 13:44:54.340307 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:54.340270    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-9c6033-855647cb84-pkjp9" podUID="3a5999a2-3c46-4554-8b5b-0e1ccf464ec8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:44:54.340704 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:54.340326    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9c6033-6546465fff-z4rrd" podUID="362b87d6-a207-44a8-a247-92bc1e05e554" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:44:54.767774 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:54.767752    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k"
Oct 30 13:44:54.785755 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:54.783430    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-cda0d9-74764bd4cf-l64mp"]
Oct 30 13:44:54.900213 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:54.900171    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bed178a-380c-456c-ac60-ed0020bc32f5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bed178a-380c-456c-ac60-ed0020bc32f5-istiod-ca-cert\") pod \"9bed178a-380c-456c-ac60-ed0020bc32f5\" (UID: \"9bed178a-380c-456c-ac60-ed0020bc32f5\") "
Oct 30 13:44:54.900430 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:54.900403    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bed178a-380c-456c-ac60-ed0020bc32f5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bed178a-380c-456c-ac60-ed0020bc32f5-workload-certs\") pod \"9bed178a-380c-456c-ac60-ed0020bc32f5\" (UID: \"9bed178a-380c-456c-ac60-ed0020bc32f5\") "
Oct 30 13:44:54.901701 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:54.901676    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bed178a-380c-456c-ac60-ed0020bc32f5-kube-api-access-6dcwk\" (UniqueName: \"kubernetes.io/projected/9bed178a-380c-456c-ac60-ed0020bc32f5-kube-api-access-6dcwk\") pod \"9bed178a-380c-456c-ac60-ed0020bc32f5\" (UID: \"9bed178a-380c-456c-ac60-ed0020bc32f5\") "
Oct 30 13:44:54.901865 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:54.901851    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bed178a-380c-456c-ac60-ed0020bc32f5-istio-token\" (UniqueName: \"kubernetes.io/projected/9bed178a-380c-456c-ac60-ed0020bc32f5-istio-token\") pod \"9bed178a-380c-456c-ac60-ed0020bc32f5\" (UID: \"9bed178a-380c-456c-ac60-ed0020bc32f5\") "
Oct 30 13:44:54.902037 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:54.902023    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bed178a-380c-456c-ac60-ed0020bc32f5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bed178a-380c-456c-ac60-ed0020bc32f5-workload-socket\") pod \"9bed178a-380c-456c-ac60-ed0020bc32f5\" (UID: \"9bed178a-380c-456c-ac60-ed0020bc32f5\") "
Oct 30 13:44:54.902191 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:54.902178    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bed178a-380c-456c-ac60-ed0020bc32f5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bed178a-380c-456c-ac60-ed0020bc32f5-istio-data\") pod \"9bed178a-380c-456c-ac60-ed0020bc32f5\" (UID: \"9bed178a-380c-456c-ac60-ed0020bc32f5\") "
Oct 30 13:44:54.902311 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:54.902298    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9bed178a-380c-456c-ac60-ed0020bc32f5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bed178a-380c-456c-ac60-ed0020bc32f5-istio-podinfo\") pod \"9bed178a-380c-456c-ac60-ed0020bc32f5\" (UID: \"9bed178a-380c-456c-ac60-ed0020bc32f5\") "
Oct 30 13:44:54.902471 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:54.902456    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bed178a-380c-456c-ac60-ed0020bc32f5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bed178a-380c-456c-ac60-ed0020bc32f5-istio-envoy\") pod \"9bed178a-380c-456c-ac60-ed0020bc32f5\" (UID: \"9bed178a-380c-456c-ac60-ed0020bc32f5\") "
Oct 30 13:44:54.902660 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:54.902646    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bed178a-380c-456c-ac60-ed0020bc32f5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bed178a-380c-456c-ac60-ed0020bc32f5-credential-socket\") pod \"9bed178a-380c-456c-ac60-ed0020bc32f5\" (UID: \"9bed178a-380c-456c-ac60-ed0020bc32f5\") "
Oct 30 13:44:54.908495 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:54.908472    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bed178a-380c-456c-ac60-ed0020bc32f5-credential-socket" pod "9bed178a-380c-456c-ac60-ed0020bc32f5" (UID: "9bed178a-380c-456c-ac60-ed0020bc32f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:54.908867 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:54.908846    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bed178a-380c-456c-ac60-ed0020bc32f5-workload-socket" pod "9bed178a-380c-456c-ac60-ed0020bc32f5" (UID: "9bed178a-380c-456c-ac60-ed0020bc32f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:54.909205 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:54.909183    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bed178a-380c-456c-ac60-ed0020bc32f5-istio-data" pod "9bed178a-380c-456c-ac60-ed0020bc32f5" (UID: "9bed178a-380c-456c-ac60-ed0020bc32f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:54.910499 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:54.910416    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bed178a-380c-456c-ac60-ed0020bc32f5-istiod-ca-cert" pod "9bed178a-380c-456c-ac60-ed0020bc32f5" (UID: "9bed178a-380c-456c-ac60-ed0020bc32f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:44:54.911087 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:54.911067    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bed178a-380c-456c-ac60-ed0020bc32f5-workload-certs" pod "9bed178a-380c-456c-ac60-ed0020bc32f5" (UID: "9bed178a-380c-456c-ac60-ed0020bc32f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:54.918433 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:54.918306    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bed178a-380c-456c-ac60-ed0020bc32f5-istio-token" pod "9bed178a-380c-456c-ac60-ed0020bc32f5" (UID: "9bed178a-380c-456c-ac60-ed0020bc32f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:44:54.919080 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:54.919045    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bed178a-380c-456c-ac60-ed0020bc32f5-kube-api-access-6dcwk" pod "9bed178a-380c-456c-ac60-ed0020bc32f5" (UID: "9bed178a-380c-456c-ac60-ed0020bc32f5"). InnerVolumeSpecName "kube-api-access-6dcwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:44:54.920732 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:54.920697    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bed178a-380c-456c-ac60-ed0020bc32f5-istio-envoy" pod "9bed178a-380c-456c-ac60-ed0020bc32f5" (UID: "9bed178a-380c-456c-ac60-ed0020bc32f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:54.923247 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:54.923216    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942"
Oct 30 13:44:54.927592 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:54.927514    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9bed178a-380c-456c-ac60-ed0020bc32f5-istio-podinfo" pod "9bed178a-380c-456c-ac60-ed0020bc32f5" (UID: "9bed178a-380c-456c-ac60-ed0020bc32f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:44:54.940866 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:54.940843    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc"
Oct 30 13:44:54.971970 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:54.971937    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b" path="/var/lib/kubelet/pods/5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b/volumes"
Oct 30 13:44:54.979924 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:54.979893    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh"]
Oct 30 13:44:55.007907 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.006351    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0bfddc61-f2f3-4f1a-bb6e-83830a2a058d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bfddc61-f2f3-4f1a-bb6e-83830a2a058d-istio-podinfo\") pod \"0bfddc61-f2f3-4f1a-bb6e-83830a2a058d\" (UID: \"0bfddc61-f2f3-4f1a-bb6e-83830a2a058d\") "
Oct 30 13:44:55.007907 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.006399    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/654161bd-9cf8-4f63-9a86-b312763e75cc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/654161bd-9cf8-4f63-9a86-b312763e75cc-istiod-ca-cert\") pod \"654161bd-9cf8-4f63-9a86-b312763e75cc\" (UID: \"654161bd-9cf8-4f63-9a86-b312763e75cc\") "
Oct 30 13:44:55.007907 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.006471    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/654161bd-9cf8-4f63-9a86-b312763e75cc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/654161bd-9cf8-4f63-9a86-b312763e75cc-istio-podinfo\") pod \"654161bd-9cf8-4f63-9a86-b312763e75cc\" (UID: \"654161bd-9cf8-4f63-9a86-b312763e75cc\") "
Oct 30 13:44:55.007907 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.006499    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/654161bd-9cf8-4f63-9a86-b312763e75cc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/654161bd-9cf8-4f63-9a86-b312763e75cc-workload-certs\") pod \"654161bd-9cf8-4f63-9a86-b312763e75cc\" (UID: \"654161bd-9cf8-4f63-9a86-b312763e75cc\") "
Oct 30 13:44:55.007907 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.006528    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bfddc61-f2f3-4f1a-bb6e-83830a2a058d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bfddc61-f2f3-4f1a-bb6e-83830a2a058d-credential-socket\") pod \"0bfddc61-f2f3-4f1a-bb6e-83830a2a058d\" (UID: \"0bfddc61-f2f3-4f1a-bb6e-83830a2a058d\") "
Oct 30 13:44:55.007907 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.006573    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bfddc61-f2f3-4f1a-bb6e-83830a2a058d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bfddc61-f2f3-4f1a-bb6e-83830a2a058d-workload-certs\") pod \"0bfddc61-f2f3-4f1a-bb6e-83830a2a058d\" (UID: \"0bfddc61-f2f3-4f1a-bb6e-83830a2a058d\") "
Oct 30 13:44:55.007907 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.006621    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bfddc61-f2f3-4f1a-bb6e-83830a2a058d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bfddc61-f2f3-4f1a-bb6e-83830a2a058d-workload-socket\") pod \"0bfddc61-f2f3-4f1a-bb6e-83830a2a058d\" (UID: \"0bfddc61-f2f3-4f1a-bb6e-83830a2a058d\") "
Oct 30 13:44:55.007907 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.006673    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bfddc61-f2f3-4f1a-bb6e-83830a2a058d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bfddc61-f2f3-4f1a-bb6e-83830a2a058d-istiod-ca-cert\") pod \"0bfddc61-f2f3-4f1a-bb6e-83830a2a058d\" (UID: \"0bfddc61-f2f3-4f1a-bb6e-83830a2a058d\") "
Oct 30 13:44:55.007907 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.006742    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bfddc61-f2f3-4f1a-bb6e-83830a2a058d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bfddc61-f2f3-4f1a-bb6e-83830a2a058d-istio-data\") pod \"0bfddc61-f2f3-4f1a-bb6e-83830a2a058d\" (UID: \"0bfddc61-f2f3-4f1a-bb6e-83830a2a058d\") "
Oct 30 13:44:55.007907 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.006793    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/654161bd-9cf8-4f63-9a86-b312763e75cc-istio-token\" (UniqueName: \"kubernetes.io/projected/654161bd-9cf8-4f63-9a86-b312763e75cc-istio-token\") pod \"654161bd-9cf8-4f63-9a86-b312763e75cc\" (UID: \"654161bd-9cf8-4f63-9a86-b312763e75cc\") "
Oct 30 13:44:55.007907 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.006851    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/654161bd-9cf8-4f63-9a86-b312763e75cc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/654161bd-9cf8-4f63-9a86-b312763e75cc-credential-socket\") pod \"654161bd-9cf8-4f63-9a86-b312763e75cc\" (UID: \"654161bd-9cf8-4f63-9a86-b312763e75cc\") "
Oct 30 13:44:55.007907 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.006908    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/654161bd-9cf8-4f63-9a86-b312763e75cc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/654161bd-9cf8-4f63-9a86-b312763e75cc-workload-socket\") pod \"654161bd-9cf8-4f63-9a86-b312763e75cc\" (UID: \"654161bd-9cf8-4f63-9a86-b312763e75cc\") "
Oct 30 13:44:55.007907 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.006938    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bfddc61-f2f3-4f1a-bb6e-83830a2a058d-istio-token\" (UniqueName: \"kubernetes.io/projected/0bfddc61-f2f3-4f1a-bb6e-83830a2a058d-istio-token\") pod \"0bfddc61-f2f3-4f1a-bb6e-83830a2a058d\" (UID: \"0bfddc61-f2f3-4f1a-bb6e-83830a2a058d\") "
Oct 30 13:44:55.007907 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.006991    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/654161bd-9cf8-4f63-9a86-b312763e75cc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/654161bd-9cf8-4f63-9a86-b312763e75cc-istio-data\") pod \"654161bd-9cf8-4f63-9a86-b312763e75cc\" (UID: \"654161bd-9cf8-4f63-9a86-b312763e75cc\") "
Oct 30 13:44:55.007907 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.007026    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bfddc61-f2f3-4f1a-bb6e-83830a2a058d-kube-api-access-7dsn6\" (UniqueName: \"kubernetes.io/projected/0bfddc61-f2f3-4f1a-bb6e-83830a2a058d-kube-api-access-7dsn6\") pod \"0bfddc61-f2f3-4f1a-bb6e-83830a2a058d\" (UID: \"0bfddc61-f2f3-4f1a-bb6e-83830a2a058d\") "
Oct 30 13:44:55.007907 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.007072    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bfddc61-f2f3-4f1a-bb6e-83830a2a058d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bfddc61-f2f3-4f1a-bb6e-83830a2a058d-istio-envoy\") pod \"0bfddc61-f2f3-4f1a-bb6e-83830a2a058d\" (UID: \"0bfddc61-f2f3-4f1a-bb6e-83830a2a058d\") "
Oct 30 13:44:55.007907 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.007131    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/654161bd-9cf8-4f63-9a86-b312763e75cc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/654161bd-9cf8-4f63-9a86-b312763e75cc-istio-envoy\") pod \"654161bd-9cf8-4f63-9a86-b312763e75cc\" (UID: \"654161bd-9cf8-4f63-9a86-b312763e75cc\") "
Oct 30 13:44:55.007907 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.007165    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/654161bd-9cf8-4f63-9a86-b312763e75cc-kube-api-access-8k242\" (UniqueName: \"kubernetes.io/projected/654161bd-9cf8-4f63-9a86-b312763e75cc-kube-api-access-8k242\") pod \"654161bd-9cf8-4f63-9a86-b312763e75cc\" (UID: \"654161bd-9cf8-4f63-9a86-b312763e75cc\") "
Oct 30 13:44:55.007907 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.007466    4364 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bed178a-380c-456c-ac60-ed0020bc32f5-credential-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:55.007907 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.007479    4364 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bed178a-380c-456c-ac60-ed0020bc32f5-istiod-ca-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:55.007907 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.007488    4364 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bed178a-380c-456c-ac60-ed0020bc32f5-workload-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:55.007907 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.007498    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dcwk\" (UniqueName: \"kubernetes.io/projected/9bed178a-380c-456c-ac60-ed0020bc32f5-kube-api-access-6dcwk\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:55.007907 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.007507    4364 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bed178a-380c-456c-ac60-ed0020bc32f5-istio-token\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:55.007907 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.007517    4364 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bed178a-380c-456c-ac60-ed0020bc32f5-workload-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:55.007907 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.007525    4364 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bed178a-380c-456c-ac60-ed0020bc32f5-istio-data\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:55.007907 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.007533    4364 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bed178a-380c-456c-ac60-ed0020bc32f5-istio-podinfo\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:55.007907 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.007541    4364 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bed178a-380c-456c-ac60-ed0020bc32f5-istio-envoy\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:55.009542 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.009109    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/654161bd-9cf8-4f63-9a86-b312763e75cc-istiod-ca-cert" pod "654161bd-9cf8-4f63-9a86-b312763e75cc" (UID: "654161bd-9cf8-4f63-9a86-b312763e75cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:44:55.009542 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.009385    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/654161bd-9cf8-4f63-9a86-b312763e75cc-credential-socket" pod "654161bd-9cf8-4f63-9a86-b312763e75cc" (UID: "654161bd-9cf8-4f63-9a86-b312763e75cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:55.009694 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.009629    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/654161bd-9cf8-4f63-9a86-b312763e75cc-workload-socket" pod "654161bd-9cf8-4f63-9a86-b312763e75cc" (UID: "654161bd-9cf8-4f63-9a86-b312763e75cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:55.010025 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.009960    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/654161bd-9cf8-4f63-9a86-b312763e75cc-workload-certs" pod "654161bd-9cf8-4f63-9a86-b312763e75cc" (UID: "654161bd-9cf8-4f63-9a86-b312763e75cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:55.010317 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.010265    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bfddc61-f2f3-4f1a-bb6e-83830a2a058d-credential-socket" pod "0bfddc61-f2f3-4f1a-bb6e-83830a2a058d" (UID: "0bfddc61-f2f3-4f1a-bb6e-83830a2a058d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:55.010401 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.010335    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/654161bd-9cf8-4f63-9a86-b312763e75cc-istio-data" pod "654161bd-9cf8-4f63-9a86-b312763e75cc" (UID: "654161bd-9cf8-4f63-9a86-b312763e75cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:55.010520 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.010501    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bfddc61-f2f3-4f1a-bb6e-83830a2a058d-workload-socket" pod "0bfddc61-f2f3-4f1a-bb6e-83830a2a058d" (UID: "0bfddc61-f2f3-4f1a-bb6e-83830a2a058d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:55.010709 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.010662    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bfddc61-f2f3-4f1a-bb6e-83830a2a058d-istio-data" pod "0bfddc61-f2f3-4f1a-bb6e-83830a2a058d" (UID: "0bfddc61-f2f3-4f1a-bb6e-83830a2a058d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:55.010851 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.010798    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bfddc61-f2f3-4f1a-bb6e-83830a2a058d-workload-certs" pod "0bfddc61-f2f3-4f1a-bb6e-83830a2a058d" (UID: "0bfddc61-f2f3-4f1a-bb6e-83830a2a058d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:55.011950 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.011925    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bfddc61-f2f3-4f1a-bb6e-83830a2a058d-istiod-ca-cert" pod "0bfddc61-f2f3-4f1a-bb6e-83830a2a058d" (UID: "0bfddc61-f2f3-4f1a-bb6e-83830a2a058d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:44:55.013275 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.013206    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/654161bd-9cf8-4f63-9a86-b312763e75cc-istio-envoy" pod "654161bd-9cf8-4f63-9a86-b312763e75cc" (UID: "654161bd-9cf8-4f63-9a86-b312763e75cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:55.013469 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.013451    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bfddc61-f2f3-4f1a-bb6e-83830a2a058d-kube-api-access-7dsn6" pod "0bfddc61-f2f3-4f1a-bb6e-83830a2a058d" (UID: "0bfddc61-f2f3-4f1a-bb6e-83830a2a058d"). InnerVolumeSpecName "kube-api-access-7dsn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:44:55.013580 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.013470    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/654161bd-9cf8-4f63-9a86-b312763e75cc-istio-podinfo" pod "654161bd-9cf8-4f63-9a86-b312763e75cc" (UID: "654161bd-9cf8-4f63-9a86-b312763e75cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:44:55.013779 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.013757    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/654161bd-9cf8-4f63-9a86-b312763e75cc-kube-api-access-8k242" pod "654161bd-9cf8-4f63-9a86-b312763e75cc" (UID: "654161bd-9cf8-4f63-9a86-b312763e75cc"). InnerVolumeSpecName "kube-api-access-8k242". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:44:55.013941 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.013926    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0bfddc61-f2f3-4f1a-bb6e-83830a2a058d-istio-podinfo" pod "0bfddc61-f2f3-4f1a-bb6e-83830a2a058d" (UID: "0bfddc61-f2f3-4f1a-bb6e-83830a2a058d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:44:55.014936 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.014910    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/654161bd-9cf8-4f63-9a86-b312763e75cc-istio-token" pod "654161bd-9cf8-4f63-9a86-b312763e75cc" (UID: "654161bd-9cf8-4f63-9a86-b312763e75cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:44:55.015022 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.014913    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bfddc61-f2f3-4f1a-bb6e-83830a2a058d-istio-envoy" pod "0bfddc61-f2f3-4f1a-bb6e-83830a2a058d" (UID: "0bfddc61-f2f3-4f1a-bb6e-83830a2a058d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:55.016745 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.016723    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bfddc61-f2f3-4f1a-bb6e-83830a2a058d-istio-token" pod "0bfddc61-f2f3-4f1a-bb6e-83830a2a058d" (UID: "0bfddc61-f2f3-4f1a-bb6e-83830a2a058d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:44:55.109318 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.109270    4364 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/654161bd-9cf8-4f63-9a86-b312763e75cc-workload-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:55.109318 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.109314    4364 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bfddc61-f2f3-4f1a-bb6e-83830a2a058d-credential-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:55.109318 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.109328    4364 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bfddc61-f2f3-4f1a-bb6e-83830a2a058d-workload-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:55.109647 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.109341    4364 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bfddc61-f2f3-4f1a-bb6e-83830a2a058d-workload-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:55.109647 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.109355    4364 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bfddc61-f2f3-4f1a-bb6e-83830a2a058d-istiod-ca-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:55.109647 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.109367    4364 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bfddc61-f2f3-4f1a-bb6e-83830a2a058d-istio-data\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:55.109647 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.109380    4364 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/654161bd-9cf8-4f63-9a86-b312763e75cc-istio-token\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:55.109647 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.109392    4364 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/654161bd-9cf8-4f63-9a86-b312763e75cc-credential-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:55.109647 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.109404    4364 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/654161bd-9cf8-4f63-9a86-b312763e75cc-workload-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:55.109647 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.109417    4364 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bfddc61-f2f3-4f1a-bb6e-83830a2a058d-istio-token\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:55.109647 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.109428    4364 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/654161bd-9cf8-4f63-9a86-b312763e75cc-istio-data\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:55.109647 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.109444    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dsn6\" (UniqueName: \"kubernetes.io/projected/0bfddc61-f2f3-4f1a-bb6e-83830a2a058d-kube-api-access-7dsn6\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:55.109647 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.109456    4364 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bfddc61-f2f3-4f1a-bb6e-83830a2a058d-istio-envoy\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:55.109647 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.109468    4364 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/654161bd-9cf8-4f63-9a86-b312763e75cc-istio-envoy\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:55.109647 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.109481    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8k242\" (UniqueName: \"kubernetes.io/projected/654161bd-9cf8-4f63-9a86-b312763e75cc-kube-api-access-8k242\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:55.109647 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.109493    4364 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bfddc61-f2f3-4f1a-bb6e-83830a2a058d-istio-podinfo\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:55.109647 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.109505    4364 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/654161bd-9cf8-4f63-9a86-b312763e75cc-istiod-ca-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:55.109647 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.109517    4364 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/654161bd-9cf8-4f63-9a86-b312763e75cc-istio-podinfo\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:55.553685 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.553648    4364 generic.go:356] "Generic (PLEG): container finished" podID="654161bd-9cf8-4f63-9a86-b312763e75cc" containerID="1f172dad62aacefaf343b3e23df93faaa6b45b4430a568d93053d8210fb265cc" exitCode=0
Oct 30 13:44:55.554179 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.553714    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942"
Oct 30 13:44:55.554179 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.553730    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942" event={"ID":"654161bd-9cf8-4f63-9a86-b312763e75cc","Type":"ContainerDied","Data":"1f172dad62aacefaf343b3e23df93faaa6b45b4430a568d93053d8210fb265cc"}
Oct 30 13:44:55.554179 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.553783    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942" event={"ID":"654161bd-9cf8-4f63-9a86-b312763e75cc","Type":"ContainerDied","Data":"8a700846e9fc29ee39c5a14736c18535ef62b64609cec205a586f7c4800af125"}
Oct 30 13:44:55.554179 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.553816    4364 scope.go:122] "RemoveContainer" containerID="1f172dad62aacefaf343b3e23df93faaa6b45b4430a568d93053d8210fb265cc"
Oct 30 13:44:55.555343 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.555318    4364 generic.go:356] "Generic (PLEG): container finished" podID="9bed178a-380c-456c-ac60-ed0020bc32f5" containerID="c0eaf1f1ede3e5d2e012b256f720c4b52985b3ffc4058fcb70a0e6ff72e18a3d" exitCode=0
Oct 30 13:44:55.555501 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.555370    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k"
Oct 30 13:44:55.555501 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.555393    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k" event={"ID":"9bed178a-380c-456c-ac60-ed0020bc32f5","Type":"ContainerDied","Data":"c0eaf1f1ede3e5d2e012b256f720c4b52985b3ffc4058fcb70a0e6ff72e18a3d"}
Oct 30 13:44:55.555501 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.555462    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k" event={"ID":"9bed178a-380c-456c-ac60-ed0020bc32f5","Type":"ContainerDied","Data":"7fb6d7d0b112c98a9e1e0b3874207299bdf6e8acd43f232a05e8040fca68d1fd"}
Oct 30 13:44:55.558059 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.558036    4364 generic.go:356] "Generic (PLEG): container finished" podID="0bfddc61-f2f3-4f1a-bb6e-83830a2a058d" containerID="d1085ae44d933ca483e2acb81446c148d704921311f4910228c917ed6dadb2cd" exitCode=0
Oct 30 13:44:55.558185 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.558086    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc" event={"ID":"0bfddc61-f2f3-4f1a-bb6e-83830a2a058d","Type":"ContainerDied","Data":"d1085ae44d933ca483e2acb81446c148d704921311f4910228c917ed6dadb2cd"}
Oct 30 13:44:55.558185 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.558090    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc"
Oct 30 13:44:55.558185 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.558121    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc" event={"ID":"0bfddc61-f2f3-4f1a-bb6e-83830a2a058d","Type":"ContainerDied","Data":"14736826b40baf80e716a12295c0298840fc107b32bab63bc87d1f1fc6d9ee8e"}
Oct 30 13:44:55.565429 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.565411    4364 scope.go:122] "RemoveContainer" containerID="c61d09677f91d362767a518d53c66a58b0fac3495ef391ec0421b63a1c4a6c95"
Oct 30 13:44:55.579821 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.579754    4364 scope.go:122] "RemoveContainer" containerID="1f172dad62aacefaf343b3e23df93faaa6b45b4430a568d93053d8210fb265cc"
Oct 30 13:44:55.580158 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:44:55.580136    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f172dad62aacefaf343b3e23df93faaa6b45b4430a568d93053d8210fb265cc\": container with ID starting with 1f172dad62aacefaf343b3e23df93faaa6b45b4430a568d93053d8210fb265cc not found: ID does not exist" containerID="1f172dad62aacefaf343b3e23df93faaa6b45b4430a568d93053d8210fb265cc"
Oct 30 13:44:55.580219 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.580168    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f172dad62aacefaf343b3e23df93faaa6b45b4430a568d93053d8210fb265cc"} err="failed to get container status \"1f172dad62aacefaf343b3e23df93faaa6b45b4430a568d93053d8210fb265cc\": rpc error: code = NotFound desc = could not find container \"1f172dad62aacefaf343b3e23df93faaa6b45b4430a568d93053d8210fb265cc\": container with ID starting with 1f172dad62aacefaf343b3e23df93faaa6b45b4430a568d93053d8210fb265cc not found: ID does not exist"
Oct 30 13:44:55.580219 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.580190    4364 scope.go:122] "RemoveContainer" containerID="c61d09677f91d362767a518d53c66a58b0fac3495ef391ec0421b63a1c4a6c95"
Oct 30 13:44:55.580469 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:44:55.580436    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c61d09677f91d362767a518d53c66a58b0fac3495ef391ec0421b63a1c4a6c95\": container with ID starting with c61d09677f91d362767a518d53c66a58b0fac3495ef391ec0421b63a1c4a6c95 not found: ID does not exist" containerID="c61d09677f91d362767a518d53c66a58b0fac3495ef391ec0421b63a1c4a6c95"
Oct 30 13:44:55.580579 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.580473    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c61d09677f91d362767a518d53c66a58b0fac3495ef391ec0421b63a1c4a6c95"} err="failed to get container status \"c61d09677f91d362767a518d53c66a58b0fac3495ef391ec0421b63a1c4a6c95\": rpc error: code = NotFound desc = could not find container \"c61d09677f91d362767a518d53c66a58b0fac3495ef391ec0421b63a1c4a6c95\": container with ID starting with c61d09677f91d362767a518d53c66a58b0fac3495ef391ec0421b63a1c4a6c95 not found: ID does not exist"
Oct 30 13:44:55.580579 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.580497    4364 scope.go:122] "RemoveContainer" containerID="c0eaf1f1ede3e5d2e012b256f720c4b52985b3ffc4058fcb70a0e6ff72e18a3d"
Oct 30 13:44:55.591776 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.591744    4364 scope.go:122] "RemoveContainer" containerID="c0eaf1f1ede3e5d2e012b256f720c4b52985b3ffc4058fcb70a0e6ff72e18a3d"
Oct 30 13:44:55.592080 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:44:55.592055    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0eaf1f1ede3e5d2e012b256f720c4b52985b3ffc4058fcb70a0e6ff72e18a3d\": container with ID starting with c0eaf1f1ede3e5d2e012b256f720c4b52985b3ffc4058fcb70a0e6ff72e18a3d not found: ID does not exist" containerID="c0eaf1f1ede3e5d2e012b256f720c4b52985b3ffc4058fcb70a0e6ff72e18a3d"
Oct 30 13:44:55.592154 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.592093    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0eaf1f1ede3e5d2e012b256f720c4b52985b3ffc4058fcb70a0e6ff72e18a3d"} err="failed to get container status \"c0eaf1f1ede3e5d2e012b256f720c4b52985b3ffc4058fcb70a0e6ff72e18a3d\": rpc error: code = NotFound desc = could not find container \"c0eaf1f1ede3e5d2e012b256f720c4b52985b3ffc4058fcb70a0e6ff72e18a3d\": container with ID starting with c0eaf1f1ede3e5d2e012b256f720c4b52985b3ffc4058fcb70a0e6ff72e18a3d not found: ID does not exist"
Oct 30 13:44:55.592154 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.592115    4364 scope.go:122] "RemoveContainer" containerID="d1085ae44d933ca483e2acb81446c148d704921311f4910228c917ed6dadb2cd"
Oct 30 13:44:55.602470 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.602435    4364 scope.go:122] "RemoveContainer" containerID="d1085ae44d933ca483e2acb81446c148d704921311f4910228c917ed6dadb2cd"
Oct 30 13:44:55.602763 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:44:55.602742    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1085ae44d933ca483e2acb81446c148d704921311f4910228c917ed6dadb2cd\": container with ID starting with d1085ae44d933ca483e2acb81446c148d704921311f4910228c917ed6dadb2cd not found: ID does not exist" containerID="d1085ae44d933ca483e2acb81446c148d704921311f4910228c917ed6dadb2cd"
Oct 30 13:44:55.602851 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.602768    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1085ae44d933ca483e2acb81446c148d704921311f4910228c917ed6dadb2cd"} err="failed to get container status \"d1085ae44d933ca483e2acb81446c148d704921311f4910228c917ed6dadb2cd\": rpc error: code = NotFound desc = could not find container \"d1085ae44d933ca483e2acb81446c148d704921311f4910228c917ed6dadb2cd\": container with ID starting with d1085ae44d933ca483e2acb81446c148d704921311f4910228c917ed6dadb2cd not found: ID does not exist"
Oct 30 13:44:55.616239 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.616210    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k"]
Oct 30 13:44:55.630026 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.630005    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-vgq4k"]
Oct 30 13:44:55.661407 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.661379    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc"]
Oct 30 13:44:55.667432 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.667407    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-sdwvc"]
Oct 30 13:44:55.712409 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.712375    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942"]
Oct 30 13:44:55.717779 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:55.717751    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-mk942"]
Oct 30 13:44:56.969733 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:56.969690    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bfddc61-f2f3-4f1a-bb6e-83830a2a058d" path="/var/lib/kubelet/pods/0bfddc61-f2f3-4f1a-bb6e-83830a2a058d/volumes"
Oct 30 13:44:56.970371 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:56.970350    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="654161bd-9cf8-4f63-9a86-b312763e75cc" path="/var/lib/kubelet/pods/654161bd-9cf8-4f63-9a86-b312763e75cc/volumes"
Oct 30 13:44:56.970871 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:56.970850    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bed178a-380c-456c-ac60-ed0020bc32f5" path="/var/lib/kubelet/pods/9bed178a-380c-456c-ac60-ed0020bc32f5/volumes"
Oct 30 13:44:57.759942 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:57.759908    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t"]
Oct 30 13:44:57.760172 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:57.760144    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t" podUID="58073475-3ca6-4a52-b021-e08e935fad18" containerName="istio-proxy" containerID="cri-o://259a6384f1c83dd7ebaa0c73d9c969906b653928ebce6df6b1842fc2ce0338bf" gracePeriod=30
Oct 30 13:44:58.353085 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.353059    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-9c6033-855647cb84-pkjp9"
Oct 30 13:44:58.379279 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.379255    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9c6033-6546465fff-z4rrd"
Oct 30 13:44:58.453456 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.453426    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/362b87d6-a207-44a8-a247-92bc1e05e554-kube-api-access-t84gq\" (UniqueName: \"kubernetes.io/projected/362b87d6-a207-44a8-a247-92bc1e05e554-kube-api-access-t84gq\") pod \"362b87d6-a207-44a8-a247-92bc1e05e554\" (UID: \"362b87d6-a207-44a8-a247-92bc1e05e554\") "
Oct 30 13:44:58.453661 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.453549    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/362b87d6-a207-44a8-a247-92bc1e05e554-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/362b87d6-a207-44a8-a247-92bc1e05e554-plugins-config-volume\") pod \"362b87d6-a207-44a8-a247-92bc1e05e554\" (UID: \"362b87d6-a207-44a8-a247-92bc1e05e554\") "
Oct 30 13:44:58.453661 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.453643    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a5999a2-3c46-4554-8b5b-0e1ccf464ec8-kube-api-access-g5hwn\" (UniqueName: \"kubernetes.io/projected/3a5999a2-3c46-4554-8b5b-0e1ccf464ec8-kube-api-access-g5hwn\") pod \"3a5999a2-3c46-4554-8b5b-0e1ccf464ec8\" (UID: \"3a5999a2-3c46-4554-8b5b-0e1ccf464ec8\") "
Oct 30 13:44:58.453811 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.453694    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a5999a2-3c46-4554-8b5b-0e1ccf464ec8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a5999a2-3c46-4554-8b5b-0e1ccf464ec8-plugins-config-volume\") pod \"3a5999a2-3c46-4554-8b5b-0e1ccf464ec8\" (UID: \"3a5999a2-3c46-4554-8b5b-0e1ccf464ec8\") "
Oct 30 13:44:58.453982 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.453961    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/362b87d6-a207-44a8-a247-92bc1e05e554-plugins-config-volume" pod "362b87d6-a207-44a8-a247-92bc1e05e554" (UID: "362b87d6-a207-44a8-a247-92bc1e05e554"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:44:58.454312 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.454278    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a5999a2-3c46-4554-8b5b-0e1ccf464ec8-plugins-config-volume" pod "3a5999a2-3c46-4554-8b5b-0e1ccf464ec8" (UID: "3a5999a2-3c46-4554-8b5b-0e1ccf464ec8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:44:58.454405 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.454323    4364 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/362b87d6-a207-44a8-a247-92bc1e05e554-plugins-config-volume\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:58.455816 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.455789    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a5999a2-3c46-4554-8b5b-0e1ccf464ec8-kube-api-access-g5hwn" pod "3a5999a2-3c46-4554-8b5b-0e1ccf464ec8" (UID: "3a5999a2-3c46-4554-8b5b-0e1ccf464ec8"). InnerVolumeSpecName "kube-api-access-g5hwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:44:58.456198 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.456169    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/362b87d6-a207-44a8-a247-92bc1e05e554-kube-api-access-t84gq" pod "362b87d6-a207-44a8-a247-92bc1e05e554" (UID: "362b87d6-a207-44a8-a247-92bc1e05e554"). InnerVolumeSpecName "kube-api-access-t84gq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:44:58.555694 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.555587    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t84gq\" (UniqueName: \"kubernetes.io/projected/362b87d6-a207-44a8-a247-92bc1e05e554-kube-api-access-t84gq\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:58.555694 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.555644    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5hwn\" (UniqueName: \"kubernetes.io/projected/3a5999a2-3c46-4554-8b5b-0e1ccf464ec8-kube-api-access-g5hwn\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:58.555694 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.555656    4364 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a5999a2-3c46-4554-8b5b-0e1ccf464ec8-plugins-config-volume\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:44:58.579768 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.579737    4364 generic.go:356] "Generic (PLEG): container finished" podID="3a5999a2-3c46-4554-8b5b-0e1ccf464ec8" containerID="2bf3c9192711fe8aec4966e4f745f5294735a4991f5bd8529fdbf4cd36ab7c95" exitCode=0
Oct 30 13:44:58.579928 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.579789    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-9c6033-855647cb84-pkjp9"
Oct 30 13:44:58.579928 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.579817    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-9c6033-855647cb84-pkjp9" event={"ID":"3a5999a2-3c46-4554-8b5b-0e1ccf464ec8","Type":"ContainerDied","Data":"2bf3c9192711fe8aec4966e4f745f5294735a4991f5bd8529fdbf4cd36ab7c95"}
Oct 30 13:44:58.579928 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.579870    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-9c6033-855647cb84-pkjp9" event={"ID":"3a5999a2-3c46-4554-8b5b-0e1ccf464ec8","Type":"ContainerDied","Data":"709e0bb87f162f72bb0f96e13d426cda3811ddd64f58d1c072c445efeb57d924"}
Oct 30 13:44:58.579928 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.579896    4364 scope.go:122] "RemoveContainer" containerID="2bf3c9192711fe8aec4966e4f745f5294735a4991f5bd8529fdbf4cd36ab7c95"
Oct 30 13:44:58.581361 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.581332    4364 generic.go:356] "Generic (PLEG): container finished" podID="362b87d6-a207-44a8-a247-92bc1e05e554" containerID="56faa51cac10db60007730500063d06a536488b96e31d9fd0ee541fc9a53519e" exitCode=0
Oct 30 13:44:58.581489 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.581391    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9c6033-6546465fff-z4rrd"
Oct 30 13:44:58.581489 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.581406    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9c6033-6546465fff-z4rrd" event={"ID":"362b87d6-a207-44a8-a247-92bc1e05e554","Type":"ContainerDied","Data":"56faa51cac10db60007730500063d06a536488b96e31d9fd0ee541fc9a53519e"}
Oct 30 13:44:58.581489 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.581450    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9c6033-6546465fff-z4rrd" event={"ID":"362b87d6-a207-44a8-a247-92bc1e05e554","Type":"ContainerDied","Data":"b7fd3694ef72c53853735edcae204044a20448ac78b52e6f2b168d59479a3c94"}
Oct 30 13:44:58.591311 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.591292    4364 scope.go:122] "RemoveContainer" containerID="2bf3c9192711fe8aec4966e4f745f5294735a4991f5bd8529fdbf4cd36ab7c95"
Oct 30 13:44:58.591563 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:44:58.591543    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bf3c9192711fe8aec4966e4f745f5294735a4991f5bd8529fdbf4cd36ab7c95\": container with ID starting with 2bf3c9192711fe8aec4966e4f745f5294735a4991f5bd8529fdbf4cd36ab7c95 not found: ID does not exist" containerID="2bf3c9192711fe8aec4966e4f745f5294735a4991f5bd8529fdbf4cd36ab7c95"
Oct 30 13:44:58.591771 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.591572    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bf3c9192711fe8aec4966e4f745f5294735a4991f5bd8529fdbf4cd36ab7c95"} err="failed to get container status \"2bf3c9192711fe8aec4966e4f745f5294735a4991f5bd8529fdbf4cd36ab7c95\": rpc error: code = NotFound desc = could not find container \"2bf3c9192711fe8aec4966e4f745f5294735a4991f5bd8529fdbf4cd36ab7c95\": container with ID starting with 2bf3c9192711fe8aec4966e4f745f5294735a4991f5bd8529fdbf4cd36ab7c95 not found: ID does not exist"
Oct 30 13:44:58.591771 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.591589    4364 scope.go:122] "RemoveContainer" containerID="56faa51cac10db60007730500063d06a536488b96e31d9fd0ee541fc9a53519e"
Oct 30 13:44:58.600361 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.600342    4364 scope.go:122] "RemoveContainer" containerID="56faa51cac10db60007730500063d06a536488b96e31d9fd0ee541fc9a53519e"
Oct 30 13:44:58.600702 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:44:58.600586    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56faa51cac10db60007730500063d06a536488b96e31d9fd0ee541fc9a53519e\": container with ID starting with 56faa51cac10db60007730500063d06a536488b96e31d9fd0ee541fc9a53519e not found: ID does not exist" containerID="56faa51cac10db60007730500063d06a536488b96e31d9fd0ee541fc9a53519e"
Oct 30 13:44:58.600760 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.600710    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56faa51cac10db60007730500063d06a536488b96e31d9fd0ee541fc9a53519e"} err="failed to get container status \"56faa51cac10db60007730500063d06a536488b96e31d9fd0ee541fc9a53519e\": rpc error: code = NotFound desc = could not find container \"56faa51cac10db60007730500063d06a536488b96e31d9fd0ee541fc9a53519e\": container with ID starting with 56faa51cac10db60007730500063d06a536488b96e31d9fd0ee541fc9a53519e not found: ID does not exist"
Oct 30 13:44:58.645510 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.645479    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9c6033-6546465fff-z4rrd"]
Oct 30 13:44:58.654047 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.654016    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9c6033-6546465fff-z4rrd"]
Oct 30 13:44:58.697756 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.697723    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-9c6033-855647cb84-pkjp9"]
Oct 30 13:44:58.701429 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.701399    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-9c6033-855647cb84-pkjp9"]
Oct 30 13:44:58.842623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.842521    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f86e315-sxk99"]
Oct 30 13:44:58.843106 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.843090    4364 cpu_manager.go:414] "containerMap: removing container" podUID="f1ebdbad-f9d4-4758-a7a8-e2513b6d837d" containerName="revoke-keys"
Oct 30 13:44:58.843106 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.843106    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1ebdbad-f9d4-4758-a7a8-e2513b6d837d" containerName="revoke-keys"
Oct 30 13:44:58.843225 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.843124    4364 cpu_manager.go:414] "containerMap: removing container" podUID="5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b" containerName="istio-proxy"
Oct 30 13:44:58.843225 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.843129    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b" containerName="istio-proxy"
Oct 30 13:44:58.843225 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.843142    4364 cpu_manager.go:414] "containerMap: removing container" podUID="d675cfa9-7df1-446a-984d-a4e1b0dda352" containerName="istio-proxy"
Oct 30 13:44:58.843225 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.843147    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d675cfa9-7df1-446a-984d-a4e1b0dda352" containerName="istio-proxy"
Oct 30 13:44:58.843225 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.843154    4364 cpu_manager.go:414] "containerMap: removing container" podUID="3a5999a2-3c46-4554-8b5b-0e1ccf464ec8" containerName="payload-processing"
Oct 30 13:44:58.843225 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.843159    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a5999a2-3c46-4554-8b5b-0e1ccf464ec8" containerName="payload-processing"
Oct 30 13:44:58.843225 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.843173    4364 cpu_manager.go:414] "containerMap: removing container" podUID="0bfddc61-f2f3-4f1a-bb6e-83830a2a058d" containerName="istio-proxy"
Oct 30 13:44:58.843225 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.843178    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bfddc61-f2f3-4f1a-bb6e-83830a2a058d" containerName="istio-proxy"
Oct 30 13:44:58.843225 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.843186    4364 cpu_manager.go:414] "containerMap: removing container" podUID="5ce05a81-71f3-4dab-96e1-44c9ea628a7b" containerName="istio-proxy"
Oct 30 13:44:58.843225 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.843191    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ce05a81-71f3-4dab-96e1-44c9ea628a7b" containerName="istio-proxy"
Oct 30 13:44:58.843225 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.843198    4364 cpu_manager.go:414] "containerMap: removing container" podUID="9a967f02-7c93-484e-9bd4-97e1a4f5fdf5" containerName="istio-proxy"
Oct 30 13:44:58.843225 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.843203    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a967f02-7c93-484e-9bd4-97e1a4f5fdf5" containerName="istio-proxy"
Oct 30 13:44:58.843225 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.843214    4364 cpu_manager.go:414] "containerMap: removing container" podUID="5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b" containerName="istio-proxy"
Oct 30 13:44:58.843225 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.843219    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b" containerName="istio-proxy"
Oct 30 13:44:58.843225 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.843227    4364 cpu_manager.go:414] "containerMap: removing container" podUID="9bed178a-380c-456c-ac60-ed0020bc32f5" containerName="istio-proxy"
Oct 30 13:44:58.843225 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.843232    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bed178a-380c-456c-ac60-ed0020bc32f5" containerName="istio-proxy"
Oct 30 13:44:58.843740 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.843238    4364 cpu_manager.go:414] "containerMap: removing container" podUID="654161bd-9cf8-4f63-9a86-b312763e75cc" containerName="istio-proxy"
Oct 30 13:44:58.843740 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.843243    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="654161bd-9cf8-4f63-9a86-b312763e75cc" containerName="istio-proxy"
Oct 30 13:44:58.843740 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.843249    4364 cpu_manager.go:414] "containerMap: removing container" podUID="d675cfa9-7df1-446a-984d-a4e1b0dda352" containerName="istio-proxy"
Oct 30 13:44:58.843740 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.843253    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d675cfa9-7df1-446a-984d-a4e1b0dda352" containerName="istio-proxy"
Oct 30 13:44:58.843740 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.843261    4364 cpu_manager.go:414] "containerMap: removing container" podUID="362b87d6-a207-44a8-a247-92bc1e05e554" containerName="payload-pre-processing"
Oct 30 13:44:58.843740 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.843265    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="362b87d6-a207-44a8-a247-92bc1e05e554" containerName="payload-pre-processing"
Oct 30 13:44:58.843740 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.843274    4364 cpu_manager.go:414] "containerMap: removing container" podUID="654161bd-9cf8-4f63-9a86-b312763e75cc" containerName="istio-proxy"
Oct 30 13:44:58.843740 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.843278    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="654161bd-9cf8-4f63-9a86-b312763e75cc" containerName="istio-proxy"
Oct 30 13:44:58.843740 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.843339    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bfddc61-f2f3-4f1a-bb6e-83830a2a058d" containerName="istio-proxy"
Oct 30 13:44:58.843740 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.843348    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b" containerName="istio-proxy"
Oct 30 13:44:58.843740 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.843354    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bed178a-380c-456c-ac60-ed0020bc32f5" containerName="istio-proxy"
Oct 30 13:44:58.843740 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.843360    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="654161bd-9cf8-4f63-9a86-b312763e75cc" containerName="istio-proxy"
Oct 30 13:44:58.843740 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.843369    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="d675cfa9-7df1-446a-984d-a4e1b0dda352" containerName="istio-proxy"
Oct 30 13:44:58.843740 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.843377    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="362b87d6-a207-44a8-a247-92bc1e05e554" containerName="payload-pre-processing"
Oct 30 13:44:58.843740 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.843383    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="654161bd-9cf8-4f63-9a86-b312763e75cc" containerName="istio-proxy"
Oct 30 13:44:58.843740 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.843388    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1ebdbad-f9d4-4758-a7a8-e2513b6d837d" containerName="revoke-keys"
Oct 30 13:44:58.843740 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.843394    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e958d3b-8a4e-41d2-b7ad-fd60d1fc9e9b" containerName="istio-proxy"
Oct 30 13:44:58.843740 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.843400    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a5999a2-3c46-4554-8b5b-0e1ccf464ec8" containerName="payload-processing"
Oct 30 13:44:58.848269 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.848248    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f86e315-sxk99"
Oct 30 13:44:58.870037 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.870011    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f86e315-sxk99"]
Oct 30 13:44:58.885158 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.885137    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:44:58.885283 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.885268    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\""
Oct 30 13:44:58.960046 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.960020    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dead7294-e167-41ef-937e-21f038062b38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f86e315-sxk99\" (UID: \"dead7294-e167-41ef-937e-21f038062b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f86e315-sxk99"
Oct 30 13:44:58.960201 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.960078    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czn8j\" (UniqueName: \"kubernetes.io/projected/dead7294-e167-41ef-937e-21f038062b38-kube-api-access-czn8j\") pod \"maas-api-revoke-sub-2f86e315-sxk99\" (UID: \"dead7294-e167-41ef-937e-21f038062b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f86e315-sxk99"
Oct 30 13:44:58.960201 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.960115    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dead7294-e167-41ef-937e-21f038062b38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f86e315-sxk99\" (UID: \"dead7294-e167-41ef-937e-21f038062b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f86e315-sxk99"
Oct 30 13:44:58.970766 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.970740    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="362b87d6-a207-44a8-a247-92bc1e05e554" path="/var/lib/kubelet/pods/362b87d6-a207-44a8-a247-92bc1e05e554/volumes"
Oct 30 13:44:58.971119 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:58.971107    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a5999a2-3c46-4554-8b5b-0e1ccf464ec8" path="/var/lib/kubelet/pods/3a5999a2-3c46-4554-8b5b-0e1ccf464ec8/volumes"
Oct 30 13:44:59.061149 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:59.061123    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dead7294-e167-41ef-937e-21f038062b38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f86e315-sxk99\" (UID: \"dead7294-e167-41ef-937e-21f038062b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f86e315-sxk99"
Oct 30 13:44:59.061275 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:59.061256    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czn8j\" (UniqueName: \"kubernetes.io/projected/dead7294-e167-41ef-937e-21f038062b38-kube-api-access-czn8j\") pod \"maas-api-revoke-sub-2f86e315-sxk99\" (UID: \"dead7294-e167-41ef-937e-21f038062b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f86e315-sxk99"
Oct 30 13:44:59.061327 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:59.061289    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dead7294-e167-41ef-937e-21f038062b38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f86e315-sxk99\" (UID: \"dead7294-e167-41ef-937e-21f038062b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f86e315-sxk99"
Oct 30 13:44:59.061922 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:59.061899    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dead7294-e167-41ef-937e-21f038062b38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f86e315-sxk99\" (UID: \"dead7294-e167-41ef-937e-21f038062b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f86e315-sxk99"
Oct 30 13:44:59.063342 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:59.063310    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dead7294-e167-41ef-937e-21f038062b38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f86e315-sxk99\" (UID: \"dead7294-e167-41ef-937e-21f038062b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f86e315-sxk99"
Oct 30 13:44:59.087096 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:59.087070    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czn8j\" (UniqueName: \"kubernetes.io/projected/dead7294-e167-41ef-937e-21f038062b38-kube-api-access-czn8j\") pod \"maas-api-revoke-sub-2f86e315-sxk99\" (UID: \"dead7294-e167-41ef-937e-21f038062b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f86e315-sxk99"
Oct 30 13:44:59.161633 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:59.161591    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f86e315-sxk99"
Oct 30 13:44:59.489174 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:59.489141    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f86e315-sxk99"]
Oct 30 13:44:59.489564 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:44:59.489383    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddead7294_e167_41ef_937e_21f038062b38.slice/crio-c9fbe34075def3030505e3f093b10a141daf4d4c5aa93a53e479138b98c88dcb WatchSource:0}: Error finding container c9fbe34075def3030505e3f093b10a141daf4d4c5aa93a53e479138b98c88dcb: Status 404 returned error can't find the container with id c9fbe34075def3030505e3f093b10a141daf4d4c5aa93a53e479138b98c88dcb
Oct 30 13:44:59.588626 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:59.588585    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f86e315-sxk99" event={"ID":"dead7294-e167-41ef-937e-21f038062b38","Type":"ContainerStarted","Data":"c9fbe34075def3030505e3f093b10a141daf4d4c5aa93a53e479138b98c88dcb"}
Oct 30 13:44:59.784460 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:44:59.784421    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-cda0d9-74764bd4cf-l64mp" podUID="bd1b7e78-a663-4d45-a742-5994a0259a24" containerName="payload-pre-processing" containerID="cri-o://e7b248d8c1f66460e433ea637fdb1a9e085e5d7c8ed4532d5cf5c6b9603a9ce7" gracePeriod=25
Oct 30 13:45:00.154653 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:00.154621    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889465-r87lk"]
Oct 30 13:45:00.155175 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:00.155161    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="d675cfa9-7df1-446a-984d-a4e1b0dda352" containerName="istio-proxy"
Oct 30 13:45:00.158484 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:00.158457    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-r87lk"
Oct 30 13:45:00.161522 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:00.161499    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889465-26rfr"]
Oct 30 13:45:00.165089 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:00.165066    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889465-wdgj9"]
Oct 30 13:45:00.165199 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:00.165176    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889465-26rfr"
Oct 30 13:45:00.168678 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:00.168661    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889465-wdgj9"
Oct 30 13:45:00.174981 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:00.174954    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-658gp\" (UniqueName: \"kubernetes.io/projected/aaf4329d-4d56-4ce3-8eea-dbaf8da1c07a-kube-api-access-658gp\") pod \"maas-api-key-cleanup-29889465-r87lk\" (UID: \"aaf4329d-4d56-4ce3-8eea-dbaf8da1c07a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-r87lk"
Oct 30 13:45:00.175163 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:00.175142    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaf4329d-4d56-4ce3-8eea-dbaf8da1c07a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889465-r87lk\" (UID: \"aaf4329d-4d56-4ce3-8eea-dbaf8da1c07a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-r87lk"
Oct 30 13:45:00.182920 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:00.182897    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889465-r87lk"]
Oct 30 13:45:00.205534 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:00.205464    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889465-wdgj9"]
Oct 30 13:45:00.208253 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:00.208228    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889465-26rfr"]
Oct 30 13:45:00.276237 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:00.276205    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-658gp\" (UniqueName: \"kubernetes.io/projected/aaf4329d-4d56-4ce3-8eea-dbaf8da1c07a-kube-api-access-658gp\") pod \"maas-api-key-cleanup-29889465-r87lk\" (UID: \"aaf4329d-4d56-4ce3-8eea-dbaf8da1c07a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-r87lk"
Oct 30 13:45:00.276393 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:00.276261    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1453dcde-0c6d-427d-890b-8f3542a0775a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889465-26rfr\" (UID: \"1453dcde-0c6d-427d-890b-8f3542a0775a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889465-26rfr"
Oct 30 13:45:00.276393 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:00.276341    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7pw6\" (UniqueName: \"kubernetes.io/projected/c0628c77-ff94-4a69-a0a0-af20a2b64e13-kube-api-access-x7pw6\") pod \"maas-api-key-cleanup-e2e-worker-w1-52a649-29889465-wdgj9\" (UID: \"c0628c77-ff94-4a69-a0a0-af20a2b64e13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889465-wdgj9"
Oct 30 13:45:00.276491 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:00.276439    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaf4329d-4d56-4ce3-8eea-dbaf8da1c07a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889465-r87lk\" (UID: \"aaf4329d-4d56-4ce3-8eea-dbaf8da1c07a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-r87lk"
Oct 30 13:45:00.276772 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:00.276521    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0628c77-ff94-4a69-a0a0-af20a2b64e13-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-52a649-29889465-wdgj9\" (UID: \"c0628c77-ff94-4a69-a0a0-af20a2b64e13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889465-wdgj9"
Oct 30 13:45:00.285935 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:00.277054    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdvgr\" (UniqueName: \"kubernetes.io/projected/1453dcde-0c6d-427d-890b-8f3542a0775a-kube-api-access-tdvgr\") pod \"maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889465-26rfr\" (UID: \"1453dcde-0c6d-427d-890b-8f3542a0775a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889465-26rfr"
Oct 30 13:45:00.285935 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:00.277454    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaf4329d-4d56-4ce3-8eea-dbaf8da1c07a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889465-r87lk\" (UID: \"aaf4329d-4d56-4ce3-8eea-dbaf8da1c07a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-r87lk"
Oct 30 13:45:00.308578 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:00.308548    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-658gp\" (UniqueName: \"kubernetes.io/projected/aaf4329d-4d56-4ce3-8eea-dbaf8da1c07a-kube-api-access-658gp\") pod \"maas-api-key-cleanup-29889465-r87lk\" (UID: \"aaf4329d-4d56-4ce3-8eea-dbaf8da1c07a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-r87lk"
Oct 30 13:45:00.377934 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:00.377904    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0628c77-ff94-4a69-a0a0-af20a2b64e13-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-52a649-29889465-wdgj9\" (UID: \"c0628c77-ff94-4a69-a0a0-af20a2b64e13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889465-wdgj9"
Oct 30 13:45:00.378098 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:00.377969    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdvgr\" (UniqueName: \"kubernetes.io/projected/1453dcde-0c6d-427d-890b-8f3542a0775a-kube-api-access-tdvgr\") pod \"maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889465-26rfr\" (UID: \"1453dcde-0c6d-427d-890b-8f3542a0775a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889465-26rfr"
Oct 30 13:45:00.378162 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:00.378127    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1453dcde-0c6d-427d-890b-8f3542a0775a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889465-26rfr\" (UID: \"1453dcde-0c6d-427d-890b-8f3542a0775a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889465-26rfr"
Oct 30 13:45:00.378239 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:00.378212    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7pw6\" (UniqueName: \"kubernetes.io/projected/c0628c77-ff94-4a69-a0a0-af20a2b64e13-kube-api-access-x7pw6\") pod \"maas-api-key-cleanup-e2e-worker-w1-52a649-29889465-wdgj9\" (UID: \"c0628c77-ff94-4a69-a0a0-af20a2b64e13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889465-wdgj9"
Oct 30 13:45:00.378626 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:00.378581    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0628c77-ff94-4a69-a0a0-af20a2b64e13-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-52a649-29889465-wdgj9\" (UID: \"c0628c77-ff94-4a69-a0a0-af20a2b64e13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889465-wdgj9"
Oct 30 13:45:00.378782 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:00.378759    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1453dcde-0c6d-427d-890b-8f3542a0775a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889465-26rfr\" (UID: \"1453dcde-0c6d-427d-890b-8f3542a0775a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889465-26rfr"
Oct 30 13:45:00.403493 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:00.403465    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7pw6\" (UniqueName: \"kubernetes.io/projected/c0628c77-ff94-4a69-a0a0-af20a2b64e13-kube-api-access-x7pw6\") pod \"maas-api-key-cleanup-e2e-worker-w1-52a649-29889465-wdgj9\" (UID: \"c0628c77-ff94-4a69-a0a0-af20a2b64e13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889465-wdgj9"
Oct 30 13:45:00.403624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:00.403564    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdvgr\" (UniqueName: \"kubernetes.io/projected/1453dcde-0c6d-427d-890b-8f3542a0775a-kube-api-access-tdvgr\") pod \"maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889465-26rfr\" (UID: \"1453dcde-0c6d-427d-890b-8f3542a0775a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889465-26rfr"
Oct 30 13:45:00.474666 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:00.474566    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-r87lk"
Oct 30 13:45:00.485044 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:00.485022    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889465-26rfr"
Oct 30 13:45:00.493924 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:00.493902    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889465-wdgj9"
Oct 30 13:45:00.575426 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:00.575393    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-52a649-65c5fbbff9-n6rqr"]
Oct 30 13:45:00.599838 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:00.599801    4364 generic.go:356] "Generic (PLEG): container finished" podID="dead7294-e167-41ef-937e-21f038062b38" containerID="240741113a5723174ed33765ab2a09e654b914880a7a568895a5227855d95d10" exitCode=0
Oct 30 13:45:00.600037 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:00.600016    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f86e315-sxk99" event={"ID":"dead7294-e167-41ef-937e-21f038062b38","Type":"ContainerDied","Data":"240741113a5723174ed33765ab2a09e654b914880a7a568895a5227855d95d10"}
Oct 30 13:45:00.799161 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:00.795007    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889465-r87lk"]
Oct 30 13:45:00.802641 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:45:00.802472    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaf4329d_4d56_4ce3_8eea_dbaf8da1c07a.slice/crio-07fa84aedb853b5a78567795a682098ecf2f1c7b2acf346fea4e59e102f1de85 WatchSource:0}: Error finding container 07fa84aedb853b5a78567795a682098ecf2f1c7b2acf346fea4e59e102f1de85: Status 404 returned error can't find the container with id 07fa84aedb853b5a78567795a682098ecf2f1c7b2acf346fea4e59e102f1de85
Oct 30 13:45:00.820632 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:00.820580    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889465-26rfr"]
Oct 30 13:45:00.822040 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:45:00.822014    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1453dcde_0c6d_427d_890b_8f3542a0775a.slice/crio-d748ee5fee9e4ef35f9d07302bdc5e7aed8b6e680a31b56f7c99d65c0eacc95a WatchSource:0}: Error finding container d748ee5fee9e4ef35f9d07302bdc5e7aed8b6e680a31b56f7c99d65c0eacc95a: Status 404 returned error can't find the container with id d748ee5fee9e4ef35f9d07302bdc5e7aed8b6e680a31b56f7c99d65c0eacc95a
Oct 30 13:45:00.832966 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:00.832940    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889465-wdgj9"]
Oct 30 13:45:00.839731 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:45:00.839710    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0628c77_ff94_4a69_a0a0_af20a2b64e13.slice/crio-12fcc8b478fc4d25d4643595894f44da88fe428f25cecafe3acb9226ee5a688a WatchSource:0}: Error finding container 12fcc8b478fc4d25d4643595894f44da88fe428f25cecafe3acb9226ee5a688a: Status 404 returned error can't find the container with id 12fcc8b478fc4d25d4643595894f44da88fe428f25cecafe3acb9226ee5a688a
Oct 30 13:45:00.888297 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:00.888265    4364 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:45:00.888402 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:00.888320    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t" podUID="58073475-3ca6-4a52-b021-e08e935fad18" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:45:01.607001 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:01.606969    4364 generic.go:356] "Generic (PLEG): container finished" podID="1453dcde-0c6d-427d-890b-8f3542a0775a" containerID="f9c7e5c897d8c142131a07db7157b759fb8aedf5ed8de757b907f84d8c72bab2" exitCode=60
Oct 30 13:45:01.607340 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:01.607046    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889465-26rfr" event={"ID":"1453dcde-0c6d-427d-890b-8f3542a0775a","Type":"ContainerDied","Data":"f9c7e5c897d8c142131a07db7157b759fb8aedf5ed8de757b907f84d8c72bab2"}
Oct 30 13:45:01.607340 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:01.607103    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889465-26rfr" event={"ID":"1453dcde-0c6d-427d-890b-8f3542a0775a","Type":"ContainerStarted","Data":"d748ee5fee9e4ef35f9d07302bdc5e7aed8b6e680a31b56f7c99d65c0eacc95a"}
Oct 30 13:45:01.607468 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:01.607349    4364 scope.go:122] "RemoveContainer" containerID="f9c7e5c897d8c142131a07db7157b759fb8aedf5ed8de757b907f84d8c72bab2"
Oct 30 13:45:01.608762 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:01.608736    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-r87lk" event={"ID":"aaf4329d-4d56-4ce3-8eea-dbaf8da1c07a","Type":"ContainerStarted","Data":"4d4e22eea6b2826e8b216093fe38b5a6c90bf9a1b74fb7c9bf6a5fa0ffce21a4"}
Oct 30 13:45:01.608867 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:01.608789    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-r87lk" event={"ID":"aaf4329d-4d56-4ce3-8eea-dbaf8da1c07a","Type":"ContainerStarted","Data":"07fa84aedb853b5a78567795a682098ecf2f1c7b2acf346fea4e59e102f1de85"}
Oct 30 13:45:01.610444 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:01.610424    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889465-wdgj9" event={"ID":"c0628c77-ff94-4a69-a0a0-af20a2b64e13","Type":"ContainerStarted","Data":"3310c37f36330a6457b40ad08bf9b6f43fc5feeabcd4e3d12e04343f34bd3b2c"}
Oct 30 13:45:01.610504 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:01.610460    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889465-wdgj9" event={"ID":"c0628c77-ff94-4a69-a0a0-af20a2b64e13","Type":"ContainerStarted","Data":"12fcc8b478fc4d25d4643595894f44da88fe428f25cecafe3acb9226ee5a688a"}
Oct 30 13:45:01.706293 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:01.706236    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-r87lk" podStartSLOduration=1.706193227 podStartE2EDuration="1.706193227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:01.704889372 +0000 UTC m=+3200.871254021" watchObservedRunningTime="2026-10-30 13:45:01.706193227 +0000 UTC m=+3200.872557875"
Oct 30 13:45:01.753937 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:01.753877    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889465-wdgj9" podStartSLOduration=1.75385798 podStartE2EDuration="1.75385798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:01.750177574 +0000 UTC m=+3200.916542223" watchObservedRunningTime="2026-10-30 13:45:01.75385798 +0000 UTC m=+3200.920222633"
Oct 30 13:45:01.916188 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:01.916162    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f86e315-sxk99"
Oct 30 13:45:02.007093 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:02.007065    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dead7294-e167-41ef-937e-21f038062b38-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dead7294-e167-41ef-937e-21f038062b38-maas-api-xks-ca\") pod \"dead7294-e167-41ef-937e-21f038062b38\" (UID: \"dead7294-e167-41ef-937e-21f038062b38\") "
Oct 30 13:45:02.007252 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:02.007201    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dead7294-e167-41ef-937e-21f038062b38-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dead7294-e167-41ef-937e-21f038062b38-maas-api-openshift-ca\") pod \"dead7294-e167-41ef-937e-21f038062b38\" (UID: \"dead7294-e167-41ef-937e-21f038062b38\") "
Oct 30 13:45:02.007328 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:02.007261    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dead7294-e167-41ef-937e-21f038062b38-kube-api-access-czn8j\" (UniqueName: \"kubernetes.io/projected/dead7294-e167-41ef-937e-21f038062b38-kube-api-access-czn8j\") pod \"dead7294-e167-41ef-937e-21f038062b38\" (UID: \"dead7294-e167-41ef-937e-21f038062b38\") "
Oct 30 13:45:02.007560 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:02.007539    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dead7294-e167-41ef-937e-21f038062b38-maas-api-openshift-ca" pod "dead7294-e167-41ef-937e-21f038062b38" (UID: "dead7294-e167-41ef-937e-21f038062b38"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:02.007795 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:02.007670    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dead7294-e167-41ef-937e-21f038062b38-maas-api-openshift-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:02.009104 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:02.009077    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dead7294-e167-41ef-937e-21f038062b38-maas-api-xks-ca" pod "dead7294-e167-41ef-937e-21f038062b38" (UID: "dead7294-e167-41ef-937e-21f038062b38"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:45:02.009425 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:02.009408    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dead7294-e167-41ef-937e-21f038062b38-kube-api-access-czn8j" pod "dead7294-e167-41ef-937e-21f038062b38" (UID: "dead7294-e167-41ef-937e-21f038062b38"). InnerVolumeSpecName "kube-api-access-czn8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:02.109234 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:02.109174    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czn8j\" (UniqueName: \"kubernetes.io/projected/dead7294-e167-41ef-937e-21f038062b38-kube-api-access-czn8j\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:02.109234 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:02.109198    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dead7294-e167-41ef-937e-21f038062b38-maas-api-xks-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:02.618589 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:02.618556    4364 generic.go:356] "Generic (PLEG): container finished" podID="1453dcde-0c6d-427d-890b-8f3542a0775a" containerID="0ef0bf7a4b5435d8c32063cc4edaebd85964bce0554515dc5d15ac68f65ad06c" exitCode=60
Oct 30 13:45:02.619045 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:02.618641    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889465-26rfr" event={"ID":"1453dcde-0c6d-427d-890b-8f3542a0775a","Type":"ContainerDied","Data":"0ef0bf7a4b5435d8c32063cc4edaebd85964bce0554515dc5d15ac68f65ad06c"}
Oct 30 13:45:02.619045 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:02.618709    4364 scope.go:122] "RemoveContainer" containerID="f9c7e5c897d8c142131a07db7157b759fb8aedf5ed8de757b907f84d8c72bab2"
Oct 30 13:45:02.619045 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:02.618952    4364 scope.go:122] "RemoveContainer" containerID="0ef0bf7a4b5435d8c32063cc4edaebd85964bce0554515dc5d15ac68f65ad06c"
Oct 30 13:45:02.619318 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:45:02.619254    4364 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889465-26rfr_odh-ai-gateway-infra(1453dcde-0c6d-427d-890b-8f3542a0775a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889465-26rfr" podUID="1453dcde-0c6d-427d-890b-8f3542a0775a"
Oct 30 13:45:02.620500 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:02.620475    4364 generic.go:356] "Generic (PLEG): container finished" podID="aaf4329d-4d56-4ce3-8eea-dbaf8da1c07a" containerID="4d4e22eea6b2826e8b216093fe38b5a6c90bf9a1b74fb7c9bf6a5fa0ffce21a4" exitCode=60
Oct 30 13:45:02.620650 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:02.620562    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-r87lk" event={"ID":"aaf4329d-4d56-4ce3-8eea-dbaf8da1c07a","Type":"ContainerDied","Data":"4d4e22eea6b2826e8b216093fe38b5a6c90bf9a1b74fb7c9bf6a5fa0ffce21a4"}
Oct 30 13:45:02.620821 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:02.620807    4364 scope.go:122] "RemoveContainer" containerID="4d4e22eea6b2826e8b216093fe38b5a6c90bf9a1b74fb7c9bf6a5fa0ffce21a4"
Oct 30 13:45:02.622391 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:02.622370    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f86e315-sxk99" event={"ID":"dead7294-e167-41ef-937e-21f038062b38","Type":"ContainerDied","Data":"c9fbe34075def3030505e3f093b10a141daf4d4c5aa93a53e479138b98c88dcb"}
Oct 30 13:45:02.622467 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:02.622410    4364 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9fbe34075def3030505e3f093b10a141daf4d4c5aa93a53e479138b98c88dcb"
Oct 30 13:45:02.622467 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:02.622454    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f86e315-sxk99"
Oct 30 13:45:02.628260 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:02.628242    4364 generic.go:356] "Generic (PLEG): container finished" podID="c0628c77-ff94-4a69-a0a0-af20a2b64e13" containerID="3310c37f36330a6457b40ad08bf9b6f43fc5feeabcd4e3d12e04343f34bd3b2c" exitCode=60
Oct 30 13:45:02.628341 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:02.628308    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889465-wdgj9" event={"ID":"c0628c77-ff94-4a69-a0a0-af20a2b64e13","Type":"ContainerDied","Data":"3310c37f36330a6457b40ad08bf9b6f43fc5feeabcd4e3d12e04343f34bd3b2c"}
Oct 30 13:45:02.628476 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:02.628462    4364 scope.go:122] "RemoveContainer" containerID="3310c37f36330a6457b40ad08bf9b6f43fc5feeabcd4e3d12e04343f34bd3b2c"
Oct 30 13:45:03.026868 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:03.026845    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t"
Oct 30 13:45:03.119811 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:03.119767    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58073475-3ca6-4a52-b021-e08e935fad18-istio-token\" (UniqueName: \"kubernetes.io/projected/58073475-3ca6-4a52-b021-e08e935fad18-istio-token\") pod \"58073475-3ca6-4a52-b021-e08e935fad18\" (UID: \"58073475-3ca6-4a52-b021-e08e935fad18\") "
Oct 30 13:45:03.120002 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:03.119836    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58073475-3ca6-4a52-b021-e08e935fad18-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58073475-3ca6-4a52-b021-e08e935fad18-istiod-ca-cert\") pod \"58073475-3ca6-4a52-b021-e08e935fad18\" (UID: \"58073475-3ca6-4a52-b021-e08e935fad18\") "
Oct 30 13:45:03.120002 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:03.119886    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58073475-3ca6-4a52-b021-e08e935fad18-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58073475-3ca6-4a52-b021-e08e935fad18-credential-socket\") pod \"58073475-3ca6-4a52-b021-e08e935fad18\" (UID: \"58073475-3ca6-4a52-b021-e08e935fad18\") "
Oct 30 13:45:03.120002 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:03.119951    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58073475-3ca6-4a52-b021-e08e935fad18-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58073475-3ca6-4a52-b021-e08e935fad18-workload-certs\") pod \"58073475-3ca6-4a52-b021-e08e935fad18\" (UID: \"58073475-3ca6-4a52-b021-e08e935fad18\") "
Oct 30 13:45:03.120196 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:03.120072    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58073475-3ca6-4a52-b021-e08e935fad18-kube-api-access-z5vr6\" (UniqueName: \"kubernetes.io/projected/58073475-3ca6-4a52-b021-e08e935fad18-kube-api-access-z5vr6\") pod \"58073475-3ca6-4a52-b021-e08e935fad18\" (UID: \"58073475-3ca6-4a52-b021-e08e935fad18\") "
Oct 30 13:45:03.120196 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:03.120100    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58073475-3ca6-4a52-b021-e08e935fad18-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58073475-3ca6-4a52-b021-e08e935fad18-istio-envoy\") pod \"58073475-3ca6-4a52-b021-e08e935fad18\" (UID: \"58073475-3ca6-4a52-b021-e08e935fad18\") "
Oct 30 13:45:03.120196 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:03.120133    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58073475-3ca6-4a52-b021-e08e935fad18-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58073475-3ca6-4a52-b021-e08e935fad18-istio-data\") pod \"58073475-3ca6-4a52-b021-e08e935fad18\" (UID: \"58073475-3ca6-4a52-b021-e08e935fad18\") "
Oct 30 13:45:03.120196 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:03.120174    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/58073475-3ca6-4a52-b021-e08e935fad18-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58073475-3ca6-4a52-b021-e08e935fad18-istio-podinfo\") pod \"58073475-3ca6-4a52-b021-e08e935fad18\" (UID: \"58073475-3ca6-4a52-b021-e08e935fad18\") "
Oct 30 13:45:03.120489 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:03.120214    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58073475-3ca6-4a52-b021-e08e935fad18-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58073475-3ca6-4a52-b021-e08e935fad18-workload-socket\") pod \"58073475-3ca6-4a52-b021-e08e935fad18\" (UID: \"58073475-3ca6-4a52-b021-e08e935fad18\") "
Oct 30 13:45:03.120489 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:03.120295    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58073475-3ca6-4a52-b021-e08e935fad18-istiod-ca-cert" pod "58073475-3ca6-4a52-b021-e08e935fad18" (UID: "58073475-3ca6-4a52-b021-e08e935fad18"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:03.120489 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:03.120301    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58073475-3ca6-4a52-b021-e08e935fad18-credential-socket" pod "58073475-3ca6-4a52-b021-e08e935fad18" (UID: "58073475-3ca6-4a52-b021-e08e935fad18"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:45:03.120686 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:03.120593    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58073475-3ca6-4a52-b021-e08e935fad18-workload-certs" pod "58073475-3ca6-4a52-b021-e08e935fad18" (UID: "58073475-3ca6-4a52-b021-e08e935fad18"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:45:03.120686 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:03.120648    4364 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58073475-3ca6-4a52-b021-e08e935fad18-istiod-ca-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:03.120686 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:03.120672    4364 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58073475-3ca6-4a52-b021-e08e935fad18-credential-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:03.120992 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:03.120955    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58073475-3ca6-4a52-b021-e08e935fad18-istio-data" pod "58073475-3ca6-4a52-b021-e08e935fad18" (UID: "58073475-3ca6-4a52-b021-e08e935fad18"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:45:03.121083 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:03.120981    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58073475-3ca6-4a52-b021-e08e935fad18-workload-socket" pod "58073475-3ca6-4a52-b021-e08e935fad18" (UID: "58073475-3ca6-4a52-b021-e08e935fad18"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:45:03.123020 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:03.122412    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58073475-3ca6-4a52-b021-e08e935fad18-istio-token" pod "58073475-3ca6-4a52-b021-e08e935fad18" (UID: "58073475-3ca6-4a52-b021-e08e935fad18"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:03.123675 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:03.123648    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/58073475-3ca6-4a52-b021-e08e935fad18-istio-podinfo" pod "58073475-3ca6-4a52-b021-e08e935fad18" (UID: "58073475-3ca6-4a52-b021-e08e935fad18"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:45:03.123793 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:03.123652    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58073475-3ca6-4a52-b021-e08e935fad18-istio-envoy" pod "58073475-3ca6-4a52-b021-e08e935fad18" (UID: "58073475-3ca6-4a52-b021-e08e935fad18"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:45:03.123793 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:03.123781    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58073475-3ca6-4a52-b021-e08e935fad18-kube-api-access-z5vr6" pod "58073475-3ca6-4a52-b021-e08e935fad18" (UID: "58073475-3ca6-4a52-b021-e08e935fad18"). InnerVolumeSpecName "kube-api-access-z5vr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:03.221833 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:03.221787    4364 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58073475-3ca6-4a52-b021-e08e935fad18-istio-data\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:03.221833 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:03.221833    4364 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58073475-3ca6-4a52-b021-e08e935fad18-istio-podinfo\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:03.222093 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:03.221846    4364 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58073475-3ca6-4a52-b021-e08e935fad18-workload-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:03.222093 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:03.221859    4364 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58073475-3ca6-4a52-b021-e08e935fad18-istio-token\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:03.222093 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:03.221871    4364 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58073475-3ca6-4a52-b021-e08e935fad18-workload-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:03.222093 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:03.221885    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5vr6\" (UniqueName: \"kubernetes.io/projected/58073475-3ca6-4a52-b021-e08e935fad18-kube-api-access-z5vr6\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:03.222093 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:03.221900    4364 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58073475-3ca6-4a52-b021-e08e935fad18-istio-envoy\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:03.642259 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:03.642215    4364 generic.go:356] "Generic (PLEG): container finished" podID="c0628c77-ff94-4a69-a0a0-af20a2b64e13" containerID="183132d639b74fbcaf4d8049939a1c6230639d778b0a368f4d0cfcd1aaccfe4a" exitCode=60
Oct 30 13:45:03.642745 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:03.642299    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889465-wdgj9" event={"ID":"c0628c77-ff94-4a69-a0a0-af20a2b64e13","Type":"ContainerDied","Data":"183132d639b74fbcaf4d8049939a1c6230639d778b0a368f4d0cfcd1aaccfe4a"}
Oct 30 13:45:03.642745 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:03.642396    4364 scope.go:122] "RemoveContainer" containerID="3310c37f36330a6457b40ad08bf9b6f43fc5feeabcd4e3d12e04343f34bd3b2c"
Oct 30 13:45:03.642745 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:03.642705    4364 scope.go:122] "RemoveContainer" containerID="183132d639b74fbcaf4d8049939a1c6230639d778b0a368f4d0cfcd1aaccfe4a"
Oct 30 13:45:03.643042 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:45:03.643019    4364 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-52a649-29889465-wdgj9_odh-ai-gateway-infra(c0628c77-ff94-4a69-a0a0-af20a2b64e13)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889465-wdgj9" podUID="c0628c77-ff94-4a69-a0a0-af20a2b64e13"
Oct 30 13:45:03.644805 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:03.644786    4364 scope.go:122] "RemoveContainer" containerID="0ef0bf7a4b5435d8c32063cc4edaebd85964bce0554515dc5d15ac68f65ad06c"
Oct 30 13:45:03.645062 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:45:03.645034    4364 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889465-26rfr_odh-ai-gateway-infra(1453dcde-0c6d-427d-890b-8f3542a0775a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889465-26rfr" podUID="1453dcde-0c6d-427d-890b-8f3542a0775a"
Oct 30 13:45:03.646089 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:03.646067    4364 generic.go:356] "Generic (PLEG): container finished" podID="58073475-3ca6-4a52-b021-e08e935fad18" containerID="259a6384f1c83dd7ebaa0c73d9c969906b653928ebce6df6b1842fc2ce0338bf" exitCode=0
Oct 30 13:45:03.646208 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:03.646146    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t" event={"ID":"58073475-3ca6-4a52-b021-e08e935fad18","Type":"ContainerDied","Data":"259a6384f1c83dd7ebaa0c73d9c969906b653928ebce6df6b1842fc2ce0338bf"}
Oct 30 13:45:03.646208 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:03.646191    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t" event={"ID":"58073475-3ca6-4a52-b021-e08e935fad18","Type":"ContainerDied","Data":"a3408f93dd68fda42aeb89a242467ccb7a421eb3e33e5b52eaf89f8eff2e2323"}
Oct 30 13:45:03.646339 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:03.646294    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t"
Oct 30 13:45:03.648613 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:03.648581    4364 generic.go:356] "Generic (PLEG): container finished" podID="aaf4329d-4d56-4ce3-8eea-dbaf8da1c07a" containerID="18c7c111add6a6bc2aca075bea9f74df72875750e55f81442c9c73838ac921a6" exitCode=60
Oct 30 13:45:03.648727 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:03.648637    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-r87lk" event={"ID":"aaf4329d-4d56-4ce3-8eea-dbaf8da1c07a","Type":"ContainerDied","Data":"18c7c111add6a6bc2aca075bea9f74df72875750e55f81442c9c73838ac921a6"}
Oct 30 13:45:03.648950 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:03.648932    4364 scope.go:122] "RemoveContainer" containerID="18c7c111add6a6bc2aca075bea9f74df72875750e55f81442c9c73838ac921a6"
Oct 30 13:45:03.649302 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:45:03.649186    4364 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889465-r87lk_odh-ai-gateway-infra(aaf4329d-4d56-4ce3-8eea-dbaf8da1c07a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-r87lk" podUID="aaf4329d-4d56-4ce3-8eea-dbaf8da1c07a"
Oct 30 13:45:03.714325 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:03.714308    4364 scope.go:122] "RemoveContainer" containerID="259a6384f1c83dd7ebaa0c73d9c969906b653928ebce6df6b1842fc2ce0338bf"
Oct 30 13:45:03.722992 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:03.722972    4364 scope.go:122] "RemoveContainer" containerID="259a6384f1c83dd7ebaa0c73d9c969906b653928ebce6df6b1842fc2ce0338bf"
Oct 30 13:45:03.723248 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:45:03.723226    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"259a6384f1c83dd7ebaa0c73d9c969906b653928ebce6df6b1842fc2ce0338bf\": container with ID starting with 259a6384f1c83dd7ebaa0c73d9c969906b653928ebce6df6b1842fc2ce0338bf not found: ID does not exist" containerID="259a6384f1c83dd7ebaa0c73d9c969906b653928ebce6df6b1842fc2ce0338bf"
Oct 30 13:45:03.723297 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:03.723256    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"259a6384f1c83dd7ebaa0c73d9c969906b653928ebce6df6b1842fc2ce0338bf"} err="failed to get container status \"259a6384f1c83dd7ebaa0c73d9c969906b653928ebce6df6b1842fc2ce0338bf\": rpc error: code = NotFound desc = could not find container \"259a6384f1c83dd7ebaa0c73d9c969906b653928ebce6df6b1842fc2ce0338bf\": container with ID starting with 259a6384f1c83dd7ebaa0c73d9c969906b653928ebce6df6b1842fc2ce0338bf not found: ID does not exist"
Oct 30 13:45:03.723297 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:03.723271    4364 scope.go:122] "RemoveContainer" containerID="4d4e22eea6b2826e8b216093fe38b5a6c90bf9a1b74fb7c9bf6a5fa0ffce21a4"
Oct 30 13:45:03.848716 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:03.848631    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t"]
Oct 30 13:45:03.862673 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:03.862640    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-r6r8t"]
Oct 30 13:45:03.934058 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:03.934019    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc" podUID="8911b790-331d-490c-876b-647e6d2fcd14" containerName="main" containerID="cri-o://0cdd2ab96dce856d79cc9f91304a11358e3d0b285cca47ccf807a03004cf740b" gracePeriod=45
Oct 30 13:45:04.198580 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:04.198555    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc"
Oct 30 13:45:04.336819 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:04.336784    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8911b790-331d-490c-876b-647e6d2fcd14-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8911b790-331d-490c-876b-647e6d2fcd14-tmp-dir\") pod \"8911b790-331d-490c-876b-647e6d2fcd14\" (UID: \"8911b790-331d-490c-876b-647e6d2fcd14\") "
Oct 30 13:45:04.337013 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:04.336844    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8911b790-331d-490c-876b-647e6d2fcd14-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8911b790-331d-490c-876b-647e6d2fcd14-model-cache\") pod \"8911b790-331d-490c-876b-647e6d2fcd14\" (UID: \"8911b790-331d-490c-876b-647e6d2fcd14\") "
Oct 30 13:45:04.337013 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:04.336911    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8911b790-331d-490c-876b-647e6d2fcd14-tls-certs\" (UniqueName: \"kubernetes.io/secret/8911b790-331d-490c-876b-647e6d2fcd14-tls-certs\") pod \"8911b790-331d-490c-876b-647e6d2fcd14\" (UID: \"8911b790-331d-490c-876b-647e6d2fcd14\") "
Oct 30 13:45:04.337013 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:04.336993    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8911b790-331d-490c-876b-647e6d2fcd14-home\" (UniqueName: \"kubernetes.io/empty-dir/8911b790-331d-490c-876b-647e6d2fcd14-home\") pod \"8911b790-331d-490c-876b-647e6d2fcd14\" (UID: \"8911b790-331d-490c-876b-647e6d2fcd14\") "
Oct 30 13:45:04.337204 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:04.337032    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8911b790-331d-490c-876b-647e6d2fcd14-kube-api-access-jjf4m\" (UniqueName: \"kubernetes.io/projected/8911b790-331d-490c-876b-647e6d2fcd14-kube-api-access-jjf4m\") pod \"8911b790-331d-490c-876b-647e6d2fcd14\" (UID: \"8911b790-331d-490c-876b-647e6d2fcd14\") "
Oct 30 13:45:04.337204 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:04.337064    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8911b790-331d-490c-876b-647e6d2fcd14-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8911b790-331d-490c-876b-647e6d2fcd14-dshm\") pod \"8911b790-331d-490c-876b-647e6d2fcd14\" (UID: \"8911b790-331d-490c-876b-647e6d2fcd14\") "
Oct 30 13:45:04.337204 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:04.337074    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8911b790-331d-490c-876b-647e6d2fcd14-tmp-dir" pod "8911b790-331d-490c-876b-647e6d2fcd14" (UID: "8911b790-331d-490c-876b-647e6d2fcd14"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:45:04.337204 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:04.337145    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8911b790-331d-490c-876b-647e6d2fcd14-model-cache" pod "8911b790-331d-490c-876b-647e6d2fcd14" (UID: "8911b790-331d-490c-876b-647e6d2fcd14"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:45:04.337938 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:04.337514    4364 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8911b790-331d-490c-876b-647e6d2fcd14-tmp-dir\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:04.337938 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:04.337536    4364 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8911b790-331d-490c-876b-647e6d2fcd14-model-cache\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:04.337938 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:04.337648    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8911b790-331d-490c-876b-647e6d2fcd14-home" pod "8911b790-331d-490c-876b-647e6d2fcd14" (UID: "8911b790-331d-490c-876b-647e6d2fcd14"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:45:04.339870 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:04.339711    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8911b790-331d-490c-876b-647e6d2fcd14-kube-api-access-jjf4m" pod "8911b790-331d-490c-876b-647e6d2fcd14" (UID: "8911b790-331d-490c-876b-647e6d2fcd14"). InnerVolumeSpecName "kube-api-access-jjf4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:04.339870 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:04.339791    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8911b790-331d-490c-876b-647e6d2fcd14-dshm" pod "8911b790-331d-490c-876b-647e6d2fcd14" (UID: "8911b790-331d-490c-876b-647e6d2fcd14"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:45:04.340051 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:04.339885    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8911b790-331d-490c-876b-647e6d2fcd14-tls-certs" pod "8911b790-331d-490c-876b-647e6d2fcd14" (UID: "8911b790-331d-490c-876b-647e6d2fcd14"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:45:04.438501 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:04.438468    4364 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8911b790-331d-490c-876b-647e6d2fcd14-tls-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:04.438501 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:04.438496    4364 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8911b790-331d-490c-876b-647e6d2fcd14-home\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:04.438501 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:04.438507    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjf4m\" (UniqueName: \"kubernetes.io/projected/8911b790-331d-490c-876b-647e6d2fcd14-kube-api-access-jjf4m\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:04.438785 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:04.438516    4364 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8911b790-331d-490c-876b-647e6d2fcd14-dshm\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:04.657515 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:04.657488    4364 scope.go:122] "RemoveContainer" containerID="183132d639b74fbcaf4d8049939a1c6230639d778b0a368f4d0cfcd1aaccfe4a"
Oct 30 13:45:04.657988 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:45:04.657794    4364 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-52a649-29889465-wdgj9_odh-ai-gateway-infra(c0628c77-ff94-4a69-a0a0-af20a2b64e13)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889465-wdgj9" podUID="c0628c77-ff94-4a69-a0a0-af20a2b64e13"
Oct 30 13:45:04.660117 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:04.660094    4364 scope.go:122] "RemoveContainer" containerID="18c7c111add6a6bc2aca075bea9f74df72875750e55f81442c9c73838ac921a6"
Oct 30 13:45:04.660354 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:45:04.660332    4364 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889465-r87lk_odh-ai-gateway-infra(aaf4329d-4d56-4ce3-8eea-dbaf8da1c07a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-r87lk" podUID="aaf4329d-4d56-4ce3-8eea-dbaf8da1c07a"
Oct 30 13:45:04.661113 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:04.661092    4364 generic.go:356] "Generic (PLEG): container finished" podID="8911b790-331d-490c-876b-647e6d2fcd14" containerID="0cdd2ab96dce856d79cc9f91304a11358e3d0b285cca47ccf807a03004cf740b" exitCode=0
Oct 30 13:45:04.661184 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:04.661126    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc"
Oct 30 13:45:04.661184 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:04.661147    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc" event={"ID":"8911b790-331d-490c-876b-647e6d2fcd14","Type":"ContainerDied","Data":"0cdd2ab96dce856d79cc9f91304a11358e3d0b285cca47ccf807a03004cf740b"}
Oct 30 13:45:04.661265 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:04.661184    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc" event={"ID":"8911b790-331d-490c-876b-647e6d2fcd14","Type":"ContainerDied","Data":"ae22d2f1fa6147c813eb6a60c5a4ad43c4a86dd58d1bf02fbf82aab1e5bafe31"}
Oct 30 13:45:04.661265 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:04.661203    4364 scope.go:122] "RemoveContainer" containerID="0cdd2ab96dce856d79cc9f91304a11358e3d0b285cca47ccf807a03004cf740b"
Oct 30 13:45:04.672038 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:04.671937    4364 scope.go:122] "RemoveContainer" containerID="0cdd2ab96dce856d79cc9f91304a11358e3d0b285cca47ccf807a03004cf740b"
Oct 30 13:45:04.672280 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:45:04.672254    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cdd2ab96dce856d79cc9f91304a11358e3d0b285cca47ccf807a03004cf740b\": container with ID starting with 0cdd2ab96dce856d79cc9f91304a11358e3d0b285cca47ccf807a03004cf740b not found: ID does not exist" containerID="0cdd2ab96dce856d79cc9f91304a11358e3d0b285cca47ccf807a03004cf740b"
Oct 30 13:45:04.672337 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:04.672291    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cdd2ab96dce856d79cc9f91304a11358e3d0b285cca47ccf807a03004cf740b"} err="failed to get container status \"0cdd2ab96dce856d79cc9f91304a11358e3d0b285cca47ccf807a03004cf740b\": rpc error: code = NotFound desc = could not find container \"0cdd2ab96dce856d79cc9f91304a11358e3d0b285cca47ccf807a03004cf740b\": container with ID starting with 0cdd2ab96dce856d79cc9f91304a11358e3d0b285cca47ccf807a03004cf740b not found: ID does not exist"
Oct 30 13:45:04.745919 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:04.745849    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc"]
Oct 30 13:45:04.750715 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:04.750688    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-e34fd5/distinct-2-w5-e34fd5-kserve-567fb5ccc4-hn7bc"]
Oct 30 13:45:04.969233 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:04.969199    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58073475-3ca6-4a52-b021-e08e935fad18" path="/var/lib/kubelet/pods/58073475-3ca6-4a52-b021-e08e935fad18/volumes"
Oct 30 13:45:04.969657 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:04.969636    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8911b790-331d-490c-876b-647e6d2fcd14" path="/var/lib/kubelet/pods/8911b790-331d-490c-876b-647e6d2fcd14/volumes"
Oct 30 13:45:05.382295 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:05.382265    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23bf79b7-9d2p7"]
Oct 30 13:45:05.382882 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:05.382858    4364 cpu_manager.go:414] "containerMap: removing container" podUID="8911b790-331d-490c-876b-647e6d2fcd14" containerName="main"
Oct 30 13:45:05.382882 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:05.382881    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8911b790-331d-490c-876b-647e6d2fcd14" containerName="main"
Oct 30 13:45:05.383078 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:05.382897    4364 cpu_manager.go:414] "containerMap: removing container" podUID="58073475-3ca6-4a52-b021-e08e935fad18" containerName="istio-proxy"
Oct 30 13:45:05.383078 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:05.382903    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58073475-3ca6-4a52-b021-e08e935fad18" containerName="istio-proxy"
Oct 30 13:45:05.383078 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:05.382929    4364 cpu_manager.go:414] "containerMap: removing container" podUID="dead7294-e167-41ef-937e-21f038062b38" containerName="revoke-keys"
Oct 30 13:45:05.383078 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:05.382934    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dead7294-e167-41ef-937e-21f038062b38" containerName="revoke-keys"
Oct 30 13:45:05.383078 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:05.383035    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="dead7294-e167-41ef-937e-21f038062b38" containerName="revoke-keys"
Oct 30 13:45:05.383078 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:05.383047    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="8911b790-331d-490c-876b-647e6d2fcd14" containerName="main"
Oct 30 13:45:05.383078 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:05.383055    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="58073475-3ca6-4a52-b021-e08e935fad18" containerName="istio-proxy"
Oct 30 13:45:05.387555 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:05.387535    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23bf79b7-9d2p7"
Oct 30 13:45:05.406937 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:05.406906    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23bf79b7-9d2p7"]
Oct 30 13:45:05.417086 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:05.417053    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:45:05.447405 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:05.447376    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93362c37-0fc6-48b2-85b8-ee62492ba31f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23bf79b7-9d2p7\" (UID: \"93362c37-0fc6-48b2-85b8-ee62492ba31f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23bf79b7-9d2p7"
Oct 30 13:45:05.447582 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:05.447460    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zcwb\" (UniqueName: \"kubernetes.io/projected/93362c37-0fc6-48b2-85b8-ee62492ba31f-kube-api-access-8zcwb\") pod \"maas-api-revoke-sub-23bf79b7-9d2p7\" (UID: \"93362c37-0fc6-48b2-85b8-ee62492ba31f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23bf79b7-9d2p7"
Oct 30 13:45:05.447682 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:05.447657    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93362c37-0fc6-48b2-85b8-ee62492ba31f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23bf79b7-9d2p7\" (UID: \"93362c37-0fc6-48b2-85b8-ee62492ba31f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23bf79b7-9d2p7"
Oct 30 13:45:05.548444 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:05.548402    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zcwb\" (UniqueName: \"kubernetes.io/projected/93362c37-0fc6-48b2-85b8-ee62492ba31f-kube-api-access-8zcwb\") pod \"maas-api-revoke-sub-23bf79b7-9d2p7\" (UID: \"93362c37-0fc6-48b2-85b8-ee62492ba31f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23bf79b7-9d2p7"
Oct 30 13:45:05.548645 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:05.548504    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93362c37-0fc6-48b2-85b8-ee62492ba31f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23bf79b7-9d2p7\" (UID: \"93362c37-0fc6-48b2-85b8-ee62492ba31f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23bf79b7-9d2p7"
Oct 30 13:45:05.548645 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:05.548556    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93362c37-0fc6-48b2-85b8-ee62492ba31f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23bf79b7-9d2p7\" (UID: \"93362c37-0fc6-48b2-85b8-ee62492ba31f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23bf79b7-9d2p7"
Oct 30 13:45:05.549253 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:05.549228    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93362c37-0fc6-48b2-85b8-ee62492ba31f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23bf79b7-9d2p7\" (UID: \"93362c37-0fc6-48b2-85b8-ee62492ba31f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23bf79b7-9d2p7"
Oct 30 13:45:05.550984 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:05.550963    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93362c37-0fc6-48b2-85b8-ee62492ba31f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23bf79b7-9d2p7\" (UID: \"93362c37-0fc6-48b2-85b8-ee62492ba31f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23bf79b7-9d2p7"
Oct 30 13:45:05.574127 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:05.574103    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zcwb\" (UniqueName: \"kubernetes.io/projected/93362c37-0fc6-48b2-85b8-ee62492ba31f-kube-api-access-8zcwb\") pod \"maas-api-revoke-sub-23bf79b7-9d2p7\" (UID: \"93362c37-0fc6-48b2-85b8-ee62492ba31f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23bf79b7-9d2p7"
Oct 30 13:45:05.576830 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:05.576807    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-52a649-65c5fbbff9-n6rqr" podUID="22d0f282-215c-45d9-9fbb-15190b9ebd10" containerName="payload-pre-processing" containerID="cri-o://70236c8c1dd87912d3fa93edd91532b2c1d9d7ac2b49a60c725d3ac8c3ba5e54" gracePeriod=25
Oct 30 13:45:05.701723 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:05.701642    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23bf79b7-9d2p7"
Oct 30 13:45:05.884141 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:05.884103    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:45:06.003834 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:06.003808    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23bf79b7-9d2p7"]
Oct 30 13:45:06.004524 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:45:06.004494    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93362c37_0fc6_48b2_85b8_ee62492ba31f.slice/crio-63003543605b72649ef200039e9baa112b71205f56557e078c71b24d598ce32e WatchSource:0}: Error finding container 63003543605b72649ef200039e9baa112b71205f56557e078c71b24d598ce32e: Status 404 returned error can't find the container with id 63003543605b72649ef200039e9baa112b71205f56557e078c71b24d598ce32e
Oct 30 13:45:06.675777 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:06.675738    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23bf79b7-9d2p7" event={"ID":"93362c37-0fc6-48b2-85b8-ee62492ba31f","Type":"ContainerStarted","Data":"25462c8cbbac98743555fed75e826c6285c38522c87bf1cdfcc666362151effb"}
Oct 30 13:45:06.675982 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:06.675803    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23bf79b7-9d2p7" event={"ID":"93362c37-0fc6-48b2-85b8-ee62492ba31f","Type":"ContainerStarted","Data":"63003543605b72649ef200039e9baa112b71205f56557e078c71b24d598ce32e"}
Oct 30 13:45:06.720477 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:06.720425    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23bf79b7-9d2p7" podStartSLOduration=1.720410888 podStartE2EDuration="1.720410888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:06.720356197 +0000 UTC m=+3205.886720845" watchObservedRunningTime="2026-10-30 13:45:06.720410888 +0000 UTC m=+3205.886775535"
Oct 30 13:45:07.029798 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:07.029720    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-cda0d9-74764bd4cf-l64mp" podUID="bd1b7e78-a663-4d45-a742-5994a0259a24" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:45:07.682913 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:07.682872    4364 generic.go:356] "Generic (PLEG): container finished" podID="93362c37-0fc6-48b2-85b8-ee62492ba31f" containerID="25462c8cbbac98743555fed75e826c6285c38522c87bf1cdfcc666362151effb" exitCode=0
Oct 30 13:45:07.683114 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:07.682970    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23bf79b7-9d2p7" event={"ID":"93362c37-0fc6-48b2-85b8-ee62492ba31f","Type":"ContainerDied","Data":"25462c8cbbac98743555fed75e826c6285c38522c87bf1cdfcc666362151effb"}
Oct 30 13:45:08.113402 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:08.113300    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz" podUID="a568f1ce-602d-4069-a41c-9fe1a7635c5a" containerName="main" containerID="cri-o://50aab43258389aa70e33c69e44aad520f63075b8efac299e245b80b9365d852e" gracePeriod=45
Oct 30 13:45:08.122794 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:08.122752    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz" podUID="bc5caf69-a649-47e5-8910-9b8f331dfd4c" containerName="main" containerID="cri-o://2ec7063c449c85cf544188eafe6b344cb3344f6f2574924f27ce8248d593d747" gracePeriod=45
Oct 30 13:45:08.402664 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:08.402637    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz"
Oct 30 13:45:08.419303 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:08.419274    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz"
Oct 30 13:45:08.480392 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:08.480330    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a568f1ce-602d-4069-a41c-9fe1a7635c5a-kube-api-access-n6jnz\" (UniqueName: \"kubernetes.io/projected/a568f1ce-602d-4069-a41c-9fe1a7635c5a-kube-api-access-n6jnz\") pod \"a568f1ce-602d-4069-a41c-9fe1a7635c5a\" (UID: \"a568f1ce-602d-4069-a41c-9fe1a7635c5a\") "
Oct 30 13:45:08.480626 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:08.480450    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc5caf69-a649-47e5-8910-9b8f331dfd4c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc5caf69-a649-47e5-8910-9b8f331dfd4c-tmp-dir\") pod \"bc5caf69-a649-47e5-8910-9b8f331dfd4c\" (UID: \"bc5caf69-a649-47e5-8910-9b8f331dfd4c\") "
Oct 30 13:45:08.480626 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:08.480538    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a568f1ce-602d-4069-a41c-9fe1a7635c5a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a568f1ce-602d-4069-a41c-9fe1a7635c5a-model-cache\") pod \"a568f1ce-602d-4069-a41c-9fe1a7635c5a\" (UID: \"a568f1ce-602d-4069-a41c-9fe1a7635c5a\") "
Oct 30 13:45:08.480626 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:08.480621    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc5caf69-a649-47e5-8910-9b8f331dfd4c-home\" (UniqueName: \"kubernetes.io/empty-dir/bc5caf69-a649-47e5-8910-9b8f331dfd4c-home\") pod \"bc5caf69-a649-47e5-8910-9b8f331dfd4c\" (UID: \"bc5caf69-a649-47e5-8910-9b8f331dfd4c\") "
Oct 30 13:45:08.480837 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:08.480665    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a568f1ce-602d-4069-a41c-9fe1a7635c5a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a568f1ce-602d-4069-a41c-9fe1a7635c5a-dshm\") pod \"a568f1ce-602d-4069-a41c-9fe1a7635c5a\" (UID: \"a568f1ce-602d-4069-a41c-9fe1a7635c5a\") "
Oct 30 13:45:08.480837 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:08.480702    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a568f1ce-602d-4069-a41c-9fe1a7635c5a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a568f1ce-602d-4069-a41c-9fe1a7635c5a-tmp-dir\") pod \"a568f1ce-602d-4069-a41c-9fe1a7635c5a\" (UID: \"a568f1ce-602d-4069-a41c-9fe1a7635c5a\") "
Oct 30 13:45:08.480837 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:08.480780    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc5caf69-a649-47e5-8910-9b8f331dfd4c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc5caf69-a649-47e5-8910-9b8f331dfd4c-model-cache\") pod \"bc5caf69-a649-47e5-8910-9b8f331dfd4c\" (UID: \"bc5caf69-a649-47e5-8910-9b8f331dfd4c\") "
Oct 30 13:45:08.481005 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:08.480861    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc5caf69-a649-47e5-8910-9b8f331dfd4c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc5caf69-a649-47e5-8910-9b8f331dfd4c-dshm\") pod \"bc5caf69-a649-47e5-8910-9b8f331dfd4c\" (UID: \"bc5caf69-a649-47e5-8910-9b8f331dfd4c\") "
Oct 30 13:45:08.481263 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:08.481208    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc5caf69-a649-47e5-8910-9b8f331dfd4c-tmp-dir" pod "bc5caf69-a649-47e5-8910-9b8f331dfd4c" (UID: "bc5caf69-a649-47e5-8910-9b8f331dfd4c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:45:08.481263 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:08.481213    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a568f1ce-602d-4069-a41c-9fe1a7635c5a-tmp-dir" pod "a568f1ce-602d-4069-a41c-9fe1a7635c5a" (UID: "a568f1ce-602d-4069-a41c-9fe1a7635c5a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:45:08.481442 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:08.481422    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a568f1ce-602d-4069-a41c-9fe1a7635c5a-model-cache" pod "a568f1ce-602d-4069-a41c-9fe1a7635c5a" (UID: "a568f1ce-602d-4069-a41c-9fe1a7635c5a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:45:08.481442 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:08.481430    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc5caf69-a649-47e5-8910-9b8f331dfd4c-model-cache" pod "bc5caf69-a649-47e5-8910-9b8f331dfd4c" (UID: "bc5caf69-a649-47e5-8910-9b8f331dfd4c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:45:08.481811 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:08.481784    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc5caf69-a649-47e5-8910-9b8f331dfd4c-home" pod "bc5caf69-a649-47e5-8910-9b8f331dfd4c" (UID: "bc5caf69-a649-47e5-8910-9b8f331dfd4c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:45:08.481911 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:08.481816    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc5caf69-a649-47e5-8910-9b8f331dfd4c-kube-api-access-gprtw\" (UniqueName: \"kubernetes.io/projected/bc5caf69-a649-47e5-8910-9b8f331dfd4c-kube-api-access-gprtw\") pod \"bc5caf69-a649-47e5-8910-9b8f331dfd4c\" (UID: \"bc5caf69-a649-47e5-8910-9b8f331dfd4c\") "
Oct 30 13:45:08.481911 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:08.481902    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a568f1ce-602d-4069-a41c-9fe1a7635c5a-tls-certs\" (UniqueName: \"kubernetes.io/secret/a568f1ce-602d-4069-a41c-9fe1a7635c5a-tls-certs\") pod \"a568f1ce-602d-4069-a41c-9fe1a7635c5a\" (UID: \"a568f1ce-602d-4069-a41c-9fe1a7635c5a\") "
Oct 30 13:45:08.482035 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:08.481973    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a568f1ce-602d-4069-a41c-9fe1a7635c5a-home\" (UniqueName: \"kubernetes.io/empty-dir/a568f1ce-602d-4069-a41c-9fe1a7635c5a-home\") pod \"a568f1ce-602d-4069-a41c-9fe1a7635c5a\" (UID: \"a568f1ce-602d-4069-a41c-9fe1a7635c5a\") "
Oct 30 13:45:08.482035 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:08.482000    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc5caf69-a649-47e5-8910-9b8f331dfd4c-tls-certs\" (UniqueName: \"kubernetes.io/secret/bc5caf69-a649-47e5-8910-9b8f331dfd4c-tls-certs\") pod \"bc5caf69-a649-47e5-8910-9b8f331dfd4c\" (UID: \"bc5caf69-a649-47e5-8910-9b8f331dfd4c\") "
Oct 30 13:45:08.482413 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:08.482389    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a568f1ce-602d-4069-a41c-9fe1a7635c5a-home" pod "a568f1ce-602d-4069-a41c-9fe1a7635c5a" (UID: "a568f1ce-602d-4069-a41c-9fe1a7635c5a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:45:08.482545 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:08.482521    4364 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc5caf69-a649-47e5-8910-9b8f331dfd4c-tmp-dir\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:08.482649 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:08.482551    4364 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a568f1ce-602d-4069-a41c-9fe1a7635c5a-model-cache\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:08.482649 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:08.482567    4364 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc5caf69-a649-47e5-8910-9b8f331dfd4c-home\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:08.482649 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:08.482581    4364 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a568f1ce-602d-4069-a41c-9fe1a7635c5a-tmp-dir\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:08.482649 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:08.482609    4364 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc5caf69-a649-47e5-8910-9b8f331dfd4c-model-cache\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:08.482649 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:08.482622    4364 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a568f1ce-602d-4069-a41c-9fe1a7635c5a-home\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:08.483502 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:08.483462    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a568f1ce-602d-4069-a41c-9fe1a7635c5a-kube-api-access-n6jnz" pod "a568f1ce-602d-4069-a41c-9fe1a7635c5a" (UID: "a568f1ce-602d-4069-a41c-9fe1a7635c5a"). InnerVolumeSpecName "kube-api-access-n6jnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:08.483631 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:08.483542    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a568f1ce-602d-4069-a41c-9fe1a7635c5a-dshm" pod "a568f1ce-602d-4069-a41c-9fe1a7635c5a" (UID: "a568f1ce-602d-4069-a41c-9fe1a7635c5a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:45:08.484462 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:08.484438    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc5caf69-a649-47e5-8910-9b8f331dfd4c-kube-api-access-gprtw" pod "bc5caf69-a649-47e5-8910-9b8f331dfd4c" (UID: "bc5caf69-a649-47e5-8910-9b8f331dfd4c"). InnerVolumeSpecName "kube-api-access-gprtw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:08.484861 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:08.484832    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc5caf69-a649-47e5-8910-9b8f331dfd4c-tls-certs" pod "bc5caf69-a649-47e5-8910-9b8f331dfd4c" (UID: "bc5caf69-a649-47e5-8910-9b8f331dfd4c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:45:08.485365 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:08.485334    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc5caf69-a649-47e5-8910-9b8f331dfd4c-dshm" pod "bc5caf69-a649-47e5-8910-9b8f331dfd4c" (UID: "bc5caf69-a649-47e5-8910-9b8f331dfd4c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:45:08.485547 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:08.485520    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a568f1ce-602d-4069-a41c-9fe1a7635c5a-tls-certs" pod "a568f1ce-602d-4069-a41c-9fe1a7635c5a" (UID: "a568f1ce-602d-4069-a41c-9fe1a7635c5a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:45:08.584140 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:08.584104    4364 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a568f1ce-602d-4069-a41c-9fe1a7635c5a-dshm\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:08.584140 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:08.584138    4364 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc5caf69-a649-47e5-8910-9b8f331dfd4c-dshm\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:08.584342 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:08.584147    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gprtw\" (UniqueName: \"kubernetes.io/projected/bc5caf69-a649-47e5-8910-9b8f331dfd4c-kube-api-access-gprtw\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:08.584342 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:08.584158    4364 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a568f1ce-602d-4069-a41c-9fe1a7635c5a-tls-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:08.584342 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:08.584169    4364 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc5caf69-a649-47e5-8910-9b8f331dfd4c-tls-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:08.584342 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:08.584177    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6jnz\" (UniqueName: \"kubernetes.io/projected/a568f1ce-602d-4069-a41c-9fe1a7635c5a-kube-api-access-n6jnz\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:08.691569 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:08.691479    4364 generic.go:356] "Generic (PLEG): container finished" podID="a568f1ce-602d-4069-a41c-9fe1a7635c5a" containerID="50aab43258389aa70e33c69e44aad520f63075b8efac299e245b80b9365d852e" exitCode=0
Oct 30 13:45:08.691770 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:08.691561    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz"
Oct 30 13:45:08.691770 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:08.691559    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz" event={"ID":"a568f1ce-602d-4069-a41c-9fe1a7635c5a","Type":"ContainerDied","Data":"50aab43258389aa70e33c69e44aad520f63075b8efac299e245b80b9365d852e"}
Oct 30 13:45:08.691770 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:08.691663    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz" event={"ID":"a568f1ce-602d-4069-a41c-9fe1a7635c5a","Type":"ContainerDied","Data":"738942d921c36bae35b0f7772c729ec21be5dc90a01d975b9d83a4d0c477e406"}
Oct 30 13:45:08.691770 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:08.691682    4364 scope.go:122] "RemoveContainer" containerID="50aab43258389aa70e33c69e44aad520f63075b8efac299e245b80b9365d852e"
Oct 30 13:45:08.693388 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:08.693359    4364 generic.go:356] "Generic (PLEG): container finished" podID="bc5caf69-a649-47e5-8910-9b8f331dfd4c" containerID="2ec7063c449c85cf544188eafe6b344cb3344f6f2574924f27ce8248d593d747" exitCode=0
Oct 30 13:45:08.693700 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:08.693657    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz" event={"ID":"bc5caf69-a649-47e5-8910-9b8f331dfd4c","Type":"ContainerDied","Data":"2ec7063c449c85cf544188eafe6b344cb3344f6f2574924f27ce8248d593d747"}
Oct 30 13:45:08.693700 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:08.693695    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz" event={"ID":"bc5caf69-a649-47e5-8910-9b8f331dfd4c","Type":"ContainerDied","Data":"cfc488672987dc30f155d337874f1b808af4826566d274b3a4b9ce71f00aff25"}
Oct 30 13:45:08.693858 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:08.693717    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz"
Oct 30 13:45:08.712716 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:08.712691    4364 scope.go:122] "RemoveContainer" containerID="50aab43258389aa70e33c69e44aad520f63075b8efac299e245b80b9365d852e"
Oct 30 13:45:08.713092 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:45:08.713054    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50aab43258389aa70e33c69e44aad520f63075b8efac299e245b80b9365d852e\": container with ID starting with 50aab43258389aa70e33c69e44aad520f63075b8efac299e245b80b9365d852e not found: ID does not exist" containerID="50aab43258389aa70e33c69e44aad520f63075b8efac299e245b80b9365d852e"
Oct 30 13:45:08.713174 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:08.713105    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50aab43258389aa70e33c69e44aad520f63075b8efac299e245b80b9365d852e"} err="failed to get container status \"50aab43258389aa70e33c69e44aad520f63075b8efac299e245b80b9365d852e\": rpc error: code = NotFound desc = could not find container \"50aab43258389aa70e33c69e44aad520f63075b8efac299e245b80b9365d852e\": container with ID starting with 50aab43258389aa70e33c69e44aad520f63075b8efac299e245b80b9365d852e not found: ID does not exist"
Oct 30 13:45:08.713174 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:08.713129    4364 scope.go:122] "RemoveContainer" containerID="2ec7063c449c85cf544188eafe6b344cb3344f6f2574924f27ce8248d593d747"
Oct 30 13:45:08.722770 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:08.722746    4364 scope.go:122] "RemoveContainer" containerID="2ec7063c449c85cf544188eafe6b344cb3344f6f2574924f27ce8248d593d747"
Oct 30 13:45:08.723069 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:45:08.723052    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ec7063c449c85cf544188eafe6b344cb3344f6f2574924f27ce8248d593d747\": container with ID starting with 2ec7063c449c85cf544188eafe6b344cb3344f6f2574924f27ce8248d593d747 not found: ID does not exist" containerID="2ec7063c449c85cf544188eafe6b344cb3344f6f2574924f27ce8248d593d747"
Oct 30 13:45:08.723145 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:08.723078    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ec7063c449c85cf544188eafe6b344cb3344f6f2574924f27ce8248d593d747"} err="failed to get container status \"2ec7063c449c85cf544188eafe6b344cb3344f6f2574924f27ce8248d593d747\": rpc error: code = NotFound desc = could not find container \"2ec7063c449c85cf544188eafe6b344cb3344f6f2574924f27ce8248d593d747\": container with ID starting with 2ec7063c449c85cf544188eafe6b344cb3344f6f2574924f27ce8248d593d747 not found: ID does not exist"
Oct 30 13:45:08.754366 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:08.754325    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz"]
Oct 30 13:45:08.757044 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:08.757015    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-f07dfe/distinct-w0-f07dfe-kserve-74c74cc4d9-m46qz"]
Oct 30 13:45:08.797701 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:08.797638    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz"]
Oct 30 13:45:08.801120 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:08.801091    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-f07dfe/distinct-2-w0-f07dfe-kserve-77b778cbf4-2ksbz"]
Oct 30 13:45:08.826945 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:08.826922    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23bf79b7-9d2p7"
Oct 30 13:45:08.886983 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:08.886953    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93362c37-0fc6-48b2-85b8-ee62492ba31f-kube-api-access-8zcwb\" (UniqueName: \"kubernetes.io/projected/93362c37-0fc6-48b2-85b8-ee62492ba31f-kube-api-access-8zcwb\") pod \"93362c37-0fc6-48b2-85b8-ee62492ba31f\" (UID: \"93362c37-0fc6-48b2-85b8-ee62492ba31f\") "
Oct 30 13:45:08.887153 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:08.887073    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93362c37-0fc6-48b2-85b8-ee62492ba31f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93362c37-0fc6-48b2-85b8-ee62492ba31f-maas-api-openshift-ca\") pod \"93362c37-0fc6-48b2-85b8-ee62492ba31f\" (UID: \"93362c37-0fc6-48b2-85b8-ee62492ba31f\") "
Oct 30 13:45:08.887234 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:08.887189    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93362c37-0fc6-48b2-85b8-ee62492ba31f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93362c37-0fc6-48b2-85b8-ee62492ba31f-maas-api-xks-ca\") pod \"93362c37-0fc6-48b2-85b8-ee62492ba31f\" (UID: \"93362c37-0fc6-48b2-85b8-ee62492ba31f\") "
Oct 30 13:45:08.887522 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:08.887490    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93362c37-0fc6-48b2-85b8-ee62492ba31f-maas-api-openshift-ca" pod "93362c37-0fc6-48b2-85b8-ee62492ba31f" (UID: "93362c37-0fc6-48b2-85b8-ee62492ba31f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:08.889388 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:08.889353    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93362c37-0fc6-48b2-85b8-ee62492ba31f-maas-api-xks-ca" pod "93362c37-0fc6-48b2-85b8-ee62492ba31f" (UID: "93362c37-0fc6-48b2-85b8-ee62492ba31f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:45:08.889668 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:08.889640    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93362c37-0fc6-48b2-85b8-ee62492ba31f-kube-api-access-8zcwb" pod "93362c37-0fc6-48b2-85b8-ee62492ba31f" (UID: "93362c37-0fc6-48b2-85b8-ee62492ba31f"). InnerVolumeSpecName "kube-api-access-8zcwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:08.969459 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:08.969381    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a568f1ce-602d-4069-a41c-9fe1a7635c5a" path="/var/lib/kubelet/pods/a568f1ce-602d-4069-a41c-9fe1a7635c5a/volumes"
Oct 30 13:45:08.969884 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:08.969868    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc5caf69-a649-47e5-8910-9b8f331dfd4c" path="/var/lib/kubelet/pods/bc5caf69-a649-47e5-8910-9b8f331dfd4c/volumes"
Oct 30 13:45:08.988646 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:08.988587    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93362c37-0fc6-48b2-85b8-ee62492ba31f-maas-api-xks-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:08.988646 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:08.988624    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zcwb\" (UniqueName: \"kubernetes.io/projected/93362c37-0fc6-48b2-85b8-ee62492ba31f-kube-api-access-8zcwb\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:08.988646 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:08.988634    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93362c37-0fc6-48b2-85b8-ee62492ba31f-maas-api-openshift-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:09.273547 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:09.273458    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-52a649-65c5fbbff9-n6rqr" podUID="22d0f282-215c-45d9-9fbb-15190b9ebd10" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:45:09.702138 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:09.702106    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23bf79b7-9d2p7"
Oct 30 13:45:09.702138 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:09.702117    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23bf79b7-9d2p7" event={"ID":"93362c37-0fc6-48b2-85b8-ee62492ba31f","Type":"ContainerDied","Data":"63003543605b72649ef200039e9baa112b71205f56557e078c71b24d598ce32e"}
Oct 30 13:45:09.702451 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:09.702170    4364 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63003543605b72649ef200039e9baa112b71205f56557e078c71b24d598ce32e"
Oct 30 13:45:10.010114 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:10.010016    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh" podUID="d41ac9e5-eeed-4e07-a7fa-05d3260630b6" containerName="main" containerID="cri-o://e8043a1abb02b2769b630de3c42da99798366c063cff496dbe5d1c742ef16fe8" gracePeriod=45
Oct 30 13:45:10.271417 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:10.271347    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh"
Oct 30 13:45:10.405168 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:10.405129    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d41ac9e5-eeed-4e07-a7fa-05d3260630b6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d41ac9e5-eeed-4e07-a7fa-05d3260630b6-model-cache\") pod \"d41ac9e5-eeed-4e07-a7fa-05d3260630b6\" (UID: \"d41ac9e5-eeed-4e07-a7fa-05d3260630b6\") "
Oct 30 13:45:10.405705 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:10.405281    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d41ac9e5-eeed-4e07-a7fa-05d3260630b6-tls-certs\" (UniqueName: \"kubernetes.io/secret/d41ac9e5-eeed-4e07-a7fa-05d3260630b6-tls-certs\") pod \"d41ac9e5-eeed-4e07-a7fa-05d3260630b6\" (UID: \"d41ac9e5-eeed-4e07-a7fa-05d3260630b6\") "
Oct 30 13:45:10.405705 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:10.405348    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d41ac9e5-eeed-4e07-a7fa-05d3260630b6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d41ac9e5-eeed-4e07-a7fa-05d3260630b6-tmp-dir\") pod \"d41ac9e5-eeed-4e07-a7fa-05d3260630b6\" (UID: \"d41ac9e5-eeed-4e07-a7fa-05d3260630b6\") "
Oct 30 13:45:10.405705 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:10.405417    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d41ac9e5-eeed-4e07-a7fa-05d3260630b6-home\" (UniqueName: \"kubernetes.io/empty-dir/d41ac9e5-eeed-4e07-a7fa-05d3260630b6-home\") pod \"d41ac9e5-eeed-4e07-a7fa-05d3260630b6\" (UID: \"d41ac9e5-eeed-4e07-a7fa-05d3260630b6\") "
Oct 30 13:45:10.405705 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:10.405443    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d41ac9e5-eeed-4e07-a7fa-05d3260630b6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d41ac9e5-eeed-4e07-a7fa-05d3260630b6-dshm\") pod \"d41ac9e5-eeed-4e07-a7fa-05d3260630b6\" (UID: \"d41ac9e5-eeed-4e07-a7fa-05d3260630b6\") "
Oct 30 13:45:10.405705 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:10.405474    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d41ac9e5-eeed-4e07-a7fa-05d3260630b6-kube-api-access-sds5p\" (UniqueName: \"kubernetes.io/projected/d41ac9e5-eeed-4e07-a7fa-05d3260630b6-kube-api-access-sds5p\") pod \"d41ac9e5-eeed-4e07-a7fa-05d3260630b6\" (UID: \"d41ac9e5-eeed-4e07-a7fa-05d3260630b6\") "
Oct 30 13:45:10.406089 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:10.406064    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d41ac9e5-eeed-4e07-a7fa-05d3260630b6-tmp-dir" pod "d41ac9e5-eeed-4e07-a7fa-05d3260630b6" (UID: "d41ac9e5-eeed-4e07-a7fa-05d3260630b6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:45:10.406240 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:10.406217    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d41ac9e5-eeed-4e07-a7fa-05d3260630b6-model-cache" pod "d41ac9e5-eeed-4e07-a7fa-05d3260630b6" (UID: "d41ac9e5-eeed-4e07-a7fa-05d3260630b6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:45:10.406800 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:10.406769    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d41ac9e5-eeed-4e07-a7fa-05d3260630b6-home" pod "d41ac9e5-eeed-4e07-a7fa-05d3260630b6" (UID: "d41ac9e5-eeed-4e07-a7fa-05d3260630b6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:45:10.408895 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:10.408864    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d41ac9e5-eeed-4e07-a7fa-05d3260630b6-kube-api-access-sds5p" pod "d41ac9e5-eeed-4e07-a7fa-05d3260630b6" (UID: "d41ac9e5-eeed-4e07-a7fa-05d3260630b6"). InnerVolumeSpecName "kube-api-access-sds5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:10.408895 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:10.408873    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d41ac9e5-eeed-4e07-a7fa-05d3260630b6-dshm" pod "d41ac9e5-eeed-4e07-a7fa-05d3260630b6" (UID: "d41ac9e5-eeed-4e07-a7fa-05d3260630b6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:45:10.409420 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:10.409398    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d41ac9e5-eeed-4e07-a7fa-05d3260630b6-tls-certs" pod "d41ac9e5-eeed-4e07-a7fa-05d3260630b6" (UID: "d41ac9e5-eeed-4e07-a7fa-05d3260630b6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:45:10.507339 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:10.507301    4364 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d41ac9e5-eeed-4e07-a7fa-05d3260630b6-model-cache\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:10.507339 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:10.507337    4364 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d41ac9e5-eeed-4e07-a7fa-05d3260630b6-tls-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:10.507339 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:10.507353    4364 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d41ac9e5-eeed-4e07-a7fa-05d3260630b6-tmp-dir\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:10.507339 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:10.507365    4364 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d41ac9e5-eeed-4e07-a7fa-05d3260630b6-home\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:10.507850 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:10.507376    4364 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d41ac9e5-eeed-4e07-a7fa-05d3260630b6-dshm\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:10.508300 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:10.507388    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sds5p\" (UniqueName: \"kubernetes.io/projected/d41ac9e5-eeed-4e07-a7fa-05d3260630b6-kube-api-access-sds5p\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:10.710177 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:10.710144    4364 generic.go:356] "Generic (PLEG): container finished" podID="d41ac9e5-eeed-4e07-a7fa-05d3260630b6" containerID="e8043a1abb02b2769b630de3c42da99798366c063cff496dbe5d1c742ef16fe8" exitCode=0
Oct 30 13:45:10.710367 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:10.710189    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh"
Oct 30 13:45:10.710367 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:10.710235    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh" event={"ID":"d41ac9e5-eeed-4e07-a7fa-05d3260630b6","Type":"ContainerDied","Data":"e8043a1abb02b2769b630de3c42da99798366c063cff496dbe5d1c742ef16fe8"}
Oct 30 13:45:10.710367 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:10.710281    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh" event={"ID":"d41ac9e5-eeed-4e07-a7fa-05d3260630b6","Type":"ContainerDied","Data":"98bcbb0b4ad0d03832c6ffc8a0d77cbf757173f1256260e236e3ecccf2f94bb3"}
Oct 30 13:45:10.710367 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:10.710305    4364 scope.go:122] "RemoveContainer" containerID="e8043a1abb02b2769b630de3c42da99798366c063cff496dbe5d1c742ef16fe8"
Oct 30 13:45:10.720739 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:10.720587    4364 scope.go:122] "RemoveContainer" containerID="e8043a1abb02b2769b630de3c42da99798366c063cff496dbe5d1c742ef16fe8"
Oct 30 13:45:10.720967 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:45:10.720941    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8043a1abb02b2769b630de3c42da99798366c063cff496dbe5d1c742ef16fe8\": container with ID starting with e8043a1abb02b2769b630de3c42da99798366c063cff496dbe5d1c742ef16fe8 not found: ID does not exist" containerID="e8043a1abb02b2769b630de3c42da99798366c063cff496dbe5d1c742ef16fe8"
Oct 30 13:45:10.721082 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:10.720978    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8043a1abb02b2769b630de3c42da99798366c063cff496dbe5d1c742ef16fe8"} err="failed to get container status \"e8043a1abb02b2769b630de3c42da99798366c063cff496dbe5d1c742ef16fe8\": rpc error: code = NotFound desc = could not find container \"e8043a1abb02b2769b630de3c42da99798366c063cff496dbe5d1c742ef16fe8\": container with ID starting with e8043a1abb02b2769b630de3c42da99798366c063cff496dbe5d1c742ef16fe8 not found: ID does not exist"
Oct 30 13:45:10.772709 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:10.772673    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh"]
Oct 30 13:45:10.786182 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:10.786150    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-52a649/distinct-w1-52a649-kserve-f5d4c7b95-ldndh"]
Oct 30 13:45:10.970528 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:10.970452    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d41ac9e5-eeed-4e07-a7fa-05d3260630b6" path="/var/lib/kubelet/pods/d41ac9e5-eeed-4e07-a7fa-05d3260630b6/volumes"
Oct 30 13:45:11.504512 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:45:11.504469    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:45:13.963054 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:13.963015    4364 scope.go:122] "RemoveContainer" containerID="0ef0bf7a4b5435d8c32063cc4edaebd85964bce0554515dc5d15ac68f65ad06c"
Oct 30 13:45:14.301056 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:14.301027    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f07dfe-bbj94"]
Oct 30 13:45:14.314782 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:14.314751    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f07dfe-bbj94"]
Oct 30 13:45:14.746160 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:14.746127    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889465-26rfr" event={"ID":"1453dcde-0c6d-427d-890b-8f3542a0775a","Type":"ContainerStarted","Data":"0c10f074630824f5d7f80b92e38a2cfff56c7124733957231cee492c5fab3e34"}
Oct 30 13:45:14.795207 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:14.795155    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889465-26rfr" podStartSLOduration=14.795143219 podStartE2EDuration="14.795143219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:14.792742928 +0000 UTC m=+3213.959107575" watchObservedRunningTime="2026-10-30 13:45:14.795143219 +0000 UTC m=+3213.961507867"
Oct 30 13:45:14.962496 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:14.962466    4364 scope.go:122] "RemoveContainer" containerID="18c7c111add6a6bc2aca075bea9f74df72875750e55f81442c9c73838ac921a6"
Oct 30 13:45:14.962680 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:14.962507    4364 scope.go:122] "RemoveContainer" containerID="183132d639b74fbcaf4d8049939a1c6230639d778b0a368f4d0cfcd1aaccfe4a"
Oct 30 13:45:14.970053 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:14.970027    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1ebdbad-f9d4-4758-a7a8-e2513b6d837d" path="/var/lib/kubelet/pods/f1ebdbad-f9d4-4758-a7a8-e2513b6d837d/volumes"
Oct 30 13:45:15.757773 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:15.757739    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889465-wdgj9" event={"ID":"c0628c77-ff94-4a69-a0a0-af20a2b64e13","Type":"ContainerStarted","Data":"10012c141e8c91d4d36979c7895cec774cbda0bc34bab96d92fef8e616e4daf2"}
Oct 30 13:45:15.759483 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:15.759456    4364 generic.go:356] "Generic (PLEG): container finished" podID="1453dcde-0c6d-427d-890b-8f3542a0775a" containerID="0c10f074630824f5d7f80b92e38a2cfff56c7124733957231cee492c5fab3e34" exitCode=60
Oct 30 13:45:15.759636 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:15.759524    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889465-26rfr" event={"ID":"1453dcde-0c6d-427d-890b-8f3542a0775a","Type":"ContainerDied","Data":"0c10f074630824f5d7f80b92e38a2cfff56c7124733957231cee492c5fab3e34"}
Oct 30 13:45:15.759636 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:15.759572    4364 scope.go:122] "RemoveContainer" containerID="0ef0bf7a4b5435d8c32063cc4edaebd85964bce0554515dc5d15ac68f65ad06c"
Oct 30 13:45:15.759917 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:15.759899    4364 scope.go:122] "RemoveContainer" containerID="0c10f074630824f5d7f80b92e38a2cfff56c7124733957231cee492c5fab3e34"
Oct 30 13:45:15.760194 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:45:15.760173    4364 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889465-26rfr_odh-ai-gateway-infra(1453dcde-0c6d-427d-890b-8f3542a0775a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889465-26rfr" podUID="1453dcde-0c6d-427d-890b-8f3542a0775a"
Oct 30 13:45:15.761395 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:15.761374    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-r87lk" event={"ID":"aaf4329d-4d56-4ce3-8eea-dbaf8da1c07a","Type":"ContainerStarted","Data":"67b7dbc6efa3742857cf633d53629a6335a671c461d320fc58ecb28aa654bf80"}
Oct 30 13:45:15.803434 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:15.803403    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889465-26rfr"]
Oct 30 13:45:16.001659 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:16.001627    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889465-wdgj9"]
Oct 30 13:45:16.770010 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:16.769971    4364 generic.go:356] "Generic (PLEG): container finished" podID="c0628c77-ff94-4a69-a0a0-af20a2b64e13" containerID="10012c141e8c91d4d36979c7895cec774cbda0bc34bab96d92fef8e616e4daf2" exitCode=60
Oct 30 13:45:16.772061 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:16.770054    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889465-wdgj9" event={"ID":"c0628c77-ff94-4a69-a0a0-af20a2b64e13","Type":"ContainerDied","Data":"10012c141e8c91d4d36979c7895cec774cbda0bc34bab96d92fef8e616e4daf2"}
Oct 30 13:45:16.772061 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:16.770123    4364 scope.go:122] "RemoveContainer" containerID="183132d639b74fbcaf4d8049939a1c6230639d778b0a368f4d0cfcd1aaccfe4a"
Oct 30 13:45:16.774108 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:16.774082    4364 generic.go:356] "Generic (PLEG): container finished" podID="aaf4329d-4d56-4ce3-8eea-dbaf8da1c07a" containerID="67b7dbc6efa3742857cf633d53629a6335a671c461d320fc58ecb28aa654bf80" exitCode=60
Oct 30 13:45:16.774262 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:16.774125    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-r87lk" event={"ID":"aaf4329d-4d56-4ce3-8eea-dbaf8da1c07a","Type":"ContainerDied","Data":"67b7dbc6efa3742857cf633d53629a6335a671c461d320fc58ecb28aa654bf80"}
Oct 30 13:45:16.774549 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:16.774528    4364 scope.go:122] "RemoveContainer" containerID="67b7dbc6efa3742857cf633d53629a6335a671c461d320fc58ecb28aa654bf80"
Oct 30 13:45:16.774900 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:45:16.774873    4364 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889465-r87lk_odh-ai-gateway-infra(aaf4329d-4d56-4ce3-8eea-dbaf8da1c07a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-r87lk" podUID="aaf4329d-4d56-4ce3-8eea-dbaf8da1c07a"
Oct 30 13:45:16.784483 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:16.784461    4364 scope.go:122] "RemoveContainer" containerID="18c7c111add6a6bc2aca075bea9f74df72875750e55f81442c9c73838ac921a6"
Oct 30 13:45:16.905256 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:16.905228    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889465-r87lk"]
Oct 30 13:45:16.944440 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:16.944409    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889465-26rfr"
Oct 30 13:45:16.987583 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:16.987556    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889465-wdgj9"
Oct 30 13:45:17.029710 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:17.029644    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-cda0d9-74764bd4cf-l64mp" podUID="bd1b7e78-a663-4d45-a742-5994a0259a24" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:45:17.081366 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:17.081336    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0628c77-ff94-4a69-a0a0-af20a2b64e13-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0628c77-ff94-4a69-a0a0-af20a2b64e13-maas-api-service-ca\") pod \"c0628c77-ff94-4a69-a0a0-af20a2b64e13\" (UID: \"c0628c77-ff94-4a69-a0a0-af20a2b64e13\") "
Oct 30 13:45:17.081530 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:17.081437    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1453dcde-0c6d-427d-890b-8f3542a0775a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1453dcde-0c6d-427d-890b-8f3542a0775a-maas-api-service-ca\") pod \"1453dcde-0c6d-427d-890b-8f3542a0775a\" (UID: \"1453dcde-0c6d-427d-890b-8f3542a0775a\") "
Oct 30 13:45:17.081530 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:17.081493    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0628c77-ff94-4a69-a0a0-af20a2b64e13-kube-api-access-x7pw6\" (UniqueName: \"kubernetes.io/projected/c0628c77-ff94-4a69-a0a0-af20a2b64e13-kube-api-access-x7pw6\") pod \"c0628c77-ff94-4a69-a0a0-af20a2b64e13\" (UID: \"c0628c77-ff94-4a69-a0a0-af20a2b64e13\") "
Oct 30 13:45:17.081530 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:17.081520    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1453dcde-0c6d-427d-890b-8f3542a0775a-kube-api-access-tdvgr\" (UniqueName: \"kubernetes.io/projected/1453dcde-0c6d-427d-890b-8f3542a0775a-kube-api-access-tdvgr\") pod \"1453dcde-0c6d-427d-890b-8f3542a0775a\" (UID: \"1453dcde-0c6d-427d-890b-8f3542a0775a\") "
Oct 30 13:45:17.081769 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:17.081744    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0628c77-ff94-4a69-a0a0-af20a2b64e13-maas-api-service-ca" pod "c0628c77-ff94-4a69-a0a0-af20a2b64e13" (UID: "c0628c77-ff94-4a69-a0a0-af20a2b64e13"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:17.081859 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:17.081836    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1453dcde-0c6d-427d-890b-8f3542a0775a-maas-api-service-ca" pod "1453dcde-0c6d-427d-890b-8f3542a0775a" (UID: "1453dcde-0c6d-427d-890b-8f3542a0775a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:17.082022 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:17.081991    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1453dcde-0c6d-427d-890b-8f3542a0775a-maas-api-service-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:17.082022 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:17.082023    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0628c77-ff94-4a69-a0a0-af20a2b64e13-maas-api-service-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:17.083811 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:17.083789    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1453dcde-0c6d-427d-890b-8f3542a0775a-kube-api-access-tdvgr" pod "1453dcde-0c6d-427d-890b-8f3542a0775a" (UID: "1453dcde-0c6d-427d-890b-8f3542a0775a"). InnerVolumeSpecName "kube-api-access-tdvgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:17.083923 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:17.083878    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0628c77-ff94-4a69-a0a0-af20a2b64e13-kube-api-access-x7pw6" pod "c0628c77-ff94-4a69-a0a0-af20a2b64e13" (UID: "c0628c77-ff94-4a69-a0a0-af20a2b64e13"). InnerVolumeSpecName "kube-api-access-x7pw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:17.182891 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:17.182859    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7pw6\" (UniqueName: \"kubernetes.io/projected/c0628c77-ff94-4a69-a0a0-af20a2b64e13-kube-api-access-x7pw6\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:17.182891 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:17.182886    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdvgr\" (UniqueName: \"kubernetes.io/projected/1453dcde-0c6d-427d-890b-8f3542a0775a-kube-api-access-tdvgr\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:17.781879 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:17.781850    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889465-wdgj9"
Oct 30 13:45:17.782284 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:17.781882    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889465-wdgj9" event={"ID":"c0628c77-ff94-4a69-a0a0-af20a2b64e13","Type":"ContainerDied","Data":"12fcc8b478fc4d25d4643595894f44da88fe428f25cecafe3acb9226ee5a688a"}
Oct 30 13:45:17.782411 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:17.782373    4364 scope.go:122] "RemoveContainer" containerID="10012c141e8c91d4d36979c7895cec774cbda0bc34bab96d92fef8e616e4daf2"
Oct 30 13:45:17.784000 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:17.783861    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889465-26rfr" event={"ID":"1453dcde-0c6d-427d-890b-8f3542a0775a","Type":"ContainerDied","Data":"d748ee5fee9e4ef35f9d07302bdc5e7aed8b6e680a31b56f7c99d65c0eacc95a"}
Oct 30 13:45:17.784000 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:17.783957    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889465-26rfr"
Oct 30 13:45:17.804913 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:17.804888    4364 scope.go:122] "RemoveContainer" containerID="0c10f074630824f5d7f80b92e38a2cfff56c7124733957231cee492c5fab3e34"
Oct 30 13:45:17.864166 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:17.863943    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889465-26rfr"]
Oct 30 13:45:17.876939 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:17.876907    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e34fd5-29889465-26rfr"]
Oct 30 13:45:17.950771 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:17.950750    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-r87lk"
Oct 30 13:45:17.957459 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:17.957437    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889465-wdgj9"]
Oct 30 13:45:17.963836 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:17.963815    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889465-wdgj9"]
Oct 30 13:45:18.093072 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:18.093004    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaf4329d-4d56-4ce3-8eea-dbaf8da1c07a-kube-api-access-658gp\" (UniqueName: \"kubernetes.io/projected/aaf4329d-4d56-4ce3-8eea-dbaf8da1c07a-kube-api-access-658gp\") pod \"aaf4329d-4d56-4ce3-8eea-dbaf8da1c07a\" (UID: \"aaf4329d-4d56-4ce3-8eea-dbaf8da1c07a\") "
Oct 30 13:45:18.093431 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:18.093138    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaf4329d-4d56-4ce3-8eea-dbaf8da1c07a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaf4329d-4d56-4ce3-8eea-dbaf8da1c07a-maas-api-service-ca\") pod \"aaf4329d-4d56-4ce3-8eea-dbaf8da1c07a\" (UID: \"aaf4329d-4d56-4ce3-8eea-dbaf8da1c07a\") "
Oct 30 13:45:18.093505 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:18.093483    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaf4329d-4d56-4ce3-8eea-dbaf8da1c07a-maas-api-service-ca" pod "aaf4329d-4d56-4ce3-8eea-dbaf8da1c07a" (UID: "aaf4329d-4d56-4ce3-8eea-dbaf8da1c07a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:18.095094 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:18.095075    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaf4329d-4d56-4ce3-8eea-dbaf8da1c07a-kube-api-access-658gp" pod "aaf4329d-4d56-4ce3-8eea-dbaf8da1c07a" (UID: "aaf4329d-4d56-4ce3-8eea-dbaf8da1c07a"). InnerVolumeSpecName "kube-api-access-658gp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:18.194885 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:18.194852    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaf4329d-4d56-4ce3-8eea-dbaf8da1c07a-maas-api-service-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:18.194885 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:18.194880    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-658gp\" (UniqueName: \"kubernetes.io/projected/aaf4329d-4d56-4ce3-8eea-dbaf8da1c07a-kube-api-access-658gp\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:18.603248 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:18.603216    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-e34fd5-758756996-tl6mw"]
Oct 30 13:45:18.603434 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:18.603417    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e34fd5-758756996-tl6mw" podUID="638d73f1-8339-4b13-a492-2ab426a0ddd6" containerName="maas-api" containerID="cri-o://5cef9e3c86a225340ff0075624244a18a0290ed9a09086736ccfb65cb4ff3850" gracePeriod=30
Oct 30 13:45:18.812358 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:18.812321    4364 generic.go:356] "Generic (PLEG): container finished" podID="638d73f1-8339-4b13-a492-2ab426a0ddd6" containerID="5cef9e3c86a225340ff0075624244a18a0290ed9a09086736ccfb65cb4ff3850" exitCode=0
Oct 30 13:45:18.812529 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:18.812396    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e34fd5-758756996-tl6mw" event={"ID":"638d73f1-8339-4b13-a492-2ab426a0ddd6","Type":"ContainerDied","Data":"5cef9e3c86a225340ff0075624244a18a0290ed9a09086736ccfb65cb4ff3850"}
Oct 30 13:45:18.815758 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:18.815735    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-r87lk"
Oct 30 13:45:18.815972 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:18.815760    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-r87lk" event={"ID":"aaf4329d-4d56-4ce3-8eea-dbaf8da1c07a","Type":"ContainerDied","Data":"07fa84aedb853b5a78567795a682098ecf2f1c7b2acf346fea4e59e102f1de85"}
Oct 30 13:45:18.816083 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:18.816016    4364 scope.go:122] "RemoveContainer" containerID="67b7dbc6efa3742857cf633d53629a6335a671c461d320fc58ecb28aa654bf80"
Oct 30 13:45:18.867245 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:18.861620    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e34fd5-758756996-tl6mw"
Oct 30 13:45:18.876999 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:18.876959    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889465-r87lk"]
Oct 30 13:45:18.881194 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:18.881166    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889465-r87lk"]
Oct 30 13:45:18.970305 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:18.970272    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1453dcde-0c6d-427d-890b-8f3542a0775a" path="/var/lib/kubelet/pods/1453dcde-0c6d-427d-890b-8f3542a0775a/volumes"
Oct 30 13:45:18.970754 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:18.970737    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aaf4329d-4d56-4ce3-8eea-dbaf8da1c07a" path="/var/lib/kubelet/pods/aaf4329d-4d56-4ce3-8eea-dbaf8da1c07a/volumes"
Oct 30 13:45:18.971129 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:18.971117    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0628c77-ff94-4a69-a0a0-af20a2b64e13" path="/var/lib/kubelet/pods/c0628c77-ff94-4a69-a0a0-af20a2b64e13/volumes"
Oct 30 13:45:19.002365 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:19.002338    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/638d73f1-8339-4b13-a492-2ab426a0ddd6-kube-api-access-584zn\" (UniqueName: \"kubernetes.io/projected/638d73f1-8339-4b13-a492-2ab426a0ddd6-kube-api-access-584zn\") pod \"638d73f1-8339-4b13-a492-2ab426a0ddd6\" (UID: \"638d73f1-8339-4b13-a492-2ab426a0ddd6\") "
Oct 30 13:45:19.002501 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:19.002410    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/638d73f1-8339-4b13-a492-2ab426a0ddd6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/638d73f1-8339-4b13-a492-2ab426a0ddd6-openshift-service-ca\") pod \"638d73f1-8339-4b13-a492-2ab426a0ddd6\" (UID: \"638d73f1-8339-4b13-a492-2ab426a0ddd6\") "
Oct 30 13:45:19.002501 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:19.002469    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/638d73f1-8339-4b13-a492-2ab426a0ddd6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/638d73f1-8339-4b13-a492-2ab426a0ddd6-maas-api-tls\") pod \"638d73f1-8339-4b13-a492-2ab426a0ddd6\" (UID: \"638d73f1-8339-4b13-a492-2ab426a0ddd6\") "
Oct 30 13:45:19.002660 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:19.002549    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/638d73f1-8339-4b13-a492-2ab426a0ddd6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/638d73f1-8339-4b13-a492-2ab426a0ddd6-metrics-certs\") pod \"638d73f1-8339-4b13-a492-2ab426a0ddd6\" (UID: \"638d73f1-8339-4b13-a492-2ab426a0ddd6\") "
Oct 30 13:45:19.002854 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:19.002829    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/638d73f1-8339-4b13-a492-2ab426a0ddd6-openshift-service-ca" pod "638d73f1-8339-4b13-a492-2ab426a0ddd6" (UID: "638d73f1-8339-4b13-a492-2ab426a0ddd6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:19.002981 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:19.002965    4364 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/638d73f1-8339-4b13-a492-2ab426a0ddd6-openshift-service-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:19.004427 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:19.004400    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/638d73f1-8339-4b13-a492-2ab426a0ddd6-kube-api-access-584zn" pod "638d73f1-8339-4b13-a492-2ab426a0ddd6" (UID: "638d73f1-8339-4b13-a492-2ab426a0ddd6"). InnerVolumeSpecName "kube-api-access-584zn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:19.004541 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:19.004525    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/638d73f1-8339-4b13-a492-2ab426a0ddd6-maas-api-tls" pod "638d73f1-8339-4b13-a492-2ab426a0ddd6" (UID: "638d73f1-8339-4b13-a492-2ab426a0ddd6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:45:19.004666 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:19.004637    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/638d73f1-8339-4b13-a492-2ab426a0ddd6-metrics-certs" pod "638d73f1-8339-4b13-a492-2ab426a0ddd6" (UID: "638d73f1-8339-4b13-a492-2ab426a0ddd6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:45:19.104079 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:19.104042    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-584zn\" (UniqueName: \"kubernetes.io/projected/638d73f1-8339-4b13-a492-2ab426a0ddd6-kube-api-access-584zn\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:19.104079 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:19.104079    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/638d73f1-8339-4b13-a492-2ab426a0ddd6-maas-api-tls\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:19.104493 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:19.104093    4364 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/638d73f1-8339-4b13-a492-2ab426a0ddd6-metrics-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:19.271826 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:19.271742    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-52a649-65c5fbbff9-n6rqr" podUID="22d0f282-215c-45d9-9fbb-15190b9ebd10" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:45:19.824802 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:19.824772    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e34fd5-758756996-tl6mw"
Oct 30 13:45:19.824948 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:19.824809    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e34fd5-758756996-tl6mw" event={"ID":"638d73f1-8339-4b13-a492-2ab426a0ddd6","Type":"ContainerDied","Data":"abf86361b0102b39852fdaaef2963c717f917be1eae99447cbc8c4eeda976045"}
Oct 30 13:45:19.824948 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:19.824882    4364 scope.go:122] "RemoveContainer" containerID="5cef9e3c86a225340ff0075624244a18a0290ed9a09086736ccfb65cb4ff3850"
Oct 30 13:45:19.886133 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:19.886102    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-e34fd5-758756996-tl6mw"]
Oct 30 13:45:19.900538 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:19.900512    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-e34fd5-758756996-tl6mw"]
Oct 30 13:45:20.038759 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:20.038733    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-cda0d9-74764bd4cf-l64mp"
Oct 30 13:45:20.213791 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:20.213765    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd1b7e78-a663-4d45-a742-5994a0259a24-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd1b7e78-a663-4d45-a742-5994a0259a24-plugins-config-volume\") pod \"bd1b7e78-a663-4d45-a742-5994a0259a24\" (UID: \"bd1b7e78-a663-4d45-a742-5994a0259a24\") "
Oct 30 13:45:20.214189 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:20.213839    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd1b7e78-a663-4d45-a742-5994a0259a24-kube-api-access-kjxrf\" (UniqueName: \"kubernetes.io/projected/bd1b7e78-a663-4d45-a742-5994a0259a24-kube-api-access-kjxrf\") pod \"bd1b7e78-a663-4d45-a742-5994a0259a24\" (UID: \"bd1b7e78-a663-4d45-a742-5994a0259a24\") "
Oct 30 13:45:20.214189 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:20.214098    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd1b7e78-a663-4d45-a742-5994a0259a24-plugins-config-volume" pod "bd1b7e78-a663-4d45-a742-5994a0259a24" (UID: "bd1b7e78-a663-4d45-a742-5994a0259a24"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:20.216205 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:20.216179    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd1b7e78-a663-4d45-a742-5994a0259a24-kube-api-access-kjxrf" pod "bd1b7e78-a663-4d45-a742-5994a0259a24" (UID: "bd1b7e78-a663-4d45-a742-5994a0259a24"). InnerVolumeSpecName "kube-api-access-kjxrf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:20.314474 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:20.314448    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjxrf\" (UniqueName: \"kubernetes.io/projected/bd1b7e78-a663-4d45-a742-5994a0259a24-kube-api-access-kjxrf\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:20.314474 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:20.314472    4364 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd1b7e78-a663-4d45-a742-5994a0259a24-plugins-config-volume\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:20.834312 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:20.834269    4364 generic.go:356] "Generic (PLEG): container finished" podID="bd1b7e78-a663-4d45-a742-5994a0259a24" containerID="e7b248d8c1f66460e433ea637fdb1a9e085e5d7c8ed4532d5cf5c6b9603a9ce7" exitCode=0
Oct 30 13:45:20.834526 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:20.834321    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-cda0d9-74764bd4cf-l64mp"
Oct 30 13:45:20.834526 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:20.834352    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-cda0d9-74764bd4cf-l64mp" event={"ID":"bd1b7e78-a663-4d45-a742-5994a0259a24","Type":"ContainerDied","Data":"e7b248d8c1f66460e433ea637fdb1a9e085e5d7c8ed4532d5cf5c6b9603a9ce7"}
Oct 30 13:45:20.834526 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:20.834414    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-cda0d9-74764bd4cf-l64mp" event={"ID":"bd1b7e78-a663-4d45-a742-5994a0259a24","Type":"ContainerDied","Data":"3b761fc92258613301f1ff16dbe170f92a72a7a5da819268e44c87bccc7a7b8d"}
Oct 30 13:45:20.834526 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:20.834437    4364 scope.go:122] "RemoveContainer" containerID="e7b248d8c1f66460e433ea637fdb1a9e085e5d7c8ed4532d5cf5c6b9603a9ce7"
Oct 30 13:45:20.844570 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:20.844542    4364 scope.go:122] "RemoveContainer" containerID="e7b248d8c1f66460e433ea637fdb1a9e085e5d7c8ed4532d5cf5c6b9603a9ce7"
Oct 30 13:45:20.845001 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:45:20.844978    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7b248d8c1f66460e433ea637fdb1a9e085e5d7c8ed4532d5cf5c6b9603a9ce7\": container with ID starting with e7b248d8c1f66460e433ea637fdb1a9e085e5d7c8ed4532d5cf5c6b9603a9ce7 not found: ID does not exist" containerID="e7b248d8c1f66460e433ea637fdb1a9e085e5d7c8ed4532d5cf5c6b9603a9ce7"
Oct 30 13:45:20.845117 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:20.845010    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7b248d8c1f66460e433ea637fdb1a9e085e5d7c8ed4532d5cf5c6b9603a9ce7"} err="failed to get container status \"e7b248d8c1f66460e433ea637fdb1a9e085e5d7c8ed4532d5cf5c6b9603a9ce7\": rpc error: code = NotFound desc = could not find container \"e7b248d8c1f66460e433ea637fdb1a9e085e5d7c8ed4532d5cf5c6b9603a9ce7\": container with ID starting with e7b248d8c1f66460e433ea637fdb1a9e085e5d7c8ed4532d5cf5c6b9603a9ce7 not found: ID does not exist"
Oct 30 13:45:20.893213 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:20.893181    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-cda0d9-74764bd4cf-l64mp"]
Oct 30 13:45:20.895490 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:20.895469    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-cda0d9-74764bd4cf-l64mp"]
Oct 30 13:45:20.969656 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:20.969625    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="638d73f1-8339-4b13-a492-2ab426a0ddd6" path="/var/lib/kubelet/pods/638d73f1-8339-4b13-a492-2ab426a0ddd6/volumes"
Oct 30 13:45:20.970031 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:20.970018    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd1b7e78-a663-4d45-a742-5994a0259a24" path="/var/lib/kubelet/pods/bd1b7e78-a663-4d45-a742-5994a0259a24/volumes"
Oct 30 13:45:22.691126 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:22.691091    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-ee7d65-58cd4c5bbb-6tv8w"]
Oct 30 13:45:22.717411 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:22.717368    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-ee7d65-5b75dc6cc5-sv4rn"]
Oct 30 13:45:25.221662 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:25.221616    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9fa5c55-sndgt"]
Oct 30 13:45:25.222034 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:25.222018    4364 cpu_manager.go:414] "containerMap: removing container" podUID="aaf4329d-4d56-4ce3-8eea-dbaf8da1c07a" containerName="cleanup"
Oct 30 13:45:25.222034 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:25.222030    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaf4329d-4d56-4ce3-8eea-dbaf8da1c07a" containerName="cleanup"
Oct 30 13:45:25.222135 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:25.222040    4364 cpu_manager.go:414] "containerMap: removing container" podUID="638d73f1-8339-4b13-a492-2ab426a0ddd6" containerName="maas-api"
Oct 30 13:45:25.222135 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:25.222045    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="638d73f1-8339-4b13-a492-2ab426a0ddd6" containerName="maas-api"
Oct 30 13:45:25.222135 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:25.222057    4364 cpu_manager.go:414] "containerMap: removing container" podUID="1453dcde-0c6d-427d-890b-8f3542a0775a" containerName="cleanup"
Oct 30 13:45:25.222135 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:25.222062    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1453dcde-0c6d-427d-890b-8f3542a0775a" containerName="cleanup"
Oct 30 13:45:25.222135 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:25.222071    4364 cpu_manager.go:414] "containerMap: removing container" podUID="bc5caf69-a649-47e5-8910-9b8f331dfd4c" containerName="main"
Oct 30 13:45:25.222135 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:25.222075    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc5caf69-a649-47e5-8910-9b8f331dfd4c" containerName="main"
Oct 30 13:45:25.222135 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:25.222085    4364 cpu_manager.go:414] "containerMap: removing container" podUID="1453dcde-0c6d-427d-890b-8f3542a0775a" containerName="cleanup"
Oct 30 13:45:25.222135 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:25.222090    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1453dcde-0c6d-427d-890b-8f3542a0775a" containerName="cleanup"
Oct 30 13:45:25.222135 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:25.222104    4364 cpu_manager.go:414] "containerMap: removing container" podUID="d41ac9e5-eeed-4e07-a7fa-05d3260630b6" containerName="main"
Oct 30 13:45:25.222135 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:25.222109    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d41ac9e5-eeed-4e07-a7fa-05d3260630b6" containerName="main"
Oct 30 13:45:25.222135 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:25.222117    4364 cpu_manager.go:414] "containerMap: removing container" podUID="93362c37-0fc6-48b2-85b8-ee62492ba31f" containerName="revoke-keys"
Oct 30 13:45:25.222135 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:25.222121    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93362c37-0fc6-48b2-85b8-ee62492ba31f" containerName="revoke-keys"
Oct 30 13:45:25.222135 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:25.222130    4364 cpu_manager.go:414] "containerMap: removing container" podUID="a568f1ce-602d-4069-a41c-9fe1a7635c5a" containerName="main"
Oct 30 13:45:25.222135 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:25.222134    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a568f1ce-602d-4069-a41c-9fe1a7635c5a" containerName="main"
Oct 30 13:45:25.222135 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:25.222141    4364 cpu_manager.go:414] "containerMap: removing container" podUID="bd1b7e78-a663-4d45-a742-5994a0259a24" containerName="payload-pre-processing"
Oct 30 13:45:25.222642 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:25.222145    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd1b7e78-a663-4d45-a742-5994a0259a24" containerName="payload-pre-processing"
Oct 30 13:45:25.222642 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:25.222153    4364 cpu_manager.go:414] "containerMap: removing container" podUID="c0628c77-ff94-4a69-a0a0-af20a2b64e13" containerName="cleanup"
Oct 30 13:45:25.222642 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:25.222157    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0628c77-ff94-4a69-a0a0-af20a2b64e13" containerName="cleanup"
Oct 30 13:45:25.222642 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:25.222168    4364 cpu_manager.go:414] "containerMap: removing container" podUID="aaf4329d-4d56-4ce3-8eea-dbaf8da1c07a" containerName="cleanup"
Oct 30 13:45:25.222642 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:25.222173    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaf4329d-4d56-4ce3-8eea-dbaf8da1c07a" containerName="cleanup"
Oct 30 13:45:25.222642 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:25.222179    4364 cpu_manager.go:414] "containerMap: removing container" podUID="c0628c77-ff94-4a69-a0a0-af20a2b64e13" containerName="cleanup"
Oct 30 13:45:25.222642 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:25.222185    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0628c77-ff94-4a69-a0a0-af20a2b64e13" containerName="cleanup"
Oct 30 13:45:25.222642 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:25.222247    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd1b7e78-a663-4d45-a742-5994a0259a24" containerName="payload-pre-processing"
Oct 30 13:45:25.222642 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:25.222257    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0628c77-ff94-4a69-a0a0-af20a2b64e13" containerName="cleanup"
Oct 30 13:45:25.222642 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:25.222264    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0628c77-ff94-4a69-a0a0-af20a2b64e13" containerName="cleanup"
Oct 30 13:45:25.222642 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:25.222269    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaf4329d-4d56-4ce3-8eea-dbaf8da1c07a" containerName="cleanup"
Oct 30 13:45:25.222642 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:25.222275    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0628c77-ff94-4a69-a0a0-af20a2b64e13" containerName="cleanup"
Oct 30 13:45:25.222642 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:25.222281    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaf4329d-4d56-4ce3-8eea-dbaf8da1c07a" containerName="cleanup"
Oct 30 13:45:25.222642 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:25.222286    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="638d73f1-8339-4b13-a492-2ab426a0ddd6" containerName="maas-api"
Oct 30 13:45:25.222642 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:25.222292    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="1453dcde-0c6d-427d-890b-8f3542a0775a" containerName="cleanup"
Oct 30 13:45:25.222642 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:25.222296    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc5caf69-a649-47e5-8910-9b8f331dfd4c" containerName="main"
Oct 30 13:45:25.222642 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:25.222301    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="1453dcde-0c6d-427d-890b-8f3542a0775a" containerName="cleanup"
Oct 30 13:45:25.222642 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:25.222306    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaf4329d-4d56-4ce3-8eea-dbaf8da1c07a" containerName="cleanup"
Oct 30 13:45:25.222642 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:25.222312    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="d41ac9e5-eeed-4e07-a7fa-05d3260630b6" containerName="main"
Oct 30 13:45:25.222642 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:25.222317    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="93362c37-0fc6-48b2-85b8-ee62492ba31f" containerName="revoke-keys"
Oct 30 13:45:25.222642 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:25.222323    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="a568f1ce-602d-4069-a41c-9fe1a7635c5a" containerName="main"
Oct 30 13:45:25.222642 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:25.222327    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="1453dcde-0c6d-427d-890b-8f3542a0775a" containerName="cleanup"
Oct 30 13:45:25.226933 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:25.226914    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9fa5c55-sndgt"
Oct 30 13:45:25.252077 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:25.252047    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9fa5c55-sndgt"]
Oct 30 13:45:25.262213 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:25.262187    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:45:25.262340 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:25.262240    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:45:25.262340 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:25.262255    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:45:25.262340 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:25.262259    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\""
Oct 30 13:45:25.365013 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:25.364984    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzw8z\" (UniqueName: \"kubernetes.io/projected/6e561630-bbec-40f3-a5c3-143f153cc443-kube-api-access-vzw8z\") pod \"maas-api-revoke-sub-d9fa5c55-sndgt\" (UID: \"6e561630-bbec-40f3-a5c3-143f153cc443\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9fa5c55-sndgt"
Oct 30 13:45:25.365158 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:25.365023    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e561630-bbec-40f3-a5c3-143f153cc443-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9fa5c55-sndgt\" (UID: \"6e561630-bbec-40f3-a5c3-143f153cc443\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9fa5c55-sndgt"
Oct 30 13:45:25.365158 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:25.365057    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e561630-bbec-40f3-a5c3-143f153cc443-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9fa5c55-sndgt\" (UID: \"6e561630-bbec-40f3-a5c3-143f153cc443\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9fa5c55-sndgt"
Oct 30 13:45:25.466253 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:25.466215    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzw8z\" (UniqueName: \"kubernetes.io/projected/6e561630-bbec-40f3-a5c3-143f153cc443-kube-api-access-vzw8z\") pod \"maas-api-revoke-sub-d9fa5c55-sndgt\" (UID: \"6e561630-bbec-40f3-a5c3-143f153cc443\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9fa5c55-sndgt"
Oct 30 13:45:25.466420 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:25.466271    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e561630-bbec-40f3-a5c3-143f153cc443-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9fa5c55-sndgt\" (UID: \"6e561630-bbec-40f3-a5c3-143f153cc443\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9fa5c55-sndgt"
Oct 30 13:45:25.466420 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:25.466322    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e561630-bbec-40f3-a5c3-143f153cc443-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9fa5c55-sndgt\" (UID: \"6e561630-bbec-40f3-a5c3-143f153cc443\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9fa5c55-sndgt"
Oct 30 13:45:25.467065 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:25.467038    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e561630-bbec-40f3-a5c3-143f153cc443-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9fa5c55-sndgt\" (UID: \"6e561630-bbec-40f3-a5c3-143f153cc443\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9fa5c55-sndgt"
Oct 30 13:45:25.468611 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:25.468582    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e561630-bbec-40f3-a5c3-143f153cc443-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9fa5c55-sndgt\" (UID: \"6e561630-bbec-40f3-a5c3-143f153cc443\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9fa5c55-sndgt"
Oct 30 13:45:25.491726 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:25.491672    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzw8z\" (UniqueName: \"kubernetes.io/projected/6e561630-bbec-40f3-a5c3-143f153cc443-kube-api-access-vzw8z\") pod \"maas-api-revoke-sub-d9fa5c55-sndgt\" (UID: \"6e561630-bbec-40f3-a5c3-143f153cc443\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9fa5c55-sndgt"
Oct 30 13:45:25.539513 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:25.539489    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9fa5c55-sndgt"
Oct 30 13:45:25.819111 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:25.819087    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-52a649-65c5fbbff9-n6rqr"
Oct 30 13:45:25.844166 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:25.844140    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9fa5c55-sndgt"]
Oct 30 13:45:25.844298 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:45:25.844277    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e561630_bbec_40f3_a5c3_143f153cc443.slice/crio-deeff830b6097e5902a005571af4f2e7bf91c44b45dc3c8f98c2c85cfeac58a6 WatchSource:0}: Error finding container deeff830b6097e5902a005571af4f2e7bf91c44b45dc3c8f98c2c85cfeac58a6: Status 404 returned error can't find the container with id deeff830b6097e5902a005571af4f2e7bf91c44b45dc3c8f98c2c85cfeac58a6
Oct 30 13:45:25.872053 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:25.872025    4364 generic.go:356] "Generic (PLEG): container finished" podID="22d0f282-215c-45d9-9fbb-15190b9ebd10" containerID="70236c8c1dd87912d3fa93edd91532b2c1d9d7ac2b49a60c725d3ac8c3ba5e54" exitCode=0
Oct 30 13:45:25.872302 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:25.872283    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-52a649-65c5fbbff9-n6rqr"
Oct 30 13:45:25.872526 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:25.872505    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-52a649-65c5fbbff9-n6rqr" event={"ID":"22d0f282-215c-45d9-9fbb-15190b9ebd10","Type":"ContainerDied","Data":"70236c8c1dd87912d3fa93edd91532b2c1d9d7ac2b49a60c725d3ac8c3ba5e54"}
Oct 30 13:45:25.872626 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:25.872560    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-52a649-65c5fbbff9-n6rqr" event={"ID":"22d0f282-215c-45d9-9fbb-15190b9ebd10","Type":"ContainerDied","Data":"f672930499f4e4635e653d77df06bbdae3a07332debde0fae786c72578b4b08e"}
Oct 30 13:45:25.872626 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:25.872611    4364 scope.go:122] "RemoveContainer" containerID="70236c8c1dd87912d3fa93edd91532b2c1d9d7ac2b49a60c725d3ac8c3ba5e54"
Oct 30 13:45:25.874656 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:25.874626    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9fa5c55-sndgt" event={"ID":"6e561630-bbec-40f3-a5c3-143f153cc443","Type":"ContainerStarted","Data":"deeff830b6097e5902a005571af4f2e7bf91c44b45dc3c8f98c2c85cfeac58a6"}
Oct 30 13:45:25.883309 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:25.883287    4364 scope.go:122] "RemoveContainer" containerID="70236c8c1dd87912d3fa93edd91532b2c1d9d7ac2b49a60c725d3ac8c3ba5e54"
Oct 30 13:45:25.883622 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:45:25.883577    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70236c8c1dd87912d3fa93edd91532b2c1d9d7ac2b49a60c725d3ac8c3ba5e54\": container with ID starting with 70236c8c1dd87912d3fa93edd91532b2c1d9d7ac2b49a60c725d3ac8c3ba5e54 not found: ID does not exist" containerID="70236c8c1dd87912d3fa93edd91532b2c1d9d7ac2b49a60c725d3ac8c3ba5e54"
Oct 30 13:45:25.883704 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:25.883635    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70236c8c1dd87912d3fa93edd91532b2c1d9d7ac2b49a60c725d3ac8c3ba5e54"} err="failed to get container status \"70236c8c1dd87912d3fa93edd91532b2c1d9d7ac2b49a60c725d3ac8c3ba5e54\": rpc error: code = NotFound desc = could not find container \"70236c8c1dd87912d3fa93edd91532b2c1d9d7ac2b49a60c725d3ac8c3ba5e54\": container with ID starting with 70236c8c1dd87912d3fa93edd91532b2c1d9d7ac2b49a60c725d3ac8c3ba5e54 not found: ID does not exist"
Oct 30 13:45:25.969903 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:25.969880    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22d0f282-215c-45d9-9fbb-15190b9ebd10-kube-api-access-hmghd\" (UniqueName: \"kubernetes.io/projected/22d0f282-215c-45d9-9fbb-15190b9ebd10-kube-api-access-hmghd\") pod \"22d0f282-215c-45d9-9fbb-15190b9ebd10\" (UID: \"22d0f282-215c-45d9-9fbb-15190b9ebd10\") "
Oct 30 13:45:25.970038 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:25.970024    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22d0f282-215c-45d9-9fbb-15190b9ebd10-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22d0f282-215c-45d9-9fbb-15190b9ebd10-plugins-config-volume\") pod \"22d0f282-215c-45d9-9fbb-15190b9ebd10\" (UID: \"22d0f282-215c-45d9-9fbb-15190b9ebd10\") "
Oct 30 13:45:25.970394 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:25.970369    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22d0f282-215c-45d9-9fbb-15190b9ebd10-plugins-config-volume" pod "22d0f282-215c-45d9-9fbb-15190b9ebd10" (UID: "22d0f282-215c-45d9-9fbb-15190b9ebd10"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:25.971828 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:25.971808    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22d0f282-215c-45d9-9fbb-15190b9ebd10-kube-api-access-hmghd" pod "22d0f282-215c-45d9-9fbb-15190b9ebd10" (UID: "22d0f282-215c-45d9-9fbb-15190b9ebd10"). InnerVolumeSpecName "kube-api-access-hmghd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:26.071562 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:26.071507    4364 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22d0f282-215c-45d9-9fbb-15190b9ebd10-plugins-config-volume\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:26.071562 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:26.071528    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmghd\" (UniqueName: \"kubernetes.io/projected/22d0f282-215c-45d9-9fbb-15190b9ebd10-kube-api-access-hmghd\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:26.224469 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:26.224442    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-52a649-65c5fbbff9-n6rqr"]
Oct 30 13:45:26.227231 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:26.227209    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-52a649-65c5fbbff9-n6rqr"]
Oct 30 13:45:26.882860 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:26.882819    4364 generic.go:356] "Generic (PLEG): container finished" podID="6e561630-bbec-40f3-a5c3-143f153cc443" containerID="bb11a36c05ed116a3f64048a5436fab4282d194913ef93584d3936c5cca0ce5c" exitCode=0
Oct 30 13:45:26.883036 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:26.882864    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9fa5c55-sndgt" event={"ID":"6e561630-bbec-40f3-a5c3-143f153cc443","Type":"ContainerDied","Data":"bb11a36c05ed116a3f64048a5436fab4282d194913ef93584d3936c5cca0ce5c"}
Oct 30 13:45:26.977945 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:26.977913    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22d0f282-215c-45d9-9fbb-15190b9ebd10" path="/var/lib/kubelet/pods/22d0f282-215c-45d9-9fbb-15190b9ebd10/volumes"
Oct 30 13:45:27.030663 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:27.030612    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26"]
Oct 30 13:45:27.030948 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:27.030901    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26" podUID="c4b120bf-6e2b-46b5-bda8-76f8963697fb" containerName="istio-proxy" containerID="cri-o://91e066f39e1a5e9bb338794da9c823c80ea37572f64bd602be992496a3b56b29" gracePeriod=30
Oct 30 13:45:27.691890 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:27.691838    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-ee7d65-58cd4c5bbb-6tv8w" podUID="4dc7d59a-ca00-49e0-8f9c-eabe22d71c17" containerName="payload-processing" containerID="cri-o://bbb2700a8683299448b69f70ad21143d3d2a0ddec715b7b006760f597b2b48be" gracePeriod=25
Oct 30 13:45:27.718514 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:27.718442    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ee7d65-5b75dc6cc5-sv4rn" podUID="9cbfe471-67e7-43cd-9524-771994bbdb14" containerName="payload-pre-processing" containerID="cri-o://cb5c4578d610d53beff85a1a3201787e5e2ac75254cdc8e63b5b1d469935f87d" gracePeriod=25
Oct 30 13:45:28.023459 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:28.023429    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9fa5c55-sndgt"
Oct 30 13:45:28.090130 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:28.090100    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e561630-bbec-40f3-a5c3-143f153cc443-kube-api-access-vzw8z\" (UniqueName: \"kubernetes.io/projected/6e561630-bbec-40f3-a5c3-143f153cc443-kube-api-access-vzw8z\") pod \"6e561630-bbec-40f3-a5c3-143f153cc443\" (UID: \"6e561630-bbec-40f3-a5c3-143f153cc443\") "
Oct 30 13:45:28.090329 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:28.090176    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e561630-bbec-40f3-a5c3-143f153cc443-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e561630-bbec-40f3-a5c3-143f153cc443-maas-api-openshift-ca\") pod \"6e561630-bbec-40f3-a5c3-143f153cc443\" (UID: \"6e561630-bbec-40f3-a5c3-143f153cc443\") "
Oct 30 13:45:28.090329 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:28.090217    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e561630-bbec-40f3-a5c3-143f153cc443-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e561630-bbec-40f3-a5c3-143f153cc443-maas-api-xks-ca\") pod \"6e561630-bbec-40f3-a5c3-143f153cc443\" (UID: \"6e561630-bbec-40f3-a5c3-143f153cc443\") "
Oct 30 13:45:28.090611 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:28.090569    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e561630-bbec-40f3-a5c3-143f153cc443-maas-api-openshift-ca" pod "6e561630-bbec-40f3-a5c3-143f153cc443" (UID: "6e561630-bbec-40f3-a5c3-143f153cc443"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:28.092423 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:28.092394    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e561630-bbec-40f3-a5c3-143f153cc443-maas-api-xks-ca" pod "6e561630-bbec-40f3-a5c3-143f153cc443" (UID: "6e561630-bbec-40f3-a5c3-143f153cc443"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:45:28.092528 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:28.092509    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e561630-bbec-40f3-a5c3-143f153cc443-kube-api-access-vzw8z" pod "6e561630-bbec-40f3-a5c3-143f153cc443" (UID: "6e561630-bbec-40f3-a5c3-143f153cc443"). InnerVolumeSpecName "kube-api-access-vzw8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:28.191952 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:28.191924    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e561630-bbec-40f3-a5c3-143f153cc443-maas-api-xks-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:28.191952 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:28.191952    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzw8z\" (UniqueName: \"kubernetes.io/projected/6e561630-bbec-40f3-a5c3-143f153cc443-kube-api-access-vzw8z\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:28.192112 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:28.191961    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e561630-bbec-40f3-a5c3-143f153cc443-maas-api-openshift-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:28.608561 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:28.608525    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp"]
Oct 30 13:45:28.608832 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:28.608810    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp" podUID="faba8da0-1b85-49fc-aa90-f860879f40a5" containerName="istio-proxy" containerID="cri-o://07f9855fd3f5b075c58f6437479ef56e3ab9421cc19c3ecbfac80e78279ea709" gracePeriod=30
Oct 30 13:45:28.899288 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:28.899250    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9fa5c55-sndgt" event={"ID":"6e561630-bbec-40f3-a5c3-143f153cc443","Type":"ContainerDied","Data":"deeff830b6097e5902a005571af4f2e7bf91c44b45dc3c8f98c2c85cfeac58a6"}
Oct 30 13:45:28.899732 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:28.899307    4364 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="deeff830b6097e5902a005571af4f2e7bf91c44b45dc3c8f98c2c85cfeac58a6"
Oct 30 13:45:28.899732 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:28.899271    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9fa5c55-sndgt"
Oct 30 13:45:30.158876 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:30.158839    4364 patch_prober.go:29] interesting pod/e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:45:30.159271 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:30.158895    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26" podUID="c4b120bf-6e2b-46b5-bda8-76f8963697fb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:45:31.283781 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:31.283743    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ee7d65-5b75dc6cc5-sv4rn" podUID="9cbfe471-67e7-43cd-9524-771994bbdb14" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:45:31.284235 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:31.283759    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-ee7d65-58cd4c5bbb-6tv8w" podUID="4dc7d59a-ca00-49e0-8f9c-eabe22d71c17" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:45:32.293490 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:32.293432    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26"
Oct 30 13:45:32.331740 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:32.331685    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4b120bf-6e2b-46b5-bda8-76f8963697fb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4b120bf-6e2b-46b5-bda8-76f8963697fb-credential-socket\") pod \"c4b120bf-6e2b-46b5-bda8-76f8963697fb\" (UID: \"c4b120bf-6e2b-46b5-bda8-76f8963697fb\") "
Oct 30 13:45:32.331906 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:32.331765    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4b120bf-6e2b-46b5-bda8-76f8963697fb-kube-api-access-8blw8\" (UniqueName: \"kubernetes.io/projected/c4b120bf-6e2b-46b5-bda8-76f8963697fb-kube-api-access-8blw8\") pod \"c4b120bf-6e2b-46b5-bda8-76f8963697fb\" (UID: \"c4b120bf-6e2b-46b5-bda8-76f8963697fb\") "
Oct 30 13:45:32.331906 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:32.331786    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c4b120bf-6e2b-46b5-bda8-76f8963697fb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4b120bf-6e2b-46b5-bda8-76f8963697fb-istio-podinfo\") pod \"c4b120bf-6e2b-46b5-bda8-76f8963697fb\" (UID: \"c4b120bf-6e2b-46b5-bda8-76f8963697fb\") "
Oct 30 13:45:32.331906 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:32.331805    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4b120bf-6e2b-46b5-bda8-76f8963697fb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4b120bf-6e2b-46b5-bda8-76f8963697fb-workload-socket\") pod \"c4b120bf-6e2b-46b5-bda8-76f8963697fb\" (UID: \"c4b120bf-6e2b-46b5-bda8-76f8963697fb\") "
Oct 30 13:45:32.331906 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:32.331852    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4b120bf-6e2b-46b5-bda8-76f8963697fb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4b120bf-6e2b-46b5-bda8-76f8963697fb-istiod-ca-cert\") pod \"c4b120bf-6e2b-46b5-bda8-76f8963697fb\" (UID: \"c4b120bf-6e2b-46b5-bda8-76f8963697fb\") "
Oct 30 13:45:32.332138 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:32.331972    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4b120bf-6e2b-46b5-bda8-76f8963697fb-credential-socket" pod "c4b120bf-6e2b-46b5-bda8-76f8963697fb" (UID: "c4b120bf-6e2b-46b5-bda8-76f8963697fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:45:32.332138 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:32.331987    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4b120bf-6e2b-46b5-bda8-76f8963697fb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4b120bf-6e2b-46b5-bda8-76f8963697fb-istio-data\") pod \"c4b120bf-6e2b-46b5-bda8-76f8963697fb\" (UID: \"c4b120bf-6e2b-46b5-bda8-76f8963697fb\") "
Oct 30 13:45:32.332138 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:32.332052    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4b120bf-6e2b-46b5-bda8-76f8963697fb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4b120bf-6e2b-46b5-bda8-76f8963697fb-workload-certs\") pod \"c4b120bf-6e2b-46b5-bda8-76f8963697fb\" (UID: \"c4b120bf-6e2b-46b5-bda8-76f8963697fb\") "
Oct 30 13:45:32.332138 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:32.332089    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4b120bf-6e2b-46b5-bda8-76f8963697fb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4b120bf-6e2b-46b5-bda8-76f8963697fb-istio-envoy\") pod \"c4b120bf-6e2b-46b5-bda8-76f8963697fb\" (UID: \"c4b120bf-6e2b-46b5-bda8-76f8963697fb\") "
Oct 30 13:45:32.332366 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:32.332107    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4b120bf-6e2b-46b5-bda8-76f8963697fb-workload-socket" pod "c4b120bf-6e2b-46b5-bda8-76f8963697fb" (UID: "c4b120bf-6e2b-46b5-bda8-76f8963697fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:45:32.332366 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:32.332117    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4b120bf-6e2b-46b5-bda8-76f8963697fb-istio-token\" (UniqueName: \"kubernetes.io/projected/c4b120bf-6e2b-46b5-bda8-76f8963697fb-istio-token\") pod \"c4b120bf-6e2b-46b5-bda8-76f8963697fb\" (UID: \"c4b120bf-6e2b-46b5-bda8-76f8963697fb\") "
Oct 30 13:45:32.332366 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:32.332206    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4b120bf-6e2b-46b5-bda8-76f8963697fb-istio-data" pod "c4b120bf-6e2b-46b5-bda8-76f8963697fb" (UID: "c4b120bf-6e2b-46b5-bda8-76f8963697fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:45:32.332366 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:32.332250    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4b120bf-6e2b-46b5-bda8-76f8963697fb-istiod-ca-cert" pod "c4b120bf-6e2b-46b5-bda8-76f8963697fb" (UID: "c4b120bf-6e2b-46b5-bda8-76f8963697fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:32.332366 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:32.332331    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4b120bf-6e2b-46b5-bda8-76f8963697fb-workload-certs" pod "c4b120bf-6e2b-46b5-bda8-76f8963697fb" (UID: "c4b120bf-6e2b-46b5-bda8-76f8963697fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:45:32.332662 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:32.332543    4364 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4b120bf-6e2b-46b5-bda8-76f8963697fb-credential-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:32.332662 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:32.332559    4364 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4b120bf-6e2b-46b5-bda8-76f8963697fb-workload-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:32.332662 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:32.332568    4364 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4b120bf-6e2b-46b5-bda8-76f8963697fb-istiod-ca-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:32.332662 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:32.332576    4364 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4b120bf-6e2b-46b5-bda8-76f8963697fb-istio-data\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:32.332662 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:32.332583    4364 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4b120bf-6e2b-46b5-bda8-76f8963697fb-workload-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:32.334156 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:32.334129    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4b120bf-6e2b-46b5-bda8-76f8963697fb-istio-envoy" pod "c4b120bf-6e2b-46b5-bda8-76f8963697fb" (UID: "c4b120bf-6e2b-46b5-bda8-76f8963697fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:45:32.334260 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:32.334196    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4b120bf-6e2b-46b5-bda8-76f8963697fb-kube-api-access-8blw8" pod "c4b120bf-6e2b-46b5-bda8-76f8963697fb" (UID: "c4b120bf-6e2b-46b5-bda8-76f8963697fb"). InnerVolumeSpecName "kube-api-access-8blw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:32.334456 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:32.334430    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4b120bf-6e2b-46b5-bda8-76f8963697fb-istio-token" pod "c4b120bf-6e2b-46b5-bda8-76f8963697fb" (UID: "c4b120bf-6e2b-46b5-bda8-76f8963697fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:32.334456 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:32.334438    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c4b120bf-6e2b-46b5-bda8-76f8963697fb-istio-podinfo" pod "c4b120bf-6e2b-46b5-bda8-76f8963697fb" (UID: "c4b120bf-6e2b-46b5-bda8-76f8963697fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:45:32.433478 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:32.433447    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8blw8\" (UniqueName: \"kubernetes.io/projected/c4b120bf-6e2b-46b5-bda8-76f8963697fb-kube-api-access-8blw8\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:32.433478 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:32.433477    4364 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4b120bf-6e2b-46b5-bda8-76f8963697fb-istio-podinfo\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:32.433478 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:32.433487    4364 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4b120bf-6e2b-46b5-bda8-76f8963697fb-istio-envoy\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:32.433762 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:32.433494    4364 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4b120bf-6e2b-46b5-bda8-76f8963697fb-istio-token\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:32.923155 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:32.923118    4364 generic.go:356] "Generic (PLEG): container finished" podID="c4b120bf-6e2b-46b5-bda8-76f8963697fb" containerID="91e066f39e1a5e9bb338794da9c823c80ea37572f64bd602be992496a3b56b29" exitCode=0
Oct 30 13:45:32.923358 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:32.923190    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26"
Oct 30 13:45:32.923358 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:32.923219    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26" event={"ID":"c4b120bf-6e2b-46b5-bda8-76f8963697fb","Type":"ContainerDied","Data":"91e066f39e1a5e9bb338794da9c823c80ea37572f64bd602be992496a3b56b29"}
Oct 30 13:45:32.923358 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:32.923338    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26" event={"ID":"c4b120bf-6e2b-46b5-bda8-76f8963697fb","Type":"ContainerDied","Data":"defb23c122c278136caa21b1f2a2a997c4c4d2bf64e806f6ba65be84d20224ff"}
Oct 30 13:45:32.924148 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:32.924124    4364 scope.go:122] "RemoveContainer" containerID="91e066f39e1a5e9bb338794da9c823c80ea37572f64bd602be992496a3b56b29"
Oct 30 13:45:32.940314 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:32.940288    4364 scope.go:122] "RemoveContainer" containerID="91e066f39e1a5e9bb338794da9c823c80ea37572f64bd602be992496a3b56b29"
Oct 30 13:45:32.940706 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:45:32.940671    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91e066f39e1a5e9bb338794da9c823c80ea37572f64bd602be992496a3b56b29\": container with ID starting with 91e066f39e1a5e9bb338794da9c823c80ea37572f64bd602be992496a3b56b29 not found: ID does not exist" containerID="91e066f39e1a5e9bb338794da9c823c80ea37572f64bd602be992496a3b56b29"
Oct 30 13:45:32.940811 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:32.940717    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91e066f39e1a5e9bb338794da9c823c80ea37572f64bd602be992496a3b56b29"} err="failed to get container status \"91e066f39e1a5e9bb338794da9c823c80ea37572f64bd602be992496a3b56b29\": rpc error: code = NotFound desc = could not find container \"91e066f39e1a5e9bb338794da9c823c80ea37572f64bd602be992496a3b56b29\": container with ID starting with 91e066f39e1a5e9bb338794da9c823c80ea37572f64bd602be992496a3b56b29 not found: ID does not exist"
Oct 30 13:45:32.983748 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:32.983711    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26"]
Oct 30 13:45:32.988201 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:32.988176    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-thf26"]
Oct 30 13:45:33.504801 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:33.504766    4364 patch_prober.go:29] interesting pod/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:45:33.505181 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:33.504823    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp" podUID="faba8da0-1b85-49fc-aa90-f860879f40a5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:45:33.877183 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:33.877159    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp"
Oct 30 13:45:33.934112 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:33.934070    4364 generic.go:356] "Generic (PLEG): container finished" podID="faba8da0-1b85-49fc-aa90-f860879f40a5" containerID="07f9855fd3f5b075c58f6437479ef56e3ab9421cc19c3ecbfac80e78279ea709" exitCode=0
Oct 30 13:45:33.934283 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:33.934117    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp"
Oct 30 13:45:33.934283 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:33.934149    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp" event={"ID":"faba8da0-1b85-49fc-aa90-f860879f40a5","Type":"ContainerDied","Data":"07f9855fd3f5b075c58f6437479ef56e3ab9421cc19c3ecbfac80e78279ea709"}
Oct 30 13:45:33.934283 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:33.934210    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp" event={"ID":"faba8da0-1b85-49fc-aa90-f860879f40a5","Type":"ContainerDied","Data":"33e1a2fa2a816b6e9c14ec476748d2f9cce0ae464412c71f4a39922c0bc3851a"}
Oct 30 13:45:33.934283 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:33.934249    4364 scope.go:122] "RemoveContainer" containerID="07f9855fd3f5b075c58f6437479ef56e3ab9421cc19c3ecbfac80e78279ea709"
Oct 30 13:45:33.945233 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:33.945216    4364 scope.go:122] "RemoveContainer" containerID="07f9855fd3f5b075c58f6437479ef56e3ab9421cc19c3ecbfac80e78279ea709"
Oct 30 13:45:33.945541 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:45:33.945515    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07f9855fd3f5b075c58f6437479ef56e3ab9421cc19c3ecbfac80e78279ea709\": container with ID starting with 07f9855fd3f5b075c58f6437479ef56e3ab9421cc19c3ecbfac80e78279ea709 not found: ID does not exist" containerID="07f9855fd3f5b075c58f6437479ef56e3ab9421cc19c3ecbfac80e78279ea709"
Oct 30 13:45:33.945648 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:33.945552    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07f9855fd3f5b075c58f6437479ef56e3ab9421cc19c3ecbfac80e78279ea709"} err="failed to get container status \"07f9855fd3f5b075c58f6437479ef56e3ab9421cc19c3ecbfac80e78279ea709\": rpc error: code = NotFound desc = could not find container \"07f9855fd3f5b075c58f6437479ef56e3ab9421cc19c3ecbfac80e78279ea709\": container with ID starting with 07f9855fd3f5b075c58f6437479ef56e3ab9421cc19c3ecbfac80e78279ea709 not found: ID does not exist"
Oct 30 13:45:33.951914 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:33.951893    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faba8da0-1b85-49fc-aa90-f860879f40a5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faba8da0-1b85-49fc-aa90-f860879f40a5-istio-data\") pod \"faba8da0-1b85-49fc-aa90-f860879f40a5\" (UID: \"faba8da0-1b85-49fc-aa90-f860879f40a5\") "
Oct 30 13:45:33.952024 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:33.951939    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faba8da0-1b85-49fc-aa90-f860879f40a5-kube-api-access-ldhhb\" (UniqueName: \"kubernetes.io/projected/faba8da0-1b85-49fc-aa90-f860879f40a5-kube-api-access-ldhhb\") pod \"faba8da0-1b85-49fc-aa90-f860879f40a5\" (UID: \"faba8da0-1b85-49fc-aa90-f860879f40a5\") "
Oct 30 13:45:33.952024 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:33.951993    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faba8da0-1b85-49fc-aa90-f860879f40a5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faba8da0-1b85-49fc-aa90-f860879f40a5-credential-socket\") pod \"faba8da0-1b85-49fc-aa90-f860879f40a5\" (UID: \"faba8da0-1b85-49fc-aa90-f860879f40a5\") "
Oct 30 13:45:33.952145 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:33.952058    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faba8da0-1b85-49fc-aa90-f860879f40a5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faba8da0-1b85-49fc-aa90-f860879f40a5-workload-socket\") pod \"faba8da0-1b85-49fc-aa90-f860879f40a5\" (UID: \"faba8da0-1b85-49fc-aa90-f860879f40a5\") "
Oct 30 13:45:33.952145 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:33.952119    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/faba8da0-1b85-49fc-aa90-f860879f40a5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faba8da0-1b85-49fc-aa90-f860879f40a5-istio-podinfo\") pod \"faba8da0-1b85-49fc-aa90-f860879f40a5\" (UID: \"faba8da0-1b85-49fc-aa90-f860879f40a5\") "
Oct 30 13:45:33.952258 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:33.952155    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faba8da0-1b85-49fc-aa90-f860879f40a5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faba8da0-1b85-49fc-aa90-f860879f40a5-workload-certs\") pod \"faba8da0-1b85-49fc-aa90-f860879f40a5\" (UID: \"faba8da0-1b85-49fc-aa90-f860879f40a5\") "
Oct 30 13:45:33.952258 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:33.952158    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faba8da0-1b85-49fc-aa90-f860879f40a5-istio-data" pod "faba8da0-1b85-49fc-aa90-f860879f40a5" (UID: "faba8da0-1b85-49fc-aa90-f860879f40a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:45:33.952258 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:33.952189    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/faba8da0-1b85-49fc-aa90-f860879f40a5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faba8da0-1b85-49fc-aa90-f860879f40a5-istiod-ca-cert\") pod \"faba8da0-1b85-49fc-aa90-f860879f40a5\" (UID: \"faba8da0-1b85-49fc-aa90-f860879f40a5\") "
Oct 30 13:45:33.952258 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:33.952245    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faba8da0-1b85-49fc-aa90-f860879f40a5-istio-token\" (UniqueName: \"kubernetes.io/projected/faba8da0-1b85-49fc-aa90-f860879f40a5-istio-token\") pod \"faba8da0-1b85-49fc-aa90-f860879f40a5\" (UID: \"faba8da0-1b85-49fc-aa90-f860879f40a5\") "
Oct 30 13:45:33.952488 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:33.952287    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faba8da0-1b85-49fc-aa90-f860879f40a5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faba8da0-1b85-49fc-aa90-f860879f40a5-istio-envoy\") pod \"faba8da0-1b85-49fc-aa90-f860879f40a5\" (UID: \"faba8da0-1b85-49fc-aa90-f860879f40a5\") "
Oct 30 13:45:33.952488 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:33.952334    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faba8da0-1b85-49fc-aa90-f860879f40a5-credential-socket" pod "faba8da0-1b85-49fc-aa90-f860879f40a5" (UID: "faba8da0-1b85-49fc-aa90-f860879f40a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:45:33.952488 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:33.952410    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faba8da0-1b85-49fc-aa90-f860879f40a5-workload-certs" pod "faba8da0-1b85-49fc-aa90-f860879f40a5" (UID: "faba8da0-1b85-49fc-aa90-f860879f40a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:45:33.952488 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:33.952427    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faba8da0-1b85-49fc-aa90-f860879f40a5-workload-socket" pod "faba8da0-1b85-49fc-aa90-f860879f40a5" (UID: "faba8da0-1b85-49fc-aa90-f860879f40a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:45:33.952752 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:33.952661    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faba8da0-1b85-49fc-aa90-f860879f40a5-istiod-ca-cert" pod "faba8da0-1b85-49fc-aa90-f860879f40a5" (UID: "faba8da0-1b85-49fc-aa90-f860879f40a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:33.952752 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:33.952676    4364 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faba8da0-1b85-49fc-aa90-f860879f40a5-workload-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:33.952752 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:33.952695    4364 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faba8da0-1b85-49fc-aa90-f860879f40a5-istio-data\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:33.952752 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:33.952709    4364 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faba8da0-1b85-49fc-aa90-f860879f40a5-credential-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:33.952752 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:33.952722    4364 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faba8da0-1b85-49fc-aa90-f860879f40a5-workload-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:33.954367 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:33.954338    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faba8da0-1b85-49fc-aa90-f860879f40a5-kube-api-access-ldhhb" pod "faba8da0-1b85-49fc-aa90-f860879f40a5" (UID: "faba8da0-1b85-49fc-aa90-f860879f40a5"). InnerVolumeSpecName "kube-api-access-ldhhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:33.954472 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:33.954379    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faba8da0-1b85-49fc-aa90-f860879f40a5-istio-envoy" pod "faba8da0-1b85-49fc-aa90-f860879f40a5" (UID: "faba8da0-1b85-49fc-aa90-f860879f40a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:45:33.954704 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:33.954677    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/faba8da0-1b85-49fc-aa90-f860879f40a5-istio-podinfo" pod "faba8da0-1b85-49fc-aa90-f860879f40a5" (UID: "faba8da0-1b85-49fc-aa90-f860879f40a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:45:33.954834 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:33.954804    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faba8da0-1b85-49fc-aa90-f860879f40a5-istio-token" pod "faba8da0-1b85-49fc-aa90-f860879f40a5" (UID: "faba8da0-1b85-49fc-aa90-f860879f40a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:34.053782 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:34.053701    4364 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faba8da0-1b85-49fc-aa90-f860879f40a5-istio-podinfo\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:34.053782 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:34.053728    4364 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faba8da0-1b85-49fc-aa90-f860879f40a5-istiod-ca-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:34.053782 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:34.053738    4364 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faba8da0-1b85-49fc-aa90-f860879f40a5-istio-token\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:34.053782 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:34.053746    4364 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faba8da0-1b85-49fc-aa90-f860879f40a5-istio-envoy\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:34.053782 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:34.053754    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldhhb\" (UniqueName: \"kubernetes.io/projected/faba8da0-1b85-49fc-aa90-f860879f40a5-kube-api-access-ldhhb\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:34.286278 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:34.286244    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp"]
Oct 30 13:45:34.300357 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:34.300328    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-tfhmp"]
Oct 30 13:45:34.897871 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:34.897844    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:45:34.968654 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:34.968625    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4b120bf-6e2b-46b5-bda8-76f8963697fb" path="/var/lib/kubelet/pods/c4b120bf-6e2b-46b5-bda8-76f8963697fb/volumes"
Oct 30 13:45:34.969096 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:34.969081    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="faba8da0-1b85-49fc-aa90-f860879f40a5" path="/var/lib/kubelet/pods/faba8da0-1b85-49fc-aa90-f860879f40a5/volumes"
Oct 30 13:45:41.284053 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:41.284009    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ee7d65-5b75dc6cc5-sv4rn" podUID="9cbfe471-67e7-43cd-9524-771994bbdb14" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:45:41.284460 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:41.284114    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-ee7d65-58cd4c5bbb-6tv8w" podUID="4dc7d59a-ca00-49e0-8f9c-eabe22d71c17" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:45:41.502864 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:45:41.502826    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:45:43.169657 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:43.169622    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:45:47.976898 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:47.976869    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-ee7d65-58cd4c5bbb-6tv8w"
Oct 30 13:45:47.999197 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:47.999176    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ee7d65-5b75dc6cc5-sv4rn"
Oct 30 13:45:48.014004 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:48.013975    4364 generic.go:356] "Generic (PLEG): container finished" podID="9cbfe471-67e7-43cd-9524-771994bbdb14" containerID="cb5c4578d610d53beff85a1a3201787e5e2ac75254cdc8e63b5b1d469935f87d" exitCode=0
Oct 30 13:45:48.014138 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:48.014005    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ee7d65-5b75dc6cc5-sv4rn" event={"ID":"9cbfe471-67e7-43cd-9524-771994bbdb14","Type":"ContainerDied","Data":"cb5c4578d610d53beff85a1a3201787e5e2ac75254cdc8e63b5b1d469935f87d"}
Oct 30 13:45:48.014138 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:48.014070    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ee7d65-5b75dc6cc5-sv4rn" event={"ID":"9cbfe471-67e7-43cd-9524-771994bbdb14","Type":"ContainerDied","Data":"23f73d43f3bf8ea81d71e199c8de20153e83b8b4a9835ab0d378479b877073ee"}
Oct 30 13:45:48.014138 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:48.014012    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ee7d65-5b75dc6cc5-sv4rn"
Oct 30 13:45:48.014138 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:48.014100    4364 scope.go:122] "RemoveContainer" containerID="cb5c4578d610d53beff85a1a3201787e5e2ac75254cdc8e63b5b1d469935f87d"
Oct 30 13:45:48.015329 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:48.015306    4364 generic.go:356] "Generic (PLEG): container finished" podID="4dc7d59a-ca00-49e0-8f9c-eabe22d71c17" containerID="bbb2700a8683299448b69f70ad21143d3d2a0ddec715b7b006760f597b2b48be" exitCode=0
Oct 30 13:45:48.015436 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:48.015338    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-ee7d65-58cd4c5bbb-6tv8w"
Oct 30 13:45:48.015436 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:48.015350    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-ee7d65-58cd4c5bbb-6tv8w" event={"ID":"4dc7d59a-ca00-49e0-8f9c-eabe22d71c17","Type":"ContainerDied","Data":"bbb2700a8683299448b69f70ad21143d3d2a0ddec715b7b006760f597b2b48be"}
Oct 30 13:45:48.015436 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:48.015380    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-ee7d65-58cd4c5bbb-6tv8w" event={"ID":"4dc7d59a-ca00-49e0-8f9c-eabe22d71c17","Type":"ContainerDied","Data":"4b9a7704fedfa50b5b313eba31c28e36ac26912316b11c2b2db4cc7f811bf705"}
Oct 30 13:45:48.027696 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:48.027680    4364 scope.go:122] "RemoveContainer" containerID="cb5c4578d610d53beff85a1a3201787e5e2ac75254cdc8e63b5b1d469935f87d"
Oct 30 13:45:48.028035 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:45:48.028012    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb5c4578d610d53beff85a1a3201787e5e2ac75254cdc8e63b5b1d469935f87d\": container with ID starting with cb5c4578d610d53beff85a1a3201787e5e2ac75254cdc8e63b5b1d469935f87d not found: ID does not exist" containerID="cb5c4578d610d53beff85a1a3201787e5e2ac75254cdc8e63b5b1d469935f87d"
Oct 30 13:45:48.028163 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:48.028052    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb5c4578d610d53beff85a1a3201787e5e2ac75254cdc8e63b5b1d469935f87d"} err="failed to get container status \"cb5c4578d610d53beff85a1a3201787e5e2ac75254cdc8e63b5b1d469935f87d\": rpc error: code = NotFound desc = could not find container \"cb5c4578d610d53beff85a1a3201787e5e2ac75254cdc8e63b5b1d469935f87d\": container with ID starting with cb5c4578d610d53beff85a1a3201787e5e2ac75254cdc8e63b5b1d469935f87d not found: ID does not exist"
Oct 30 13:45:48.028163 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:48.028074    4364 scope.go:122] "RemoveContainer" containerID="bbb2700a8683299448b69f70ad21143d3d2a0ddec715b7b006760f597b2b48be"
Oct 30 13:45:48.037237 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:48.037218    4364 scope.go:122] "RemoveContainer" containerID="bbb2700a8683299448b69f70ad21143d3d2a0ddec715b7b006760f597b2b48be"
Oct 30 13:45:48.037508 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:45:48.037484    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbb2700a8683299448b69f70ad21143d3d2a0ddec715b7b006760f597b2b48be\": container with ID starting with bbb2700a8683299448b69f70ad21143d3d2a0ddec715b7b006760f597b2b48be not found: ID does not exist" containerID="bbb2700a8683299448b69f70ad21143d3d2a0ddec715b7b006760f597b2b48be"
Oct 30 13:45:48.037586 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:48.037515    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbb2700a8683299448b69f70ad21143d3d2a0ddec715b7b006760f597b2b48be"} err="failed to get container status \"bbb2700a8683299448b69f70ad21143d3d2a0ddec715b7b006760f597b2b48be\": rpc error: code = NotFound desc = could not find container \"bbb2700a8683299448b69f70ad21143d3d2a0ddec715b7b006760f597b2b48be\": container with ID starting with bbb2700a8683299448b69f70ad21143d3d2a0ddec715b7b006760f597b2b48be not found: ID does not exist"
Oct 30 13:45:48.085187 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:48.085134    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4dc7d59a-ca00-49e0-8f9c-eabe22d71c17-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dc7d59a-ca00-49e0-8f9c-eabe22d71c17-plugins-config-volume\") pod \"4dc7d59a-ca00-49e0-8f9c-eabe22d71c17\" (UID: \"4dc7d59a-ca00-49e0-8f9c-eabe22d71c17\") "
Oct 30 13:45:48.085389 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:48.085250    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cbfe471-67e7-43cd-9524-771994bbdb14-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cbfe471-67e7-43cd-9524-771994bbdb14-plugins-config-volume\") pod \"9cbfe471-67e7-43cd-9524-771994bbdb14\" (UID: \"9cbfe471-67e7-43cd-9524-771994bbdb14\") "
Oct 30 13:45:48.085389 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:48.085286    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dc7d59a-ca00-49e0-8f9c-eabe22d71c17-kube-api-access-pbqdd\" (UniqueName: \"kubernetes.io/projected/4dc7d59a-ca00-49e0-8f9c-eabe22d71c17-kube-api-access-pbqdd\") pod \"4dc7d59a-ca00-49e0-8f9c-eabe22d71c17\" (UID: \"4dc7d59a-ca00-49e0-8f9c-eabe22d71c17\") "
Oct 30 13:45:48.085389 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:48.085315    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cbfe471-67e7-43cd-9524-771994bbdb14-kube-api-access-jlhrm\" (UniqueName: \"kubernetes.io/projected/9cbfe471-67e7-43cd-9524-771994bbdb14-kube-api-access-jlhrm\") pod \"9cbfe471-67e7-43cd-9524-771994bbdb14\" (UID: \"9cbfe471-67e7-43cd-9524-771994bbdb14\") "
Oct 30 13:45:48.085732 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:48.085702    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dc7d59a-ca00-49e0-8f9c-eabe22d71c17-plugins-config-volume" pod "4dc7d59a-ca00-49e0-8f9c-eabe22d71c17" (UID: "4dc7d59a-ca00-49e0-8f9c-eabe22d71c17"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:48.086122 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:48.086093    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cbfe471-67e7-43cd-9524-771994bbdb14-plugins-config-volume" pod "9cbfe471-67e7-43cd-9524-771994bbdb14" (UID: "9cbfe471-67e7-43cd-9524-771994bbdb14"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:48.088111 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:48.088074    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dc7d59a-ca00-49e0-8f9c-eabe22d71c17-kube-api-access-pbqdd" pod "4dc7d59a-ca00-49e0-8f9c-eabe22d71c17" (UID: "4dc7d59a-ca00-49e0-8f9c-eabe22d71c17"). InnerVolumeSpecName "kube-api-access-pbqdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:48.088203 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:48.088180    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cbfe471-67e7-43cd-9524-771994bbdb14-kube-api-access-jlhrm" pod "9cbfe471-67e7-43cd-9524-771994bbdb14" (UID: "9cbfe471-67e7-43cd-9524-771994bbdb14"). InnerVolumeSpecName "kube-api-access-jlhrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:48.186153 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:48.186121    4364 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cbfe471-67e7-43cd-9524-771994bbdb14-plugins-config-volume\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:48.186153 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:48.186149    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbqdd\" (UniqueName: \"kubernetes.io/projected/4dc7d59a-ca00-49e0-8f9c-eabe22d71c17-kube-api-access-pbqdd\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:48.186361 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:48.186160    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlhrm\" (UniqueName: \"kubernetes.io/projected/9cbfe471-67e7-43cd-9524-771994bbdb14-kube-api-access-jlhrm\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:48.186361 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:48.186172    4364 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dc7d59a-ca00-49e0-8f9c-eabe22d71c17-plugins-config-volume\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:48.377369 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:48.377335    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-ee7d65-5b75dc6cc5-sv4rn"]
Oct 30 13:45:48.400473 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:48.400435    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-ee7d65-5b75dc6cc5-sv4rn"]
Oct 30 13:45:48.431856 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:48.431827    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb"]
Oct 30 13:45:48.432060 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:48.432038    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb" podUID="c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3" containerName="istio-proxy" containerID="cri-o://87e110aab645f150d2dab72a28f0fca46758a8173e6706024ef4a4dad396c4ab" gracePeriod=30
Oct 30 13:45:48.448779 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:48.448746    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-ee7d65-58cd4c5bbb-6tv8w"]
Oct 30 13:45:48.457798 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:48.457772    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-ee7d65-58cd4c5bbb-6tv8w"]
Oct 30 13:45:48.968227 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:48.968146    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4dc7d59a-ca00-49e0-8f9c-eabe22d71c17" path="/var/lib/kubelet/pods/4dc7d59a-ca00-49e0-8f9c-eabe22d71c17/volumes"
Oct 30 13:45:48.968544 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:48.968531    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cbfe471-67e7-43cd-9524-771994bbdb14" path="/var/lib/kubelet/pods/9cbfe471-67e7-43cd-9524-771994bbdb14/volumes"
Oct 30 13:45:49.165767 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:49.165278    4364 patch_prober.go:29] interesting pod/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:45:49.165767 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:49.165332    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb" podUID="c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:45:53.698189 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:53.698110    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb"
Oct 30 13:45:53.834928 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:53.834880    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3-workload-certs\") pod \"c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3\" (UID: \"c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3\") "
Oct 30 13:45:53.835128 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:53.834983    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3-istiod-ca-cert\") pod \"c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3\" (UID: \"c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3\") "
Oct 30 13:45:53.835128 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:53.835037    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3-istio-envoy\") pod \"c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3\" (UID: \"c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3\") "
Oct 30 13:45:53.835128 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:53.835106    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3-kube-api-access-bnfqg\" (UniqueName: \"kubernetes.io/projected/c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3-kube-api-access-bnfqg\") pod \"c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3\" (UID: \"c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3\") "
Oct 30 13:45:53.835375 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:53.835157    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3-istio-data\") pod \"c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3\" (UID: \"c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3\") "
Oct 30 13:45:53.835375 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:53.835211    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3-istio-podinfo\") pod \"c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3\" (UID: \"c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3\") "
Oct 30 13:45:53.835375 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:53.835240    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3-istio-token\" (UniqueName: \"kubernetes.io/projected/c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3-istio-token\") pod \"c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3\" (UID: \"c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3\") "
Oct 30 13:45:53.835375 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:53.835272    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3-workload-socket\") pod \"c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3\" (UID: \"c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3\") "
Oct 30 13:45:53.835375 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:53.835321    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3-credential-socket\") pod \"c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3\" (UID: \"c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3\") "
Oct 30 13:45:53.836228 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:53.836016    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3-credential-socket" pod "c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3" (UID: "c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:45:53.836453 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:53.836418    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3-istio-data" pod "c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3" (UID: "c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:45:53.837054 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:53.836752    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3-workload-socket" pod "c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3" (UID: "c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:45:53.837858 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:53.837814    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3-workload-certs" pod "c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3" (UID: "c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:45:53.838314 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:53.838291    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3-istiod-ca-cert" pod "c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3" (UID: "c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:53.848105 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:53.845242    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3-kube-api-access-bnfqg" pod "c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3" (UID: "c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3"). InnerVolumeSpecName "kube-api-access-bnfqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:53.848105 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:53.846718    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3-istio-podinfo" pod "c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3" (UID: "c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:45:53.848105 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:53.846722    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3-istio-envoy" pod "c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3" (UID: "c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:45:53.848105 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:53.846839    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3-istio-token" pod "c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3" (UID: "c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:53.937080 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:53.937039    4364 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3-istiod-ca-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:53.937080 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:53.937068    4364 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3-istio-envoy\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:53.937080 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:53.937078    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnfqg\" (UniqueName: \"kubernetes.io/projected/c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3-kube-api-access-bnfqg\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:53.937080 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:53.937087    4364 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3-istio-data\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:53.937441 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:53.937095    4364 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3-istio-podinfo\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:53.937441 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:53.937104    4364 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3-istio-token\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:53.937441 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:53.937111    4364 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3-workload-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:53.937441 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:53.937119    4364 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3-credential-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:53.937441 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:53.937127    4364 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3-workload-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:45:54.052551 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:54.052470    4364 generic.go:356] "Generic (PLEG): container finished" podID="c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3" containerID="87e110aab645f150d2dab72a28f0fca46758a8173e6706024ef4a4dad396c4ab" exitCode=0
Oct 30 13:45:54.052551 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:54.052512    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb" event={"ID":"c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3","Type":"ContainerDied","Data":"87e110aab645f150d2dab72a28f0fca46758a8173e6706024ef4a4dad396c4ab"}
Oct 30 13:45:54.052551 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:54.052530    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb"
Oct 30 13:45:54.052856 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:54.052552    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb" event={"ID":"c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3","Type":"ContainerDied","Data":"f780ebddbba70b732c603522b531da0e0a8c76d2053e3ee200b3d80b9dec0c2c"}
Oct 30 13:45:54.052856 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:54.052575    4364 scope.go:122] "RemoveContainer" containerID="87e110aab645f150d2dab72a28f0fca46758a8173e6706024ef4a4dad396c4ab"
Oct 30 13:45:54.062156 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:54.062139    4364 scope.go:122] "RemoveContainer" containerID="87e110aab645f150d2dab72a28f0fca46758a8173e6706024ef4a4dad396c4ab"
Oct 30 13:45:54.062424 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:45:54.062403    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87e110aab645f150d2dab72a28f0fca46758a8173e6706024ef4a4dad396c4ab\": container with ID starting with 87e110aab645f150d2dab72a28f0fca46758a8173e6706024ef4a4dad396c4ab not found: ID does not exist" containerID="87e110aab645f150d2dab72a28f0fca46758a8173e6706024ef4a4dad396c4ab"
Oct 30 13:45:54.062492 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:54.062433    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87e110aab645f150d2dab72a28f0fca46758a8173e6706024ef4a4dad396c4ab"} err="failed to get container status \"87e110aab645f150d2dab72a28f0fca46758a8173e6706024ef4a4dad396c4ab\": rpc error: code = NotFound desc = could not find container \"87e110aab645f150d2dab72a28f0fca46758a8173e6706024ef4a4dad396c4ab\": container with ID starting with 87e110aab645f150d2dab72a28f0fca46758a8173e6706024ef4a4dad396c4ab not found: ID does not exist"
Oct 30 13:45:54.114481 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:54.114452    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb"]
Oct 30 13:45:54.127726 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:54.127699    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-72lqb"]
Oct 30 13:45:54.968154 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:45:54.968124    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3" path="/var/lib/kubelet/pods/c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3/volumes"
Oct 30 13:46:10.360033 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:10.360001    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-wfkdb"]
Oct 30 13:46:10.360444 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:10.360367    4364 cpu_manager.go:414] "containerMap: removing container" podUID="4dc7d59a-ca00-49e0-8f9c-eabe22d71c17" containerName="payload-processing"
Oct 30 13:46:10.360444 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:10.360378    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dc7d59a-ca00-49e0-8f9c-eabe22d71c17" containerName="payload-processing"
Oct 30 13:46:10.360444 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:10.360388    4364 cpu_manager.go:414] "containerMap: removing container" podUID="aaf4329d-4d56-4ce3-8eea-dbaf8da1c07a" containerName="cleanup"
Oct 30 13:46:10.360444 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:10.360394    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaf4329d-4d56-4ce3-8eea-dbaf8da1c07a" containerName="cleanup"
Oct 30 13:46:10.360444 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:10.360407    4364 cpu_manager.go:414] "containerMap: removing container" podUID="c4b120bf-6e2b-46b5-bda8-76f8963697fb" containerName="istio-proxy"
Oct 30 13:46:10.360444 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:10.360412    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4b120bf-6e2b-46b5-bda8-76f8963697fb" containerName="istio-proxy"
Oct 30 13:46:10.360444 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:10.360422    4364 cpu_manager.go:414] "containerMap: removing container" podUID="22d0f282-215c-45d9-9fbb-15190b9ebd10" containerName="payload-pre-processing"
Oct 30 13:46:10.360444 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:10.360427    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22d0f282-215c-45d9-9fbb-15190b9ebd10" containerName="payload-pre-processing"
Oct 30 13:46:10.360444 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:10.360435    4364 cpu_manager.go:414] "containerMap: removing container" podUID="1453dcde-0c6d-427d-890b-8f3542a0775a" containerName="cleanup"
Oct 30 13:46:10.360444 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:10.360440    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1453dcde-0c6d-427d-890b-8f3542a0775a" containerName="cleanup"
Oct 30 13:46:10.360444 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:10.360447    4364 cpu_manager.go:414] "containerMap: removing container" podUID="6e561630-bbec-40f3-a5c3-143f153cc443" containerName="revoke-keys"
Oct 30 13:46:10.360843 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:10.360452    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e561630-bbec-40f3-a5c3-143f153cc443" containerName="revoke-keys"
Oct 30 13:46:10.360843 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:10.360460    4364 cpu_manager.go:414] "containerMap: removing container" podUID="c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3" containerName="istio-proxy"
Oct 30 13:46:10.360843 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:10.360464    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3" containerName="istio-proxy"
Oct 30 13:46:10.360843 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:10.360485    4364 cpu_manager.go:414] "containerMap: removing container" podUID="9cbfe471-67e7-43cd-9524-771994bbdb14" containerName="payload-pre-processing"
Oct 30 13:46:10.360843 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:10.360490    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cbfe471-67e7-43cd-9524-771994bbdb14" containerName="payload-pre-processing"
Oct 30 13:46:10.360843 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:10.360495    4364 cpu_manager.go:414] "containerMap: removing container" podUID="faba8da0-1b85-49fc-aa90-f860879f40a5" containerName="istio-proxy"
Oct 30 13:46:10.360843 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:10.360499    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faba8da0-1b85-49fc-aa90-f860879f40a5" containerName="istio-proxy"
Oct 30 13:46:10.360843 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:10.360507    4364 cpu_manager.go:414] "containerMap: removing container" podUID="c0628c77-ff94-4a69-a0a0-af20a2b64e13" containerName="cleanup"
Oct 30 13:46:10.360843 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:10.360512    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0628c77-ff94-4a69-a0a0-af20a2b64e13" containerName="cleanup"
Oct 30 13:46:10.360843 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:10.360567    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cbfe471-67e7-43cd-9524-771994bbdb14" containerName="payload-pre-processing"
Oct 30 13:46:10.360843 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:10.360574    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="faba8da0-1b85-49fc-aa90-f860879f40a5" containerName="istio-proxy"
Oct 30 13:46:10.360843 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:10.360581    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dc7d59a-ca00-49e0-8f9c-eabe22d71c17" containerName="payload-processing"
Oct 30 13:46:10.360843 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:10.360615    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4b120bf-6e2b-46b5-bda8-76f8963697fb" containerName="istio-proxy"
Oct 30 13:46:10.360843 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:10.360623    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="22d0f282-215c-45d9-9fbb-15190b9ebd10" containerName="payload-pre-processing"
Oct 30 13:46:10.360843 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:10.360630    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e561630-bbec-40f3-a5c3-143f153cc443" containerName="revoke-keys"
Oct 30 13:46:10.360843 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:10.360636    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0a2a8a5-ed1c-4b5e-829b-0e36073ca5d3" containerName="istio-proxy"
Oct 30 13:46:10.365279 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:10.365249    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-wfkdb"
Oct 30 13:46:10.400752 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:10.400712    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-wfkdb"]
Oct 30 13:46:10.484693 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:10.484660    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cac946c2-24d3-42b9-9d62-5f45833c81fc-istio-token\") pod \"maas-default-gateway-openshift-default-64d4cc998c-wfkdb\" (UID: \"cac946c2-24d3-42b9-9d62-5f45833c81fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-wfkdb"
Oct 30 13:46:10.484693 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:10.484701    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cac946c2-24d3-42b9-9d62-5f45833c81fc-credential-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-wfkdb\" (UID: \"cac946c2-24d3-42b9-9d62-5f45833c81fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-wfkdb"
Oct 30 13:46:10.484909 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:10.484718    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cac946c2-24d3-42b9-9d62-5f45833c81fc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-64d4cc998c-wfkdb\" (UID: \"cac946c2-24d3-42b9-9d62-5f45833c81fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-wfkdb"
Oct 30 13:46:10.484909 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:10.484785    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cac946c2-24d3-42b9-9d62-5f45833c81fc-istio-envoy\") pod \"maas-default-gateway-openshift-default-64d4cc998c-wfkdb\" (UID: \"cac946c2-24d3-42b9-9d62-5f45833c81fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-wfkdb"
Oct 30 13:46:10.484909 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:10.484825    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cac946c2-24d3-42b9-9d62-5f45833c81fc-workload-certs\") pod \"maas-default-gateway-openshift-default-64d4cc998c-wfkdb\" (UID: \"cac946c2-24d3-42b9-9d62-5f45833c81fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-wfkdb"
Oct 30 13:46:10.484909 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:10.484871    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cac946c2-24d3-42b9-9d62-5f45833c81fc-workload-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-wfkdb\" (UID: \"cac946c2-24d3-42b9-9d62-5f45833c81fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-wfkdb"
Oct 30 13:46:10.484909 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:10.484900    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cac946c2-24d3-42b9-9d62-5f45833c81fc-istio-data\") pod \"maas-default-gateway-openshift-default-64d4cc998c-wfkdb\" (UID: \"cac946c2-24d3-42b9-9d62-5f45833c81fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-wfkdb"
Oct 30 13:46:10.485089 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:10.484974    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbjqs\" (UniqueName: \"kubernetes.io/projected/cac946c2-24d3-42b9-9d62-5f45833c81fc-kube-api-access-wbjqs\") pod \"maas-default-gateway-openshift-default-64d4cc998c-wfkdb\" (UID: \"cac946c2-24d3-42b9-9d62-5f45833c81fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-wfkdb"
Oct 30 13:46:10.485089 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:10.485003    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cac946c2-24d3-42b9-9d62-5f45833c81fc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-64d4cc998c-wfkdb\" (UID: \"cac946c2-24d3-42b9-9d62-5f45833c81fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-wfkdb"
Oct 30 13:46:10.585929 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:10.585897    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cac946c2-24d3-42b9-9d62-5f45833c81fc-istio-token\") pod \"maas-default-gateway-openshift-default-64d4cc998c-wfkdb\" (UID: \"cac946c2-24d3-42b9-9d62-5f45833c81fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-wfkdb"
Oct 30 13:46:10.585929 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:10.585935    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cac946c2-24d3-42b9-9d62-5f45833c81fc-credential-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-wfkdb\" (UID: \"cac946c2-24d3-42b9-9d62-5f45833c81fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-wfkdb"
Oct 30 13:46:10.586168 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:10.585960    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cac946c2-24d3-42b9-9d62-5f45833c81fc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-64d4cc998c-wfkdb\" (UID: \"cac946c2-24d3-42b9-9d62-5f45833c81fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-wfkdb"
Oct 30 13:46:10.586168 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:10.585997    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cac946c2-24d3-42b9-9d62-5f45833c81fc-istio-envoy\") pod \"maas-default-gateway-openshift-default-64d4cc998c-wfkdb\" (UID: \"cac946c2-24d3-42b9-9d62-5f45833c81fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-wfkdb"
Oct 30 13:46:10.586168 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:10.586020    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cac946c2-24d3-42b9-9d62-5f45833c81fc-workload-certs\") pod \"maas-default-gateway-openshift-default-64d4cc998c-wfkdb\" (UID: \"cac946c2-24d3-42b9-9d62-5f45833c81fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-wfkdb"
Oct 30 13:46:10.586168 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:10.586047    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cac946c2-24d3-42b9-9d62-5f45833c81fc-workload-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-wfkdb\" (UID: \"cac946c2-24d3-42b9-9d62-5f45833c81fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-wfkdb"
Oct 30 13:46:10.586168 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:10.586077    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cac946c2-24d3-42b9-9d62-5f45833c81fc-istio-data\") pod \"maas-default-gateway-openshift-default-64d4cc998c-wfkdb\" (UID: \"cac946c2-24d3-42b9-9d62-5f45833c81fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-wfkdb"
Oct 30 13:46:10.586168 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:10.586122    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbjqs\" (UniqueName: \"kubernetes.io/projected/cac946c2-24d3-42b9-9d62-5f45833c81fc-kube-api-access-wbjqs\") pod \"maas-default-gateway-openshift-default-64d4cc998c-wfkdb\" (UID: \"cac946c2-24d3-42b9-9d62-5f45833c81fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-wfkdb"
Oct 30 13:46:10.586168 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:10.586158    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cac946c2-24d3-42b9-9d62-5f45833c81fc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-64d4cc998c-wfkdb\" (UID: \"cac946c2-24d3-42b9-9d62-5f45833c81fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-wfkdb"
Oct 30 13:46:10.586562 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:10.586385    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cac946c2-24d3-42b9-9d62-5f45833c81fc-credential-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-wfkdb\" (UID: \"cac946c2-24d3-42b9-9d62-5f45833c81fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-wfkdb"
Oct 30 13:46:10.586562 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:10.586446    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cac946c2-24d3-42b9-9d62-5f45833c81fc-workload-certs\") pod \"maas-default-gateway-openshift-default-64d4cc998c-wfkdb\" (UID: \"cac946c2-24d3-42b9-9d62-5f45833c81fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-wfkdb"
Oct 30 13:46:10.586657 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:10.586580    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cac946c2-24d3-42b9-9d62-5f45833c81fc-istio-data\") pod \"maas-default-gateway-openshift-default-64d4cc998c-wfkdb\" (UID: \"cac946c2-24d3-42b9-9d62-5f45833c81fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-wfkdb"
Oct 30 13:46:10.586657 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:10.586637    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cac946c2-24d3-42b9-9d62-5f45833c81fc-workload-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-wfkdb\" (UID: \"cac946c2-24d3-42b9-9d62-5f45833c81fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-wfkdb"
Oct 30 13:46:10.586816 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:10.586794    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cac946c2-24d3-42b9-9d62-5f45833c81fc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-64d4cc998c-wfkdb\" (UID: \"cac946c2-24d3-42b9-9d62-5f45833c81fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-wfkdb"
Oct 30 13:46:10.588415 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:10.588395    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cac946c2-24d3-42b9-9d62-5f45833c81fc-istio-envoy\") pod \"maas-default-gateway-openshift-default-64d4cc998c-wfkdb\" (UID: \"cac946c2-24d3-42b9-9d62-5f45833c81fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-wfkdb"
Oct 30 13:46:10.588768 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:10.588746    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cac946c2-24d3-42b9-9d62-5f45833c81fc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-64d4cc998c-wfkdb\" (UID: \"cac946c2-24d3-42b9-9d62-5f45833c81fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-wfkdb"
Oct 30 13:46:10.611943 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:10.611887    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cac946c2-24d3-42b9-9d62-5f45833c81fc-istio-token\") pod \"maas-default-gateway-openshift-default-64d4cc998c-wfkdb\" (UID: \"cac946c2-24d3-42b9-9d62-5f45833c81fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-wfkdb"
Oct 30 13:46:10.612282 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:10.612264    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbjqs\" (UniqueName: \"kubernetes.io/projected/cac946c2-24d3-42b9-9d62-5f45833c81fc-kube-api-access-wbjqs\") pod \"maas-default-gateway-openshift-default-64d4cc998c-wfkdb\" (UID: \"cac946c2-24d3-42b9-9d62-5f45833c81fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-wfkdb"
Oct 30 13:46:10.681080 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:10.681045    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-wfkdb"
Oct 30 13:46:10.999714 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:10.999685    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-wfkdb"]
Oct 30 13:46:11.000296 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:46:11.000266    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcac946c2_24d3_42b9_9d62_5f45833c81fc.slice/crio-d330f2082a07c323ff695966823947b412ac43e48121eee0987d33cc88654091 WatchSource:0}: Error finding container d330f2082a07c323ff695966823947b412ac43e48121eee0987d33cc88654091: Status 404 returned error can't find the container with id d330f2082a07c323ff695966823947b412ac43e48121eee0987d33cc88654091
Oct 30 13:46:11.002222 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:11.002187    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:46:11.002292 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:11.002262    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:46:11.002336 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:11.002291    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:46:11.141252 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:11.141205    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-wfkdb" event={"ID":"cac946c2-24d3-42b9-9d62-5f45833c81fc","Type":"ContainerStarted","Data":"41d5424152d04ea993fa694188fec9f9fadca00dcb279b8b253ecee77b314cb6"}
Oct 30 13:46:11.141454 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:11.141276    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-wfkdb" event={"ID":"cac946c2-24d3-42b9-9d62-5f45833c81fc","Type":"ContainerStarted","Data":"d330f2082a07c323ff695966823947b412ac43e48121eee0987d33cc88654091"}
Oct 30 13:46:11.198645 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:11.198581    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-wfkdb" podStartSLOduration=1.198566814 podStartE2EDuration="1.198566814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:46:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:46:11.197235776 +0000 UTC m=+3270.363600424" watchObservedRunningTime="2026-10-30 13:46:11.198566814 +0000 UTC m=+3270.364931461"
Oct 30 13:46:11.530970 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:46:11.530929    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:46:11.700623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:11.695701    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-wfkdb"
Oct 30 13:46:12.698623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:12.688662    4364 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-wfkdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body=
Oct 30 13:46:12.698623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:12.688759    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-wfkdb" podUID="cac946c2-24d3-42b9-9d62-5f45833c81fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused"
Oct 30 13:46:13.703623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:13.699651    4364 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-wfkdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body=
Oct 30 13:46:13.703623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:13.699718    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-wfkdb" podUID="cac946c2-24d3-42b9-9d62-5f45833c81fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused"
Oct 30 13:46:14.697623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:14.691993    4364 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-wfkdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body=
Oct 30 13:46:14.697623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:14.692059    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-wfkdb" podUID="cac946c2-24d3-42b9-9d62-5f45833c81fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused"
Oct 30 13:46:15.695630 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:15.692014    4364 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-wfkdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body=
Oct 30 13:46:15.695630 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:15.692082    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-wfkdb" podUID="cac946c2-24d3-42b9-9d62-5f45833c81fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused"
Oct 30 13:46:16.699622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:16.698741    4364 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-wfkdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body=
Oct 30 13:46:16.699622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:16.698806    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-wfkdb" podUID="cac946c2-24d3-42b9-9d62-5f45833c81fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused"
Oct 30 13:46:17.682364 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:17.682329    4364 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-wfkdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body=
Oct 30 13:46:17.682523 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:17.682382    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-wfkdb" podUID="cac946c2-24d3-42b9-9d62-5f45833c81fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused"
Oct 30 13:46:18.476030 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:18.475985    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-q8tpb"]
Oct 30 13:46:18.479533 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:18.479507    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-q8tpb"
Oct 30 13:46:18.500882 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:18.500844    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-q8tpb"]
Oct 30 13:46:18.510040 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:18.510011    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:46:18.510040 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:18.510032    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:46:18.510206 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:18.510020    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:46:18.513078 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:18.513062    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\""
Oct 30 13:46:18.572691 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:18.572665    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ff3f00b-a2fc-461f-a23c-32cb08dc3225-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-q8tpb\" (UID: \"2ff3f00b-a2fc-461f-a23c-32cb08dc3225\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-q8tpb"
Oct 30 13:46:18.572846 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:18.572707    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghsbt\" (UniqueName: \"kubernetes.io/projected/2ff3f00b-a2fc-461f-a23c-32cb08dc3225-kube-api-access-ghsbt\") pod \"maas-api-revoke-sub-85237e45-q8tpb\" (UID: \"2ff3f00b-a2fc-461f-a23c-32cb08dc3225\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-q8tpb"
Oct 30 13:46:18.572846 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:18.572733    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ff3f00b-a2fc-461f-a23c-32cb08dc3225-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-q8tpb\" (UID: \"2ff3f00b-a2fc-461f-a23c-32cb08dc3225\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-q8tpb"
Oct 30 13:46:18.673737 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:18.673701    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ff3f00b-a2fc-461f-a23c-32cb08dc3225-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-q8tpb\" (UID: \"2ff3f00b-a2fc-461f-a23c-32cb08dc3225\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-q8tpb"
Oct 30 13:46:18.673915 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:18.673758    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghsbt\" (UniqueName: \"kubernetes.io/projected/2ff3f00b-a2fc-461f-a23c-32cb08dc3225-kube-api-access-ghsbt\") pod \"maas-api-revoke-sub-85237e45-q8tpb\" (UID: \"2ff3f00b-a2fc-461f-a23c-32cb08dc3225\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-q8tpb"
Oct 30 13:46:18.673915 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:18.673785    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ff3f00b-a2fc-461f-a23c-32cb08dc3225-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-q8tpb\" (UID: \"2ff3f00b-a2fc-461f-a23c-32cb08dc3225\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-q8tpb"
Oct 30 13:46:18.674505 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:18.674474    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ff3f00b-a2fc-461f-a23c-32cb08dc3225-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-q8tpb\" (UID: \"2ff3f00b-a2fc-461f-a23c-32cb08dc3225\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-q8tpb"
Oct 30 13:46:18.676170 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:18.676149    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ff3f00b-a2fc-461f-a23c-32cb08dc3225-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-q8tpb\" (UID: \"2ff3f00b-a2fc-461f-a23c-32cb08dc3225\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-q8tpb"
Oct 30 13:46:18.703512 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:18.703484    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghsbt\" (UniqueName: \"kubernetes.io/projected/2ff3f00b-a2fc-461f-a23c-32cb08dc3225-kube-api-access-ghsbt\") pod \"maas-api-revoke-sub-85237e45-q8tpb\" (UID: \"2ff3f00b-a2fc-461f-a23c-32cb08dc3225\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-q8tpb"
Oct 30 13:46:18.793629 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:18.793545    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-q8tpb"
Oct 30 13:46:19.107965 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:46:19.107928    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ff3f00b_a2fc_461f_a23c_32cb08dc3225.slice/crio-6619387d162a4b7b198325f48fb892a8948c2fdef5f3d77c7bfa28a3a1d7f125 WatchSource:0}: Error finding container 6619387d162a4b7b198325f48fb892a8948c2fdef5f3d77c7bfa28a3a1d7f125: Status 404 returned error can't find the container with id 6619387d162a4b7b198325f48fb892a8948c2fdef5f3d77c7bfa28a3a1d7f125
Oct 30 13:46:19.108312 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:19.108289    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-q8tpb"]
Oct 30 13:46:19.184959 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:19.184925    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-q8tpb" event={"ID":"2ff3f00b-a2fc-461f-a23c-32cb08dc3225","Type":"ContainerStarted","Data":"6619387d162a4b7b198325f48fb892a8948c2fdef5f3d77c7bfa28a3a1d7f125"}
Oct 30 13:46:19.286961 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:19.286933    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-wfkdb"
Oct 30 13:46:19.287261 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:19.287235    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-wfkdb"
Oct 30 13:46:19.288025 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:19.288005    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-wfkdb"
Oct 30 13:46:20.192116 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:20.192082    4364 generic.go:356] "Generic (PLEG): container finished" podID="2ff3f00b-a2fc-461f-a23c-32cb08dc3225" containerID="2093bc5327f48b99611ee4bd6d24a26df4d82e3bb3f868c9c6a5a0ca6bd30a0c" exitCode=0
Oct 30 13:46:20.192562 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:20.192125    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-q8tpb" event={"ID":"2ff3f00b-a2fc-461f-a23c-32cb08dc3225","Type":"ContainerDied","Data":"2093bc5327f48b99611ee4bd6d24a26df4d82e3bb3f868c9c6a5a0ca6bd30a0c"}
Oct 30 13:46:21.377937 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:21.377909    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-q8tpb"
Oct 30 13:46:21.398675 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:21.398633    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ff3f00b-a2fc-461f-a23c-32cb08dc3225-kube-api-access-ghsbt\" (UniqueName: \"kubernetes.io/projected/2ff3f00b-a2fc-461f-a23c-32cb08dc3225-kube-api-access-ghsbt\") pod \"2ff3f00b-a2fc-461f-a23c-32cb08dc3225\" (UID: \"2ff3f00b-a2fc-461f-a23c-32cb08dc3225\") "
Oct 30 13:46:21.398905 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:21.398715    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ff3f00b-a2fc-461f-a23c-32cb08dc3225-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ff3f00b-a2fc-461f-a23c-32cb08dc3225-maas-api-xks-ca\") pod \"2ff3f00b-a2fc-461f-a23c-32cb08dc3225\" (UID: \"2ff3f00b-a2fc-461f-a23c-32cb08dc3225\") "
Oct 30 13:46:21.398905 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:21.398862    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ff3f00b-a2fc-461f-a23c-32cb08dc3225-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ff3f00b-a2fc-461f-a23c-32cb08dc3225-maas-api-openshift-ca\") pod \"2ff3f00b-a2fc-461f-a23c-32cb08dc3225\" (UID: \"2ff3f00b-a2fc-461f-a23c-32cb08dc3225\") "
Oct 30 13:46:21.399376 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:21.399345    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ff3f00b-a2fc-461f-a23c-32cb08dc3225-maas-api-openshift-ca" pod "2ff3f00b-a2fc-461f-a23c-32cb08dc3225" (UID: "2ff3f00b-a2fc-461f-a23c-32cb08dc3225"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:46:21.401537 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:21.401512    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ff3f00b-a2fc-461f-a23c-32cb08dc3225-kube-api-access-ghsbt" pod "2ff3f00b-a2fc-461f-a23c-32cb08dc3225" (UID: "2ff3f00b-a2fc-461f-a23c-32cb08dc3225"). InnerVolumeSpecName "kube-api-access-ghsbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:46:21.401791 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:21.401772    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ff3f00b-a2fc-461f-a23c-32cb08dc3225-maas-api-xks-ca" pod "2ff3f00b-a2fc-461f-a23c-32cb08dc3225" (UID: "2ff3f00b-a2fc-461f-a23c-32cb08dc3225"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:46:21.499974 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:21.499877    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ff3f00b-a2fc-461f-a23c-32cb08dc3225-maas-api-openshift-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:46:21.499974 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:21.499916    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghsbt\" (UniqueName: \"kubernetes.io/projected/2ff3f00b-a2fc-461f-a23c-32cb08dc3225-kube-api-access-ghsbt\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:46:21.499974 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:21.499934    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ff3f00b-a2fc-461f-a23c-32cb08dc3225-maas-api-xks-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:46:22.207278 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:22.207241    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-q8tpb"
Oct 30 13:46:22.207509 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:22.207231    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-q8tpb" event={"ID":"2ff3f00b-a2fc-461f-a23c-32cb08dc3225","Type":"ContainerDied","Data":"6619387d162a4b7b198325f48fb892a8948c2fdef5f3d77c7bfa28a3a1d7f125"}
Oct 30 13:46:22.207509 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:46:22.207380    4364 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6619387d162a4b7b198325f48fb892a8948c2fdef5f3d77c7bfa28a3a1d7f125"
Oct 30 13:46:41.504436 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:46:41.504394    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:47:03.498687 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:03.498653    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:47:10.376357 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:10.376317    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-l5b9d"]
Oct 30 13:47:10.376896 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:10.376855    4364 cpu_manager.go:414] "containerMap: removing container" podUID="2ff3f00b-a2fc-461f-a23c-32cb08dc3225" containerName="revoke-keys"
Oct 30 13:47:10.376896 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:10.376870    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ff3f00b-a2fc-461f-a23c-32cb08dc3225" containerName="revoke-keys"
Oct 30 13:47:10.376991 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:10.376943    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ff3f00b-a2fc-461f-a23c-32cb08dc3225" containerName="revoke-keys"
Oct 30 13:47:10.380037 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:10.380022    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-l5b9d"
Oct 30 13:47:10.410645 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:10.410619    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\""
Oct 30 13:47:10.410792 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:10.410682    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:47:10.410792 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:10.410692    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:47:10.410792 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:10.410696    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:47:10.413100 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:10.413081    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-l5b9d"]
Oct 30 13:47:10.441587 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:10.441560    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v48j\" (UniqueName: \"kubernetes.io/projected/9d96f9cb-87aa-476c-be7e-e22ccb69c755-kube-api-access-9v48j\") pod \"maas-api-revoke-sub-c1e271a9-l5b9d\" (UID: \"9d96f9cb-87aa-476c-be7e-e22ccb69c755\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-l5b9d"
Oct 30 13:47:10.441714 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:10.441592    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d96f9cb-87aa-476c-be7e-e22ccb69c755-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-l5b9d\" (UID: \"9d96f9cb-87aa-476c-be7e-e22ccb69c755\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-l5b9d"
Oct 30 13:47:10.441758 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:10.441744    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d96f9cb-87aa-476c-be7e-e22ccb69c755-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-l5b9d\" (UID: \"9d96f9cb-87aa-476c-be7e-e22ccb69c755\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-l5b9d"
Oct 30 13:47:10.542590 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:10.542555    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v48j\" (UniqueName: \"kubernetes.io/projected/9d96f9cb-87aa-476c-be7e-e22ccb69c755-kube-api-access-9v48j\") pod \"maas-api-revoke-sub-c1e271a9-l5b9d\" (UID: \"9d96f9cb-87aa-476c-be7e-e22ccb69c755\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-l5b9d"
Oct 30 13:47:10.542794 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:10.542616    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d96f9cb-87aa-476c-be7e-e22ccb69c755-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-l5b9d\" (UID: \"9d96f9cb-87aa-476c-be7e-e22ccb69c755\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-l5b9d"
Oct 30 13:47:10.542794 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:10.542682    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d96f9cb-87aa-476c-be7e-e22ccb69c755-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-l5b9d\" (UID: \"9d96f9cb-87aa-476c-be7e-e22ccb69c755\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-l5b9d"
Oct 30 13:47:10.543273 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:10.543253    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d96f9cb-87aa-476c-be7e-e22ccb69c755-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-l5b9d\" (UID: \"9d96f9cb-87aa-476c-be7e-e22ccb69c755\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-l5b9d"
Oct 30 13:47:10.545095 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:10.545075    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d96f9cb-87aa-476c-be7e-e22ccb69c755-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-l5b9d\" (UID: \"9d96f9cb-87aa-476c-be7e-e22ccb69c755\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-l5b9d"
Oct 30 13:47:10.568433 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:10.568407    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v48j\" (UniqueName: \"kubernetes.io/projected/9d96f9cb-87aa-476c-be7e-e22ccb69c755-kube-api-access-9v48j\") pod \"maas-api-revoke-sub-c1e271a9-l5b9d\" (UID: \"9d96f9cb-87aa-476c-be7e-e22ccb69c755\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-l5b9d"
Oct 30 13:47:10.692619 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:10.692520    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-l5b9d"
Oct 30 13:47:10.993400 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:10.993368    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-l5b9d"]
Oct 30 13:47:10.994279 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:47:10.994252    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d96f9cb_87aa_476c_be7e_e22ccb69c755.slice/crio-98fa2df69ccaf1bcb5fecd5a8ac7ea7980ec7b3bfec6bd635d2c9ab30a709cdf WatchSource:0}: Error finding container 98fa2df69ccaf1bcb5fecd5a8ac7ea7980ec7b3bfec6bd635d2c9ab30a709cdf: Status 404 returned error can't find the container with id 98fa2df69ccaf1bcb5fecd5a8ac7ea7980ec7b3bfec6bd635d2c9ab30a709cdf
Oct 30 13:47:11.476565 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:11.476511    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-l5b9d" event={"ID":"9d96f9cb-87aa-476c-be7e-e22ccb69c755","Type":"ContainerStarted","Data":"7bfa11d58ba7a4a5758e480cb1c8ddc5de8629c372936fc40290f7c81d45f2cc"}
Oct 30 13:47:11.476968 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:11.476585    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-l5b9d" event={"ID":"9d96f9cb-87aa-476c-be7e-e22ccb69c755","Type":"ContainerStarted","Data":"98fa2df69ccaf1bcb5fecd5a8ac7ea7980ec7b3bfec6bd635d2c9ab30a709cdf"}
Oct 30 13:47:11.502959 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:47:11.502925    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:47:11.524296 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:11.524248    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-l5b9d" podStartSLOduration=1.52423553 podStartE2EDuration="1.52423553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:47:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:47:11.520722371 +0000 UTC m=+3330.687087019" watchObservedRunningTime="2026-10-30 13:47:11.52423553 +0000 UTC m=+3330.690600206"
Oct 30 13:47:12.484366 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:12.484325    4364 generic.go:356] "Generic (PLEG): container finished" podID="9d96f9cb-87aa-476c-be7e-e22ccb69c755" containerID="7bfa11d58ba7a4a5758e480cb1c8ddc5de8629c372936fc40290f7c81d45f2cc" exitCode=0
Oct 30 13:47:12.484798 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:12.484400    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-l5b9d" event={"ID":"9d96f9cb-87aa-476c-be7e-e22ccb69c755","Type":"ContainerDied","Data":"7bfa11d58ba7a4a5758e480cb1c8ddc5de8629c372936fc40290f7c81d45f2cc"}
Oct 30 13:47:13.621813 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:13.621788    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-l5b9d"
Oct 30 13:47:13.673438 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:13.673409    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d96f9cb-87aa-476c-be7e-e22ccb69c755-kube-api-access-9v48j\" (UniqueName: \"kubernetes.io/projected/9d96f9cb-87aa-476c-be7e-e22ccb69c755-kube-api-access-9v48j\") pod \"9d96f9cb-87aa-476c-be7e-e22ccb69c755\" (UID: \"9d96f9cb-87aa-476c-be7e-e22ccb69c755\") "
Oct 30 13:47:13.673582 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:13.673454    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d96f9cb-87aa-476c-be7e-e22ccb69c755-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d96f9cb-87aa-476c-be7e-e22ccb69c755-maas-api-openshift-ca\") pod \"9d96f9cb-87aa-476c-be7e-e22ccb69c755\" (UID: \"9d96f9cb-87aa-476c-be7e-e22ccb69c755\") "
Oct 30 13:47:13.673582 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:13.673476    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d96f9cb-87aa-476c-be7e-e22ccb69c755-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d96f9cb-87aa-476c-be7e-e22ccb69c755-maas-api-xks-ca\") pod \"9d96f9cb-87aa-476c-be7e-e22ccb69c755\" (UID: \"9d96f9cb-87aa-476c-be7e-e22ccb69c755\") "
Oct 30 13:47:13.673854 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:13.673828    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d96f9cb-87aa-476c-be7e-e22ccb69c755-maas-api-openshift-ca" pod "9d96f9cb-87aa-476c-be7e-e22ccb69c755" (UID: "9d96f9cb-87aa-476c-be7e-e22ccb69c755"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:47:13.675593 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:13.675561    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d96f9cb-87aa-476c-be7e-e22ccb69c755-maas-api-xks-ca" pod "9d96f9cb-87aa-476c-be7e-e22ccb69c755" (UID: "9d96f9cb-87aa-476c-be7e-e22ccb69c755"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:47:13.675766 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:13.675663    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d96f9cb-87aa-476c-be7e-e22ccb69c755-kube-api-access-9v48j" pod "9d96f9cb-87aa-476c-be7e-e22ccb69c755" (UID: "9d96f9cb-87aa-476c-be7e-e22ccb69c755"). InnerVolumeSpecName "kube-api-access-9v48j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:47:13.774151 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:13.774082    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9v48j\" (UniqueName: \"kubernetes.io/projected/9d96f9cb-87aa-476c-be7e-e22ccb69c755-kube-api-access-9v48j\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:47:13.774151 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:13.774109    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d96f9cb-87aa-476c-be7e-e22ccb69c755-maas-api-openshift-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:47:13.774151 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:13.774119    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d96f9cb-87aa-476c-be7e-e22ccb69c755-maas-api-xks-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:47:14.495836 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:14.495804    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-l5b9d"
Oct 30 13:47:14.495836 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:14.495832    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-l5b9d" event={"ID":"9d96f9cb-87aa-476c-be7e-e22ccb69c755","Type":"ContainerDied","Data":"98fa2df69ccaf1bcb5fecd5a8ac7ea7980ec7b3bfec6bd635d2c9ab30a709cdf"}
Oct 30 13:47:14.496060 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:14.495877    4364 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98fa2df69ccaf1bcb5fecd5a8ac7ea7980ec7b3bfec6bd635d2c9ab30a709cdf"
Oct 30 13:47:21.563773 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:21.563738    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:47:25.635244 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:25.635205    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2vqwj"]
Oct 30 13:47:25.635845 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:25.635789    4364 cpu_manager.go:414] "containerMap: removing container" podUID="9d96f9cb-87aa-476c-be7e-e22ccb69c755" containerName="revoke-keys"
Oct 30 13:47:25.635845 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:25.635809    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d96f9cb-87aa-476c-be7e-e22ccb69c755" containerName="revoke-keys"
Oct 30 13:47:25.635973 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:25.635929    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d96f9cb-87aa-476c-be7e-e22ccb69c755" containerName="revoke-keys"
Oct 30 13:47:25.638994 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:25.638974    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2vqwj"
Oct 30 13:47:25.665360 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:25.665314    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2vqwj"]
Oct 30 13:47:25.678415 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:25.678388    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vt6z4"]
Oct 30 13:47:25.681852 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:25.681800    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vt6z4"
Oct 30 13:47:25.702522 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:25.702493    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vt6z4"]
Oct 30 13:47:25.784983 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:25.784952    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adbce7c5-af05-45e2-982c-183972ed458d-istio-token\") pod \"maas-default-gateway-openshift-default-64d4cc998c-vt6z4\" (UID: \"adbce7c5-af05-45e2-982c-183972ed458d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vt6z4"
Oct 30 13:47:25.785152 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:25.784993    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd6ee897-425a-4e17-978d-8afde52b1331-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-64d4cc998c-2vqwj\" (UID: \"fd6ee897-425a-4e17-978d-8afde52b1331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2vqwj"
Oct 30 13:47:25.785152 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:25.785015    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8gqn\" (UniqueName: \"kubernetes.io/projected/adbce7c5-af05-45e2-982c-183972ed458d-kube-api-access-g8gqn\") pod \"maas-default-gateway-openshift-default-64d4cc998c-vt6z4\" (UID: \"adbce7c5-af05-45e2-982c-183972ed458d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vt6z4"
Oct 30 13:47:25.785152 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:25.785095    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adbce7c5-af05-45e2-982c-183972ed458d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-64d4cc998c-vt6z4\" (UID: \"adbce7c5-af05-45e2-982c-183972ed458d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vt6z4"
Oct 30 13:47:25.785152 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:25.785143    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adbce7c5-af05-45e2-982c-183972ed458d-workload-certs\") pod \"maas-default-gateway-openshift-default-64d4cc998c-vt6z4\" (UID: \"adbce7c5-af05-45e2-982c-183972ed458d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vt6z4"
Oct 30 13:47:25.785343 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:25.785175    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd6ee897-425a-4e17-978d-8afde52b1331-istio-data\") pod \"maas-default-gateway-openshift-default-64d4cc998c-2vqwj\" (UID: \"fd6ee897-425a-4e17-978d-8afde52b1331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2vqwj"
Oct 30 13:47:25.785343 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:25.785197    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd6ee897-425a-4e17-978d-8afde52b1331-istio-podinfo\") pod \"maas-default-gateway-openshift-default-64d4cc998c-2vqwj\" (UID: \"fd6ee897-425a-4e17-978d-8afde52b1331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2vqwj"
Oct 30 13:47:25.785343 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:25.785216    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd6ee897-425a-4e17-978d-8afde52b1331-credential-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-2vqwj\" (UID: \"fd6ee897-425a-4e17-978d-8afde52b1331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2vqwj"
Oct 30 13:47:25.785343 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:25.785302    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddxsg\" (UniqueName: \"kubernetes.io/projected/fd6ee897-425a-4e17-978d-8afde52b1331-kube-api-access-ddxsg\") pod \"maas-default-gateway-openshift-default-64d4cc998c-2vqwj\" (UID: \"fd6ee897-425a-4e17-978d-8afde52b1331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2vqwj"
Oct 30 13:47:25.785552 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:25.785368    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd6ee897-425a-4e17-978d-8afde52b1331-workload-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-2vqwj\" (UID: \"fd6ee897-425a-4e17-978d-8afde52b1331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2vqwj"
Oct 30 13:47:25.785552 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:25.785393    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adbce7c5-af05-45e2-982c-183972ed458d-credential-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-vt6z4\" (UID: \"adbce7c5-af05-45e2-982c-183972ed458d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vt6z4"
Oct 30 13:47:25.785552 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:25.785417    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adbce7c5-af05-45e2-982c-183972ed458d-istio-envoy\") pod \"maas-default-gateway-openshift-default-64d4cc998c-vt6z4\" (UID: \"adbce7c5-af05-45e2-982c-183972ed458d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vt6z4"
Oct 30 13:47:25.785552 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:25.785433    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adbce7c5-af05-45e2-982c-183972ed458d-workload-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-vt6z4\" (UID: \"adbce7c5-af05-45e2-982c-183972ed458d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vt6z4"
Oct 30 13:47:25.785552 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:25.785499    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd6ee897-425a-4e17-978d-8afde52b1331-istio-envoy\") pod \"maas-default-gateway-openshift-default-64d4cc998c-2vqwj\" (UID: \"fd6ee897-425a-4e17-978d-8afde52b1331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2vqwj"
Oct 30 13:47:25.785795 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:25.785639    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adbce7c5-af05-45e2-982c-183972ed458d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-64d4cc998c-vt6z4\" (UID: \"adbce7c5-af05-45e2-982c-183972ed458d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vt6z4"
Oct 30 13:47:25.785795 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:25.785666    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd6ee897-425a-4e17-978d-8afde52b1331-workload-certs\") pod \"maas-default-gateway-openshift-default-64d4cc998c-2vqwj\" (UID: \"fd6ee897-425a-4e17-978d-8afde52b1331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2vqwj"
Oct 30 13:47:25.785795 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:25.785684    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adbce7c5-af05-45e2-982c-183972ed458d-istio-data\") pod \"maas-default-gateway-openshift-default-64d4cc998c-vt6z4\" (UID: \"adbce7c5-af05-45e2-982c-183972ed458d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vt6z4"
Oct 30 13:47:25.785795 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:25.785725    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd6ee897-425a-4e17-978d-8afde52b1331-istio-token\") pod \"maas-default-gateway-openshift-default-64d4cc998c-2vqwj\" (UID: \"fd6ee897-425a-4e17-978d-8afde52b1331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2vqwj"
Oct 30 13:47:25.886527 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:25.886444    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adbce7c5-af05-45e2-982c-183972ed458d-istio-token\") pod \"maas-default-gateway-openshift-default-64d4cc998c-vt6z4\" (UID: \"adbce7c5-af05-45e2-982c-183972ed458d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vt6z4"
Oct 30 13:47:25.886527 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:25.886499    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd6ee897-425a-4e17-978d-8afde52b1331-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-64d4cc998c-2vqwj\" (UID: \"fd6ee897-425a-4e17-978d-8afde52b1331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2vqwj"
Oct 30 13:47:25.886807 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:25.886534    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8gqn\" (UniqueName: \"kubernetes.io/projected/adbce7c5-af05-45e2-982c-183972ed458d-kube-api-access-g8gqn\") pod \"maas-default-gateway-openshift-default-64d4cc998c-vt6z4\" (UID: \"adbce7c5-af05-45e2-982c-183972ed458d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vt6z4"
Oct 30 13:47:25.886807 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:25.886566    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adbce7c5-af05-45e2-982c-183972ed458d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-64d4cc998c-vt6z4\" (UID: \"adbce7c5-af05-45e2-982c-183972ed458d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vt6z4"
Oct 30 13:47:25.886807 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:25.886637    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adbce7c5-af05-45e2-982c-183972ed458d-workload-certs\") pod \"maas-default-gateway-openshift-default-64d4cc998c-vt6z4\" (UID: \"adbce7c5-af05-45e2-982c-183972ed458d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vt6z4"
Oct 30 13:47:25.886807 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:25.886681    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd6ee897-425a-4e17-978d-8afde52b1331-istio-data\") pod \"maas-default-gateway-openshift-default-64d4cc998c-2vqwj\" (UID: \"fd6ee897-425a-4e17-978d-8afde52b1331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2vqwj"
Oct 30 13:47:25.886807 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:25.886718    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd6ee897-425a-4e17-978d-8afde52b1331-istio-podinfo\") pod \"maas-default-gateway-openshift-default-64d4cc998c-2vqwj\" (UID: \"fd6ee897-425a-4e17-978d-8afde52b1331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2vqwj"
Oct 30 13:47:25.886807 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:25.886749    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd6ee897-425a-4e17-978d-8afde52b1331-credential-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-2vqwj\" (UID: \"fd6ee897-425a-4e17-978d-8afde52b1331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2vqwj"
Oct 30 13:47:25.886807 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:25.886778    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddxsg\" (UniqueName: \"kubernetes.io/projected/fd6ee897-425a-4e17-978d-8afde52b1331-kube-api-access-ddxsg\") pod \"maas-default-gateway-openshift-default-64d4cc998c-2vqwj\" (UID: \"fd6ee897-425a-4e17-978d-8afde52b1331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2vqwj"
Oct 30 13:47:25.886807 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:25.886808    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd6ee897-425a-4e17-978d-8afde52b1331-workload-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-2vqwj\" (UID: \"fd6ee897-425a-4e17-978d-8afde52b1331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2vqwj"
Oct 30 13:47:25.887250 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:25.886850    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adbce7c5-af05-45e2-982c-183972ed458d-credential-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-vt6z4\" (UID: \"adbce7c5-af05-45e2-982c-183972ed458d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vt6z4"
Oct 30 13:47:25.887250 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:25.886893    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adbce7c5-af05-45e2-982c-183972ed458d-istio-envoy\") pod \"maas-default-gateway-openshift-default-64d4cc998c-vt6z4\" (UID: \"adbce7c5-af05-45e2-982c-183972ed458d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vt6z4"
Oct 30 13:47:25.887250 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:25.886923    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adbce7c5-af05-45e2-982c-183972ed458d-workload-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-vt6z4\" (UID: \"adbce7c5-af05-45e2-982c-183972ed458d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vt6z4"
Oct 30 13:47:25.887250 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:25.886951    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd6ee897-425a-4e17-978d-8afde52b1331-istio-envoy\") pod \"maas-default-gateway-openshift-default-64d4cc998c-2vqwj\" (UID: \"fd6ee897-425a-4e17-978d-8afde52b1331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2vqwj"
Oct 30 13:47:25.887250 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:25.887021    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adbce7c5-af05-45e2-982c-183972ed458d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-64d4cc998c-vt6z4\" (UID: \"adbce7c5-af05-45e2-982c-183972ed458d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vt6z4"
Oct 30 13:47:25.887250 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:25.887061    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd6ee897-425a-4e17-978d-8afde52b1331-workload-certs\") pod \"maas-default-gateway-openshift-default-64d4cc998c-2vqwj\" (UID: \"fd6ee897-425a-4e17-978d-8afde52b1331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2vqwj"
Oct 30 13:47:25.887250 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:25.887082    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adbce7c5-af05-45e2-982c-183972ed458d-istio-data\") pod \"maas-default-gateway-openshift-default-64d4cc998c-vt6z4\" (UID: \"adbce7c5-af05-45e2-982c-183972ed458d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vt6z4"
Oct 30 13:47:25.887250 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:25.887118    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd6ee897-425a-4e17-978d-8afde52b1331-istio-token\") pod \"maas-default-gateway-openshift-default-64d4cc998c-2vqwj\" (UID: \"fd6ee897-425a-4e17-978d-8afde52b1331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2vqwj"
Oct 30 13:47:25.887250 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:25.887116    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd6ee897-425a-4e17-978d-8afde52b1331-istio-data\") pod \"maas-default-gateway-openshift-default-64d4cc998c-2vqwj\" (UID: \"fd6ee897-425a-4e17-978d-8afde52b1331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2vqwj"
Oct 30 13:47:25.887250 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:25.887117    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adbce7c5-af05-45e2-982c-183972ed458d-workload-certs\") pod \"maas-default-gateway-openshift-default-64d4cc998c-vt6z4\" (UID: \"adbce7c5-af05-45e2-982c-183972ed458d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vt6z4"
Oct 30 13:47:25.887991 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:25.887271    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd6ee897-425a-4e17-978d-8afde52b1331-workload-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-2vqwj\" (UID: \"fd6ee897-425a-4e17-978d-8afde52b1331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2vqwj"
Oct 30 13:47:25.887991 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:25.887328    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd6ee897-425a-4e17-978d-8afde52b1331-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-64d4cc998c-2vqwj\" (UID: \"fd6ee897-425a-4e17-978d-8afde52b1331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2vqwj"
Oct 30 13:47:25.887991 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:25.887429    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adbce7c5-af05-45e2-982c-183972ed458d-istio-data\") pod \"maas-default-gateway-openshift-default-64d4cc998c-vt6z4\" (UID: \"adbce7c5-af05-45e2-982c-183972ed458d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vt6z4"
Oct 30 13:47:25.887991 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:25.887450    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd6ee897-425a-4e17-978d-8afde52b1331-workload-certs\") pod \"maas-default-gateway-openshift-default-64d4cc998c-2vqwj\" (UID: \"fd6ee897-425a-4e17-978d-8afde52b1331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2vqwj"
Oct 30 13:47:25.887991 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:25.887467    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adbce7c5-af05-45e2-982c-183972ed458d-credential-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-vt6z4\" (UID: \"adbce7c5-af05-45e2-982c-183972ed458d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vt6z4"
Oct 30 13:47:25.887991 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:25.887687    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd6ee897-425a-4e17-978d-8afde52b1331-credential-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-2vqwj\" (UID: \"fd6ee897-425a-4e17-978d-8afde52b1331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2vqwj"
Oct 30 13:47:25.887991 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:25.887739    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adbce7c5-af05-45e2-982c-183972ed458d-workload-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-vt6z4\" (UID: \"adbce7c5-af05-45e2-982c-183972ed458d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vt6z4"
Oct 30 13:47:25.888445 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:25.888424    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adbce7c5-af05-45e2-982c-183972ed458d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-64d4cc998c-vt6z4\" (UID: \"adbce7c5-af05-45e2-982c-183972ed458d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vt6z4"
Oct 30 13:47:25.889505 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:25.889477    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd6ee897-425a-4e17-978d-8afde52b1331-istio-envoy\") pod \"maas-default-gateway-openshift-default-64d4cc998c-2vqwj\" (UID: \"fd6ee897-425a-4e17-978d-8afde52b1331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2vqwj"
Oct 30 13:47:25.889633 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:25.889559    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adbce7c5-af05-45e2-982c-183972ed458d-istio-envoy\") pod \"maas-default-gateway-openshift-default-64d4cc998c-vt6z4\" (UID: \"adbce7c5-af05-45e2-982c-183972ed458d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vt6z4"
Oct 30 13:47:25.889727 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:25.889712    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adbce7c5-af05-45e2-982c-183972ed458d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-64d4cc998c-vt6z4\" (UID: \"adbce7c5-af05-45e2-982c-183972ed458d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vt6z4"
Oct 30 13:47:25.890462 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:25.890442    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd6ee897-425a-4e17-978d-8afde52b1331-istio-podinfo\") pod \"maas-default-gateway-openshift-default-64d4cc998c-2vqwj\" (UID: \"fd6ee897-425a-4e17-978d-8afde52b1331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2vqwj"
Oct 30 13:47:25.915840 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:25.915816    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adbce7c5-af05-45e2-982c-183972ed458d-istio-token\") pod \"maas-default-gateway-openshift-default-64d4cc998c-vt6z4\" (UID: \"adbce7c5-af05-45e2-982c-183972ed458d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vt6z4"
Oct 30 13:47:25.923113 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:25.923087    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8gqn\" (UniqueName: \"kubernetes.io/projected/adbce7c5-af05-45e2-982c-183972ed458d-kube-api-access-g8gqn\") pod \"maas-default-gateway-openshift-default-64d4cc998c-vt6z4\" (UID: \"adbce7c5-af05-45e2-982c-183972ed458d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vt6z4"
Oct 30 13:47:25.923242 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:25.923222    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddxsg\" (UniqueName: \"kubernetes.io/projected/fd6ee897-425a-4e17-978d-8afde52b1331-kube-api-access-ddxsg\") pod \"maas-default-gateway-openshift-default-64d4cc998c-2vqwj\" (UID: \"fd6ee897-425a-4e17-978d-8afde52b1331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2vqwj"
Oct 30 13:47:25.932700 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:25.932681    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd6ee897-425a-4e17-978d-8afde52b1331-istio-token\") pod \"maas-default-gateway-openshift-default-64d4cc998c-2vqwj\" (UID: \"fd6ee897-425a-4e17-978d-8afde52b1331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2vqwj"
Oct 30 13:47:25.952040 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:25.952017    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2vqwj"
Oct 30 13:47:26.002425 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:26.002095    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vt6z4"
Oct 30 13:47:26.048575 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:26.048525    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:47:26.268369 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:26.268335    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2vqwj"]
Oct 30 13:47:26.269315 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:47:26.269289    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd6ee897_425a_4e17_978d_8afde52b1331.slice/crio-531b19681e8d74ed5e3e49750b265b59c6ace2b3699eb3cece02fb95e999d899 WatchSource:0}: Error finding container 531b19681e8d74ed5e3e49750b265b59c6ace2b3699eb3cece02fb95e999d899: Status 404 returned error can't find the container with id 531b19681e8d74ed5e3e49750b265b59c6ace2b3699eb3cece02fb95e999d899
Oct 30 13:47:26.274954 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:26.274920    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:47:26.275035 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:26.274998    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:47:26.275035 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:26.275028    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:47:26.318472 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:26.318444    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vt6z4"]
Oct 30 13:47:26.324572 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:47:26.321060    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadbce7c5_af05_45e2_982c_183972ed458d.slice/crio-35903907ca804b6624256dcae126c12eeab88b357f565212f9baa864c8c8f12e WatchSource:0}: Error finding container 35903907ca804b6624256dcae126c12eeab88b357f565212f9baa864c8c8f12e: Status 404 returned error can't find the container with id 35903907ca804b6624256dcae126c12eeab88b357f565212f9baa864c8c8f12e
Oct 30 13:47:26.324572 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:26.323920    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:47:26.324572 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:26.324012    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:47:26.324572 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:26.324067    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:47:26.561094 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:26.560914    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2vqwj" event={"ID":"fd6ee897-425a-4e17-978d-8afde52b1331","Type":"ContainerStarted","Data":"cdeeeb331fc6ad01fa29474cbb31e9a19c09a647d933fcef9577c4ef25df51da"}
Oct 30 13:47:26.561094 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:26.560979    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2vqwj" event={"ID":"fd6ee897-425a-4e17-978d-8afde52b1331","Type":"ContainerStarted","Data":"531b19681e8d74ed5e3e49750b265b59c6ace2b3699eb3cece02fb95e999d899"}
Oct 30 13:47:26.562982 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:26.562953    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vt6z4" event={"ID":"adbce7c5-af05-45e2-982c-183972ed458d","Type":"ContainerStarted","Data":"d5945d049be72774e1202e1e93c42abcf4f0bd84833967d70a6b6a000bb56728"}
Oct 30 13:47:26.563107 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:26.563008    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vt6z4" event={"ID":"adbce7c5-af05-45e2-982c-183972ed458d","Type":"ContainerStarted","Data":"35903907ca804b6624256dcae126c12eeab88b357f565212f9baa864c8c8f12e"}
Oct 30 13:47:26.617265 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:26.617198    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2vqwj" podStartSLOduration=1.617179755 podStartE2EDuration="1.617179755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:47:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:47:26.61680175 +0000 UTC m=+3345.783166400" watchObservedRunningTime="2026-10-30 13:47:26.617179755 +0000 UTC m=+3345.783544409"
Oct 30 13:47:26.666289 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:26.666211    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vt6z4" podStartSLOduration=1.666191319 podStartE2EDuration="1.666191319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:47:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:47:26.664841573 +0000 UTC m=+3345.831206223" watchObservedRunningTime="2026-10-30 13:47:26.666191319 +0000 UTC m=+3345.832555970"
Oct 30 13:47:26.952757 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:26.952694    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2vqwj"
Oct 30 13:47:27.010622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:27.010041    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vt6z4"
Oct 30 13:47:27.019622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:27.018757    4364 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-vt6z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused" start-of-body=
Oct 30 13:47:27.019622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:27.018854    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vt6z4" podUID="adbce7c5-af05-45e2-982c-183972ed458d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused"
Oct 30 13:47:27.954743 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:27.954363    4364 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-2vqwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body=
Oct 30 13:47:27.954743 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:27.954431    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2vqwj" podUID="fd6ee897-425a-4e17-978d-8afde52b1331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused"
Oct 30 13:47:28.010237 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:28.010198    4364 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-vt6z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused" start-of-body=
Oct 30 13:47:28.010560 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:28.010528    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vt6z4" podUID="adbce7c5-af05-45e2-982c-183972ed458d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused"
Oct 30 13:47:28.953082 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:28.953040    4364 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-2vqwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body=
Oct 30 13:47:28.953447 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:28.953409    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2vqwj" podUID="fd6ee897-425a-4e17-978d-8afde52b1331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused"
Oct 30 13:47:29.003623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:29.003058    4364 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-vt6z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused" start-of-body=
Oct 30 13:47:29.003623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:29.003142    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vt6z4" podUID="adbce7c5-af05-45e2-982c-183972ed458d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused"
Oct 30 13:47:29.952932 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:29.952886    4364 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-2vqwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body=
Oct 30 13:47:29.953131 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:29.952955    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2vqwj" podUID="fd6ee897-425a-4e17-978d-8afde52b1331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused"
Oct 30 13:47:30.006622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:30.006034    4364 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-vt6z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused" start-of-body=
Oct 30 13:47:30.006622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:30.006098    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vt6z4" podUID="adbce7c5-af05-45e2-982c-183972ed458d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused"
Oct 30 13:47:30.953275 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:30.953242    4364 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-2vqwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body=
Oct 30 13:47:30.953474 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:30.953295    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2vqwj" podUID="fd6ee897-425a-4e17-978d-8afde52b1331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused"
Oct 30 13:47:31.006623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:31.003968    4364 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-vt6z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused" start-of-body=
Oct 30 13:47:31.006623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:31.004044    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vt6z4" podUID="adbce7c5-af05-45e2-982c-183972ed458d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused"
Oct 30 13:47:31.953243 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:31.953197    4364 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-2vqwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body=
Oct 30 13:47:31.953469 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:31.953270    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2vqwj" podUID="fd6ee897-425a-4e17-978d-8afde52b1331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused"
Oct 30 13:47:32.004622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:32.004032    4364 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-vt6z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused" start-of-body=
Oct 30 13:47:32.004622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:32.004107    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vt6z4" podUID="adbce7c5-af05-45e2-982c-183972ed458d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused"
Oct 30 13:47:32.952941 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:32.952892    4364 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-2vqwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body=
Oct 30 13:47:32.953449 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:32.952957    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2vqwj" podUID="fd6ee897-425a-4e17-978d-8afde52b1331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused"
Oct 30 13:47:33.003296 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:33.003264    4364 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-vt6z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused" start-of-body=
Oct 30 13:47:33.003469 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:33.003315    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vt6z4" podUID="adbce7c5-af05-45e2-982c-183972ed458d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.14:15021/healthz/ready\": dial tcp 10.133.1.14:15021: connect: connection refused"
Oct 30 13:47:34.684975 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:34.684939    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2vqwj"
Oct 30 13:47:34.685938 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:34.685911    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2vqwj"
Oct 30 13:47:34.687233 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:34.687209    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2vqwj"
Oct 30 13:47:34.776940 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:34.776911    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vt6z4"
Oct 30 13:47:34.777558 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:34.777534    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vt6z4"
Oct 30 13:47:34.778470 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:34.778428    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vt6z4"
Oct 30 13:47:41.503919 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:47:41.503883    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:47:41.882413 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:41.882378    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:47:57.484033 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:57.483951    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:47:59.897052 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:59.896688    4364 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-tbw9c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:47:59.897052 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:47:59.896765    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tbw9c" podUID="e40d7a56-5869-4a42-b8a7-33c02604c302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:48:11.503953 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:48:11.503917    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:48:20.993160 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:48:20.993127    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:48:31.394991 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:48:31.394956    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:48:35.572922 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:48:35.572886    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:48:41.504007 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:48:41.503966    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:48:52.884532 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:48:52.884496    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:48:57.723196 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:48:57.723155    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:49:11.503323 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:49:11.503271    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:49:14.500255 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:49:14.500211    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:49:19.084491 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:49:19.084457    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:49:35.597092 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:49:35.597057    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:49:41.505428 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:49:41.505386    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:49:52.773187 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:49:52.773143    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:50:02.030909 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:02.030874    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f86e315-sxk99"]
Oct 30 13:50:02.044743 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:02.044717    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f86e315-sxk99"]
Oct 30 13:50:02.968189 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:02.968152    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dead7294-e167-41ef-937e-21f038062b38" path="/var/lib/kubelet/pods/dead7294-e167-41ef-937e-21f038062b38/volumes"
Oct 30 13:50:03.580554 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:03.580511    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:50:08.694330 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:08.694294    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:50:09.034363 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:09.034278    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23bf79b7-9d2p7"]
Oct 30 13:50:09.048364 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:09.048331    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23bf79b7-9d2p7"]
Oct 30 13:50:09.547091 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:09.547056    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8xkj9"]
Oct 30 13:50:09.547275 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:09.547251    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8xkj9" podUID="01088007-6fcf-4bab-9f3a-9e0dce4e1cf6" containerName="discovery" containerID="cri-o://15f071a714a0aec14ed089363a7411622faff71531febff2ce91e066b7e1143d" gracePeriod=30
Oct 30 13:50:09.697845 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:09.697809    4364 generic.go:356] "Generic (PLEG): container finished" podID="01088007-6fcf-4bab-9f3a-9e0dce4e1cf6" containerID="15f071a714a0aec14ed089363a7411622faff71531febff2ce91e066b7e1143d" exitCode=0
Oct 30 13:50:09.698349 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:09.697880    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8xkj9" event={"ID":"01088007-6fcf-4bab-9f3a-9e0dce4e1cf6","Type":"ContainerDied","Data":"15f071a714a0aec14ed089363a7411622faff71531febff2ce91e066b7e1143d"}
Oct 30 13:50:09.804346 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:09.804271    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8xkj9"
Oct 30 13:50:09.936583 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:09.936543    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01088007-6fcf-4bab-9f3a-9e0dce4e1cf6-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/01088007-6fcf-4bab-9f3a-9e0dce4e1cf6-istio-csr-dns-cert\") pod \"01088007-6fcf-4bab-9f3a-9e0dce4e1cf6\" (UID: \"01088007-6fcf-4bab-9f3a-9e0dce4e1cf6\") "
Oct 30 13:50:09.936782 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:09.936655    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01088007-6fcf-4bab-9f3a-9e0dce4e1cf6-kube-api-access-phxx8\" (UniqueName: \"kubernetes.io/projected/01088007-6fcf-4bab-9f3a-9e0dce4e1cf6-kube-api-access-phxx8\") pod \"01088007-6fcf-4bab-9f3a-9e0dce4e1cf6\" (UID: \"01088007-6fcf-4bab-9f3a-9e0dce4e1cf6\") "
Oct 30 13:50:09.936782 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:09.936691    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01088007-6fcf-4bab-9f3a-9e0dce4e1cf6-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/01088007-6fcf-4bab-9f3a-9e0dce4e1cf6-local-certs\") pod \"01088007-6fcf-4bab-9f3a-9e0dce4e1cf6\" (UID: \"01088007-6fcf-4bab-9f3a-9e0dce4e1cf6\") "
Oct 30 13:50:09.936782 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:09.936740    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01088007-6fcf-4bab-9f3a-9e0dce4e1cf6-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/01088007-6fcf-4bab-9f3a-9e0dce4e1cf6-istio-kubeconfig\") pod \"01088007-6fcf-4bab-9f3a-9e0dce4e1cf6\" (UID: \"01088007-6fcf-4bab-9f3a-9e0dce4e1cf6\") "
Oct 30 13:50:09.936967 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:09.936783    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01088007-6fcf-4bab-9f3a-9e0dce4e1cf6-cacerts\" (UniqueName: \"kubernetes.io/secret/01088007-6fcf-4bab-9f3a-9e0dce4e1cf6-cacerts\") pod \"01088007-6fcf-4bab-9f3a-9e0dce4e1cf6\" (UID: \"01088007-6fcf-4bab-9f3a-9e0dce4e1cf6\") "
Oct 30 13:50:09.936967 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:09.936848    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01088007-6fcf-4bab-9f3a-9e0dce4e1cf6-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/01088007-6fcf-4bab-9f3a-9e0dce4e1cf6-istio-csr-ca-configmap\") pod \"01088007-6fcf-4bab-9f3a-9e0dce4e1cf6\" (UID: \"01088007-6fcf-4bab-9f3a-9e0dce4e1cf6\") "
Oct 30 13:50:09.936967 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:09.936910    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01088007-6fcf-4bab-9f3a-9e0dce4e1cf6-istio-token\" (UniqueName: \"kubernetes.io/projected/01088007-6fcf-4bab-9f3a-9e0dce4e1cf6-istio-token\") pod \"01088007-6fcf-4bab-9f3a-9e0dce4e1cf6\" (UID: \"01088007-6fcf-4bab-9f3a-9e0dce4e1cf6\") "
Oct 30 13:50:09.937320 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:09.937269    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01088007-6fcf-4bab-9f3a-9e0dce4e1cf6-istio-csr-ca-configmap" pod "01088007-6fcf-4bab-9f3a-9e0dce4e1cf6" (UID: "01088007-6fcf-4bab-9f3a-9e0dce4e1cf6"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:50:09.938996 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:09.938971    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01088007-6fcf-4bab-9f3a-9e0dce4e1cf6-local-certs" pod "01088007-6fcf-4bab-9f3a-9e0dce4e1cf6" (UID: "01088007-6fcf-4bab-9f3a-9e0dce4e1cf6"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:50:09.939173 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:09.939096    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01088007-6fcf-4bab-9f3a-9e0dce4e1cf6-istio-csr-dns-cert" pod "01088007-6fcf-4bab-9f3a-9e0dce4e1cf6" (UID: "01088007-6fcf-4bab-9f3a-9e0dce4e1cf6"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:50:09.939293 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:09.939273    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01088007-6fcf-4bab-9f3a-9e0dce4e1cf6-istio-kubeconfig" pod "01088007-6fcf-4bab-9f3a-9e0dce4e1cf6" (UID: "01088007-6fcf-4bab-9f3a-9e0dce4e1cf6"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:50:09.939372 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:09.939359    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01088007-6fcf-4bab-9f3a-9e0dce4e1cf6-cacerts" pod "01088007-6fcf-4bab-9f3a-9e0dce4e1cf6" (UID: "01088007-6fcf-4bab-9f3a-9e0dce4e1cf6"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:50:09.939491 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:09.939467    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01088007-6fcf-4bab-9f3a-9e0dce4e1cf6-kube-api-access-phxx8" pod "01088007-6fcf-4bab-9f3a-9e0dce4e1cf6" (UID: "01088007-6fcf-4bab-9f3a-9e0dce4e1cf6"). InnerVolumeSpecName "kube-api-access-phxx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:50:09.939544 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:09.939504    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01088007-6fcf-4bab-9f3a-9e0dce4e1cf6-istio-token" pod "01088007-6fcf-4bab-9f3a-9e0dce4e1cf6" (UID: "01088007-6fcf-4bab-9f3a-9e0dce4e1cf6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:50:10.037947 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:10.037912    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phxx8\" (UniqueName: \"kubernetes.io/projected/01088007-6fcf-4bab-9f3a-9e0dce4e1cf6-kube-api-access-phxx8\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:50:10.037947 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:10.037945    4364 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/01088007-6fcf-4bab-9f3a-9e0dce4e1cf6-local-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:50:10.038174 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:10.037959    4364 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/01088007-6fcf-4bab-9f3a-9e0dce4e1cf6-istio-kubeconfig\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:50:10.038174 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:10.037971    4364 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/01088007-6fcf-4bab-9f3a-9e0dce4e1cf6-cacerts\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:50:10.038174 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:10.037983    4364 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/01088007-6fcf-4bab-9f3a-9e0dce4e1cf6-istio-csr-ca-configmap\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:50:10.038174 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:10.037994    4364 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01088007-6fcf-4bab-9f3a-9e0dce4e1cf6-istio-token\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:50:10.038174 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:10.038005    4364 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/01088007-6fcf-4bab-9f3a-9e0dce4e1cf6-istio-csr-dns-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:50:10.704033 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:10.704001    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8xkj9"
Oct 30 13:50:10.704546 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:10.704041    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8xkj9" event={"ID":"01088007-6fcf-4bab-9f3a-9e0dce4e1cf6","Type":"ContainerDied","Data":"75d0f52d2c0654143a597ff1f0bd703ad88a403dcdb5aebb14db9ac50ea9674e"}
Oct 30 13:50:10.704546 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:10.704111    4364 scope.go:122] "RemoveContainer" containerID="15f071a714a0aec14ed089363a7411622faff71531febff2ce91e066b7e1143d"
Oct 30 13:50:10.776361 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:10.776333    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8xkj9"]
Oct 30 13:50:10.790838 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:10.790807    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8xkj9"]
Oct 30 13:50:10.969503 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:10.969424    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01088007-6fcf-4bab-9f3a-9e0dce4e1cf6" path="/var/lib/kubelet/pods/01088007-6fcf-4bab-9f3a-9e0dce4e1cf6/volumes"
Oct 30 13:50:10.969850 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:10.969836    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93362c37-0fc6-48b2-85b8-ee62492ba31f" path="/var/lib/kubelet/pods/93362c37-0fc6-48b2-85b8-ee62492ba31f/volumes"
Oct 30 13:50:11.503952 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:50:11.503915    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:50:13.728385 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:13.728341    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-br444"]
Oct 30 13:50:13.728955 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:13.728933    4364 cpu_manager.go:414] "containerMap: removing container" podUID="01088007-6fcf-4bab-9f3a-9e0dce4e1cf6" containerName="discovery"
Oct 30 13:50:13.729046 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:13.728957    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01088007-6fcf-4bab-9f3a-9e0dce4e1cf6" containerName="discovery"
Oct 30 13:50:13.729106 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:13.729092    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="01088007-6fcf-4bab-9f3a-9e0dce4e1cf6" containerName="discovery"
Oct 30 13:50:13.733592 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:13.733570    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-br444"
Oct 30 13:50:13.754582 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:13.754557    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-br444"]
Oct 30 13:50:13.765914 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:13.765882    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:50:13.765914 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:13.765890    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\""
Oct 30 13:50:13.766098 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:13.765934    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:50:13.766836 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:13.766816    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:50:13.881531 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:13.881492    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2k9z\" (UniqueName: \"kubernetes.io/projected/cdd6cfdf-9657-4a87-8dcd-d9f406b30b54-kube-api-access-z2k9z\") pod \"maas-api-revoke-sub-3d4156f1-br444\" (UID: \"cdd6cfdf-9657-4a87-8dcd-d9f406b30b54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-br444"
Oct 30 13:50:13.881737 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:13.881673    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdd6cfdf-9657-4a87-8dcd-d9f406b30b54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-br444\" (UID: \"cdd6cfdf-9657-4a87-8dcd-d9f406b30b54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-br444"
Oct 30 13:50:13.881794 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:13.881759    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdd6cfdf-9657-4a87-8dcd-d9f406b30b54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-br444\" (UID: \"cdd6cfdf-9657-4a87-8dcd-d9f406b30b54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-br444"
Oct 30 13:50:13.982928 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:13.982838    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdd6cfdf-9657-4a87-8dcd-d9f406b30b54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-br444\" (UID: \"cdd6cfdf-9657-4a87-8dcd-d9f406b30b54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-br444"
Oct 30 13:50:13.982928 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:13.982880    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdd6cfdf-9657-4a87-8dcd-d9f406b30b54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-br444\" (UID: \"cdd6cfdf-9657-4a87-8dcd-d9f406b30b54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-br444"
Oct 30 13:50:13.983177 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:13.982941    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2k9z\" (UniqueName: \"kubernetes.io/projected/cdd6cfdf-9657-4a87-8dcd-d9f406b30b54-kube-api-access-z2k9z\") pod \"maas-api-revoke-sub-3d4156f1-br444\" (UID: \"cdd6cfdf-9657-4a87-8dcd-d9f406b30b54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-br444"
Oct 30 13:50:13.983580 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:13.983550    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdd6cfdf-9657-4a87-8dcd-d9f406b30b54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-br444\" (UID: \"cdd6cfdf-9657-4a87-8dcd-d9f406b30b54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-br444"
Oct 30 13:50:13.985235 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:13.985215    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdd6cfdf-9657-4a87-8dcd-d9f406b30b54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-br444\" (UID: \"cdd6cfdf-9657-4a87-8dcd-d9f406b30b54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-br444"
Oct 30 13:50:14.009239 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:14.009207    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2k9z\" (UniqueName: \"kubernetes.io/projected/cdd6cfdf-9657-4a87-8dcd-d9f406b30b54-kube-api-access-z2k9z\") pod \"maas-api-revoke-sub-3d4156f1-br444\" (UID: \"cdd6cfdf-9657-4a87-8dcd-d9f406b30b54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-br444"
Oct 30 13:50:14.046705 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:14.046678    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-br444"
Oct 30 13:50:14.348396 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:14.348359    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-br444"]
Oct 30 13:50:14.348937 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:50:14.348903    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdd6cfdf_9657_4a87_8dcd_d9f406b30b54.slice/crio-82409699586bdff121da7439e3a1554c37cecb7c57f8d57ef7c15b68b10b878b WatchSource:0}: Error finding container 82409699586bdff121da7439e3a1554c37cecb7c57f8d57ef7c15b68b10b878b: Status 404 returned error can't find the container with id 82409699586bdff121da7439e3a1554c37cecb7c57f8d57ef7c15b68b10b878b
Oct 30 13:50:14.728550 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:14.728508    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-br444" event={"ID":"cdd6cfdf-9657-4a87-8dcd-d9f406b30b54","Type":"ContainerStarted","Data":"abe532b80e4fa58c3f136370102ba5bc8e88164d333f9ef0cb5880ef4e26749e"}
Oct 30 13:50:14.728957 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:14.728571    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-br444" event={"ID":"cdd6cfdf-9657-4a87-8dcd-d9f406b30b54","Type":"ContainerStarted","Data":"82409699586bdff121da7439e3a1554c37cecb7c57f8d57ef7c15b68b10b878b"}
Oct 30 13:50:14.773829 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:14.773775    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-br444" podStartSLOduration=1.7737617879999998 podStartE2EDuration="1.773761788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:50:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:50:14.772708229 +0000 UTC m=+3513.939072876" watchObservedRunningTime="2026-10-30 13:50:14.773761788 +0000 UTC m=+3513.940126436"
Oct 30 13:50:15.736862 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:15.736827    4364 generic.go:356] "Generic (PLEG): container finished" podID="cdd6cfdf-9657-4a87-8dcd-d9f406b30b54" containerID="abe532b80e4fa58c3f136370102ba5bc8e88164d333f9ef0cb5880ef4e26749e" exitCode=0
Oct 30 13:50:15.737265 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:15.736912    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-br444" event={"ID":"cdd6cfdf-9657-4a87-8dcd-d9f406b30b54","Type":"ContainerDied","Data":"abe532b80e4fa58c3f136370102ba5bc8e88164d333f9ef0cb5880ef4e26749e"}
Oct 30 13:50:16.881899 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:16.881874    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-br444"
Oct 30 13:50:16.914084 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:16.914056    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdd6cfdf-9657-4a87-8dcd-d9f406b30b54-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdd6cfdf-9657-4a87-8dcd-d9f406b30b54-maas-api-openshift-ca\") pod \"cdd6cfdf-9657-4a87-8dcd-d9f406b30b54\" (UID: \"cdd6cfdf-9657-4a87-8dcd-d9f406b30b54\") "
Oct 30 13:50:16.914231 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:16.914105    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdd6cfdf-9657-4a87-8dcd-d9f406b30b54-kube-api-access-z2k9z\" (UniqueName: \"kubernetes.io/projected/cdd6cfdf-9657-4a87-8dcd-d9f406b30b54-kube-api-access-z2k9z\") pod \"cdd6cfdf-9657-4a87-8dcd-d9f406b30b54\" (UID: \"cdd6cfdf-9657-4a87-8dcd-d9f406b30b54\") "
Oct 30 13:50:16.914231 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:16.914187    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdd6cfdf-9657-4a87-8dcd-d9f406b30b54-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdd6cfdf-9657-4a87-8dcd-d9f406b30b54-maas-api-xks-ca\") pod \"cdd6cfdf-9657-4a87-8dcd-d9f406b30b54\" (UID: \"cdd6cfdf-9657-4a87-8dcd-d9f406b30b54\") "
Oct 30 13:50:16.914415 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:16.914394    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdd6cfdf-9657-4a87-8dcd-d9f406b30b54-maas-api-openshift-ca" pod "cdd6cfdf-9657-4a87-8dcd-d9f406b30b54" (UID: "cdd6cfdf-9657-4a87-8dcd-d9f406b30b54"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:50:16.914556 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:16.914530    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdd6cfdf-9657-4a87-8dcd-d9f406b30b54-maas-api-openshift-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:50:16.916253 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:16.916225    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdd6cfdf-9657-4a87-8dcd-d9f406b30b54-maas-api-xks-ca" pod "cdd6cfdf-9657-4a87-8dcd-d9f406b30b54" (UID: "cdd6cfdf-9657-4a87-8dcd-d9f406b30b54"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:50:16.916336 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:16.916257    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdd6cfdf-9657-4a87-8dcd-d9f406b30b54-kube-api-access-z2k9z" pod "cdd6cfdf-9657-4a87-8dcd-d9f406b30b54" (UID: "cdd6cfdf-9657-4a87-8dcd-d9f406b30b54"). InnerVolumeSpecName "kube-api-access-z2k9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:50:17.015621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:17.015519    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdd6cfdf-9657-4a87-8dcd-d9f406b30b54-maas-api-xks-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:50:17.015621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:17.015554    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2k9z\" (UniqueName: \"kubernetes.io/projected/cdd6cfdf-9657-4a87-8dcd-d9f406b30b54-kube-api-access-z2k9z\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:50:17.749287 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:17.749255    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-br444"
Oct 30 13:50:17.749287 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:17.749266    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-br444" event={"ID":"cdd6cfdf-9657-4a87-8dcd-d9f406b30b54","Type":"ContainerDied","Data":"82409699586bdff121da7439e3a1554c37cecb7c57f8d57ef7c15b68b10b878b"}
Oct 30 13:50:17.749522 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:17.749312    4364 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82409699586bdff121da7439e3a1554c37cecb7c57f8d57ef7c15b68b10b878b"
Oct 30 13:50:25.710479 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:25.710439    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:50:29.037586 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:29.037551    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9fa5c55-sndgt"]
Oct 30 13:50:29.051569 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:29.051536    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9fa5c55-sndgt"]
Oct 30 13:50:30.590445 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:30.590412    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:50:30.968785 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:30.968752    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e561630-bbec-40f3-a5c3-143f153cc443" path="/var/lib/kubelet/pods/6e561630-bbec-40f3-a5c3-143f153cc443/volumes"
Oct 30 13:50:37.628270 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:37.628235    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-br444"]
Oct 30 13:50:37.642403 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:37.642361    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-br444"]
Oct 30 13:50:37.674162 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:37.674121    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-65lbb"]
Oct 30 13:50:37.674633 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:37.674613    4364 cpu_manager.go:414] "containerMap: removing container" podUID="cdd6cfdf-9657-4a87-8dcd-d9f406b30b54" containerName="revoke-keys"
Oct 30 13:50:37.674750 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:37.674634    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdd6cfdf-9657-4a87-8dcd-d9f406b30b54" containerName="revoke-keys"
Oct 30 13:50:37.674811 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:37.674785    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdd6cfdf-9657-4a87-8dcd-d9f406b30b54" containerName="revoke-keys"
Oct 30 13:50:37.678982 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:37.678960    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-65lbb"
Oct 30 13:50:37.697232 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:37.697163    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:50:37.697232 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:37.697205    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:50:37.697394 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:37.697332    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\""
Oct 30 13:50:37.699336 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:37.699315    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-65lbb"]
Oct 30 13:50:37.709772 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:37.709735    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:50:37.816682 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:37.816647    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvzgd\" (UniqueName: \"kubernetes.io/projected/2641d4ad-4a62-41d9-9081-7dece03c3ae7-kube-api-access-fvzgd\") pod \"maas-api-revoke-sub-3d4156f1-65lbb\" (UID: \"2641d4ad-4a62-41d9-9081-7dece03c3ae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-65lbb"
Oct 30 13:50:37.816895 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:37.816703    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2641d4ad-4a62-41d9-9081-7dece03c3ae7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-65lbb\" (UID: \"2641d4ad-4a62-41d9-9081-7dece03c3ae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-65lbb"
Oct 30 13:50:37.816895 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:37.816848    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2641d4ad-4a62-41d9-9081-7dece03c3ae7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-65lbb\" (UID: \"2641d4ad-4a62-41d9-9081-7dece03c3ae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-65lbb"
Oct 30 13:50:37.918538 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:37.918447    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvzgd\" (UniqueName: \"kubernetes.io/projected/2641d4ad-4a62-41d9-9081-7dece03c3ae7-kube-api-access-fvzgd\") pod \"maas-api-revoke-sub-3d4156f1-65lbb\" (UID: \"2641d4ad-4a62-41d9-9081-7dece03c3ae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-65lbb"
Oct 30 13:50:37.918538 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:37.918520    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2641d4ad-4a62-41d9-9081-7dece03c3ae7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-65lbb\" (UID: \"2641d4ad-4a62-41d9-9081-7dece03c3ae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-65lbb"
Oct 30 13:50:37.918798 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:37.918564    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2641d4ad-4a62-41d9-9081-7dece03c3ae7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-65lbb\" (UID: \"2641d4ad-4a62-41d9-9081-7dece03c3ae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-65lbb"
Oct 30 13:50:37.919275 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:37.919254    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2641d4ad-4a62-41d9-9081-7dece03c3ae7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-65lbb\" (UID: \"2641d4ad-4a62-41d9-9081-7dece03c3ae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-65lbb"
Oct 30 13:50:37.920846 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:37.920828    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2641d4ad-4a62-41d9-9081-7dece03c3ae7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-65lbb\" (UID: \"2641d4ad-4a62-41d9-9081-7dece03c3ae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-65lbb"
Oct 30 13:50:37.950341 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:37.950307    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvzgd\" (UniqueName: \"kubernetes.io/projected/2641d4ad-4a62-41d9-9081-7dece03c3ae7-kube-api-access-fvzgd\") pod \"maas-api-revoke-sub-3d4156f1-65lbb\" (UID: \"2641d4ad-4a62-41d9-9081-7dece03c3ae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-65lbb"
Oct 30 13:50:37.991545 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:37.991499    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-65lbb"
Oct 30 13:50:38.305326 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:50:38.305290    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2641d4ad_4a62_41d9_9081_7dece03c3ae7.slice/crio-3db504919a34e9bf94afeab13a1b5318a581a7a96b3acdef02e8ed866ec8d936 WatchSource:0}: Error finding container 3db504919a34e9bf94afeab13a1b5318a581a7a96b3acdef02e8ed866ec8d936: Status 404 returned error can't find the container with id 3db504919a34e9bf94afeab13a1b5318a581a7a96b3acdef02e8ed866ec8d936
Oct 30 13:50:38.305467 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:38.305342    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-65lbb"]
Oct 30 13:50:38.865397 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:38.865292    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-65lbb" event={"ID":"2641d4ad-4a62-41d9-9081-7dece03c3ae7","Type":"ContainerStarted","Data":"e79f39a8d0d2dc6b5bfdbe7ebe965f7c58833e47f97d89ec6c481e98cee77158"}
Oct 30 13:50:38.865397 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:38.865356    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-65lbb" event={"ID":"2641d4ad-4a62-41d9-9081-7dece03c3ae7","Type":"ContainerStarted","Data":"3db504919a34e9bf94afeab13a1b5318a581a7a96b3acdef02e8ed866ec8d936"}
Oct 30 13:50:38.910147 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:38.910089    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-65lbb" podStartSLOduration=1.910074681 podStartE2EDuration="1.910074681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:50:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:50:38.909520585 +0000 UTC m=+3538.075885234" watchObservedRunningTime="2026-10-30 13:50:38.910074681 +0000 UTC m=+3538.076439328"
Oct 30 13:50:38.968991 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:38.968960    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdd6cfdf-9657-4a87-8dcd-d9f406b30b54" path="/var/lib/kubelet/pods/cdd6cfdf-9657-4a87-8dcd-d9f406b30b54/volumes"
Oct 30 13:50:39.871841 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:39.871807    4364 generic.go:356] "Generic (PLEG): container finished" podID="2641d4ad-4a62-41d9-9081-7dece03c3ae7" containerID="e79f39a8d0d2dc6b5bfdbe7ebe965f7c58833e47f97d89ec6c481e98cee77158" exitCode=0
Oct 30 13:50:39.872400 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:39.871889    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-65lbb" event={"ID":"2641d4ad-4a62-41d9-9081-7dece03c3ae7","Type":"ContainerDied","Data":"e79f39a8d0d2dc6b5bfdbe7ebe965f7c58833e47f97d89ec6c481e98cee77158"}
Oct 30 13:50:41.011496 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:41.011470    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-65lbb"
Oct 30 13:50:41.152817 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:41.152736    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2641d4ad-4a62-41d9-9081-7dece03c3ae7-kube-api-access-fvzgd\" (UniqueName: \"kubernetes.io/projected/2641d4ad-4a62-41d9-9081-7dece03c3ae7-kube-api-access-fvzgd\") pod \"2641d4ad-4a62-41d9-9081-7dece03c3ae7\" (UID: \"2641d4ad-4a62-41d9-9081-7dece03c3ae7\") "
Oct 30 13:50:41.152817 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:41.152786    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2641d4ad-4a62-41d9-9081-7dece03c3ae7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2641d4ad-4a62-41d9-9081-7dece03c3ae7-maas-api-xks-ca\") pod \"2641d4ad-4a62-41d9-9081-7dece03c3ae7\" (UID: \"2641d4ad-4a62-41d9-9081-7dece03c3ae7\") "
Oct 30 13:50:41.153075 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:41.152827    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2641d4ad-4a62-41d9-9081-7dece03c3ae7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2641d4ad-4a62-41d9-9081-7dece03c3ae7-maas-api-openshift-ca\") pod \"2641d4ad-4a62-41d9-9081-7dece03c3ae7\" (UID: \"2641d4ad-4a62-41d9-9081-7dece03c3ae7\") "
Oct 30 13:50:41.153224 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:41.153199    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2641d4ad-4a62-41d9-9081-7dece03c3ae7-maas-api-openshift-ca" pod "2641d4ad-4a62-41d9-9081-7dece03c3ae7" (UID: "2641d4ad-4a62-41d9-9081-7dece03c3ae7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:50:41.154916 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:41.154888    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2641d4ad-4a62-41d9-9081-7dece03c3ae7-maas-api-xks-ca" pod "2641d4ad-4a62-41d9-9081-7dece03c3ae7" (UID: "2641d4ad-4a62-41d9-9081-7dece03c3ae7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:50:41.155008 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:41.154914    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2641d4ad-4a62-41d9-9081-7dece03c3ae7-kube-api-access-fvzgd" pod "2641d4ad-4a62-41d9-9081-7dece03c3ae7" (UID: "2641d4ad-4a62-41d9-9081-7dece03c3ae7"). InnerVolumeSpecName "kube-api-access-fvzgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:50:41.253863 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:41.253835    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2641d4ad-4a62-41d9-9081-7dece03c3ae7-maas-api-openshift-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:50:41.253863 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:41.253861    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvzgd\" (UniqueName: \"kubernetes.io/projected/2641d4ad-4a62-41d9-9081-7dece03c3ae7-kube-api-access-fvzgd\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:50:41.254035 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:41.253871    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2641d4ad-4a62-41d9-9081-7dece03c3ae7-maas-api-xks-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:50:41.504532 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:50:41.504451    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:50:41.883454 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:41.883421    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-65lbb"
Oct 30 13:50:41.883647 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:41.883455    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-65lbb" event={"ID":"2641d4ad-4a62-41d9-9081-7dece03c3ae7","Type":"ContainerDied","Data":"3db504919a34e9bf94afeab13a1b5318a581a7a96b3acdef02e8ed866ec8d936"}
Oct 30 13:50:41.883647 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:41.883499    4364 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3db504919a34e9bf94afeab13a1b5318a581a7a96b3acdef02e8ed866ec8d936"
Oct 30 13:50:49.748107 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:49.748072    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:50:54.528066 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:50:54.527976    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:51:00.072984 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:00.072948    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9"]
Oct 30 13:51:00.073624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:00.073587    4364 cpu_manager.go:414] "containerMap: removing container" podUID="2641d4ad-4a62-41d9-9081-7dece03c3ae7" containerName="revoke-keys"
Oct 30 13:51:00.073624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:00.073623    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2641d4ad-4a62-41d9-9081-7dece03c3ae7" containerName="revoke-keys"
Oct 30 13:51:00.073783 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:00.073698    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="2641d4ad-4a62-41d9-9081-7dece03c3ae7" containerName="revoke-keys"
Oct 30 13:51:00.076899 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:00.076880    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9"
Oct 30 13:51:00.096646 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:00.096624    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-a5b3f0-openshift-default-dockercfg-k2hbc\""
Oct 30 13:51:00.111739 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:00.111713    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9"]
Oct 30 13:51:00.139483 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:00.139455    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72-istio-podinfo\") pod \"e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9\" (UID: \"3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72\") " pod="openshift-ingress/e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9"
Oct 30 13:51:00.139625 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:00.139494    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htmcl\" (UniqueName: \"kubernetes.io/projected/3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72-kube-api-access-htmcl\") pod \"e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9\" (UID: \"3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72\") " pod="openshift-ingress/e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9"
Oct 30 13:51:00.139694 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:00.139625    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72-istio-token\") pod \"e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9\" (UID: \"3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72\") " pod="openshift-ingress/e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9"
Oct 30 13:51:00.139694 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:00.139646    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72-istio-data\") pod \"e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9\" (UID: \"3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72\") " pod="openshift-ingress/e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9"
Oct 30 13:51:00.139694 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:00.139666    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72-workload-certs\") pod \"e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9\" (UID: \"3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72\") " pod="openshift-ingress/e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9"
Oct 30 13:51:00.139694 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:00.139685    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72-workload-socket\") pod \"e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9\" (UID: \"3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72\") " pod="openshift-ingress/e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9"
Oct 30 13:51:00.139878 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:00.139784    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72-istio-envoy\") pod \"e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9\" (UID: \"3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72\") " pod="openshift-ingress/e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9"
Oct 30 13:51:00.139878 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:00.139854    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72-credential-socket\") pod \"e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9\" (UID: \"3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72\") " pod="openshift-ingress/e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9"
Oct 30 13:51:00.139878 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:00.139869    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72-istiod-ca-cert\") pod \"e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9\" (UID: \"3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72\") " pod="openshift-ingress/e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9"
Oct 30 13:51:00.241863 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:00.241817    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72-istio-podinfo\") pod \"e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9\" (UID: \"3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72\") " pod="openshift-ingress/e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9"
Oct 30 13:51:00.242069 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:00.241881    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htmcl\" (UniqueName: \"kubernetes.io/projected/3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72-kube-api-access-htmcl\") pod \"e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9\" (UID: \"3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72\") " pod="openshift-ingress/e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9"
Oct 30 13:51:00.242069 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:00.241971    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72-istio-token\") pod \"e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9\" (UID: \"3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72\") " pod="openshift-ingress/e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9"
Oct 30 13:51:00.242069 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:00.241999    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72-istio-data\") pod \"e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9\" (UID: \"3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72\") " pod="openshift-ingress/e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9"
Oct 30 13:51:00.242069 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:00.242028    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72-workload-certs\") pod \"e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9\" (UID: \"3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72\") " pod="openshift-ingress/e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9"
Oct 30 13:51:00.242069 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:00.242058    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72-workload-socket\") pod \"e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9\" (UID: \"3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72\") " pod="openshift-ingress/e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9"
Oct 30 13:51:00.242388 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:00.242099    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72-istio-envoy\") pod \"e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9\" (UID: \"3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72\") " pod="openshift-ingress/e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9"
Oct 30 13:51:00.242388 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:00.242187    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72-credential-socket\") pod \"e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9\" (UID: \"3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72\") " pod="openshift-ingress/e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9"
Oct 30 13:51:00.242388 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:00.242212    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72-istiod-ca-cert\") pod \"e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9\" (UID: \"3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72\") " pod="openshift-ingress/e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9"
Oct 30 13:51:00.243106 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:00.243068    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72-istiod-ca-cert\") pod \"e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9\" (UID: \"3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72\") " pod="openshift-ingress/e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9"
Oct 30 13:51:00.243384 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:00.243363    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72-workload-certs\") pod \"e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9\" (UID: \"3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72\") " pod="openshift-ingress/e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9"
Oct 30 13:51:00.243898 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:00.243876    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72-workload-socket\") pod \"e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9\" (UID: \"3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72\") " pod="openshift-ingress/e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9"
Oct 30 13:51:00.245737 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:00.245710    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72-istio-data\") pod \"e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9\" (UID: \"3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72\") " pod="openshift-ingress/e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9"
Oct 30 13:51:00.247292 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:00.247252    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72-istio-podinfo\") pod \"e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9\" (UID: \"3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72\") " pod="openshift-ingress/e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9"
Oct 30 13:51:00.247582 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:00.247561    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72-credential-socket\") pod \"e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9\" (UID: \"3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72\") " pod="openshift-ingress/e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9"
Oct 30 13:51:00.249799 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:00.249755    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72-istio-envoy\") pod \"e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9\" (UID: \"3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72\") " pod="openshift-ingress/e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9"
Oct 30 13:51:00.269930 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:00.269892    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htmcl\" (UniqueName: \"kubernetes.io/projected/3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72-kube-api-access-htmcl\") pod \"e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9\" (UID: \"3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72\") " pod="openshift-ingress/e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9"
Oct 30 13:51:00.271736 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:00.271704    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72-istio-token\") pod \"e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9\" (UID: \"3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72\") " pod="openshift-ingress/e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9"
Oct 30 13:51:00.391690 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:00.391662    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9"
Oct 30 13:51:00.709826 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:00.709798    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9"]
Oct 30 13:51:00.712502 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:51:00.712473    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f2c06e3_2c7f_4eaa_8153_6fd66eb78f72.slice/crio-30674910959f3ed2e5d1f7c5d2b3faf71913a7549df48791dc4133534df678e0 WatchSource:0}: Error finding container 30674910959f3ed2e5d1f7c5d2b3faf71913a7549df48791dc4133534df678e0: Status 404 returned error can't find the container with id 30674910959f3ed2e5d1f7c5d2b3faf71913a7549df48791dc4133534df678e0
Oct 30 13:51:00.714666 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:00.714634    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:51:00.714761 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:00.714707    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:51:00.714761 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:00.714737    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:51:00.994526 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:00.994439    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9" event={"ID":"3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72","Type":"ContainerStarted","Data":"79394241230f61cca0db87f08a70b5afc5f7849944214026ac93770ac02eb337"}
Oct 30 13:51:00.994526 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:00.994499    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9" event={"ID":"3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72","Type":"ContainerStarted","Data":"30674910959f3ed2e5d1f7c5d2b3faf71913a7549df48791dc4133534df678e0"}
Oct 30 13:51:01.044938 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:01.044887    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9" podStartSLOduration=1.044873646 podStartE2EDuration="1.044873646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:51:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:51:01.042008647 +0000 UTC m=+3560.208373295" watchObservedRunningTime="2026-10-30 13:51:01.044873646 +0000 UTC m=+3560.211238291"
Oct 30 13:51:01.392092 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:01.392060    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9"
Oct 30 13:51:01.397671 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:01.397645    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9"
Oct 30 13:51:02.000367 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:02.000338    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9"
Oct 30 13:51:02.001405 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:02.001387    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9"
Oct 30 13:51:05.121255 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:05.121213    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a5b3f0-84b6ff667d-skmlp"]
Oct 30 13:51:05.125034 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:05.125009    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a5b3f0-84b6ff667d-skmlp"
Oct 30 13:51:05.160061 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:05.160030    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:51:05.160272 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:05.160213    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-a5b3f0-serving-cert\""
Oct 30 13:51:05.160337 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:05.160296    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 13:51:05.160419 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:05.160401    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:51:05.163578 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:05.163551    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a5b3f0-84b6ff667d-skmlp"]
Oct 30 13:51:05.163905 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:05.163886    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-zw9vx\""
Oct 30 13:51:05.192378 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:05.192341    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89937d9b-eafb-4d38-a02b-484bfa26876f-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-a5b3f0-84b6ff667d-skmlp\" (UID: \"89937d9b-eafb-4d38-a02b-484bfa26876f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a5b3f0-84b6ff667d-skmlp"
Oct 30 13:51:05.192589 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:05.192523    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddts2\" (UniqueName: \"kubernetes.io/projected/89937d9b-eafb-4d38-a02b-484bfa26876f-kube-api-access-ddts2\") pod \"maas-api-e2e-worker-single-main-a5b3f0-84b6ff667d-skmlp\" (UID: \"89937d9b-eafb-4d38-a02b-484bfa26876f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a5b3f0-84b6ff667d-skmlp"
Oct 30 13:51:05.192589 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:05.192572    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89937d9b-eafb-4d38-a02b-484bfa26876f-metrics-certs\") pod \"maas-api-e2e-worker-single-main-a5b3f0-84b6ff667d-skmlp\" (UID: \"89937d9b-eafb-4d38-a02b-484bfa26876f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a5b3f0-84b6ff667d-skmlp"
Oct 30 13:51:05.192709 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:05.192663    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89937d9b-eafb-4d38-a02b-484bfa26876f-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-a5b3f0-84b6ff667d-skmlp\" (UID: \"89937d9b-eafb-4d38-a02b-484bfa26876f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a5b3f0-84b6ff667d-skmlp"
Oct 30 13:51:05.293877 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:05.293844    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddts2\" (UniqueName: \"kubernetes.io/projected/89937d9b-eafb-4d38-a02b-484bfa26876f-kube-api-access-ddts2\") pod \"maas-api-e2e-worker-single-main-a5b3f0-84b6ff667d-skmlp\" (UID: \"89937d9b-eafb-4d38-a02b-484bfa26876f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a5b3f0-84b6ff667d-skmlp"
Oct 30 13:51:05.294054 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:05.293891    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89937d9b-eafb-4d38-a02b-484bfa26876f-metrics-certs\") pod \"maas-api-e2e-worker-single-main-a5b3f0-84b6ff667d-skmlp\" (UID: \"89937d9b-eafb-4d38-a02b-484bfa26876f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a5b3f0-84b6ff667d-skmlp"
Oct 30 13:51:05.294054 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:05.293918    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89937d9b-eafb-4d38-a02b-484bfa26876f-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-a5b3f0-84b6ff667d-skmlp\" (UID: \"89937d9b-eafb-4d38-a02b-484bfa26876f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a5b3f0-84b6ff667d-skmlp"
Oct 30 13:51:05.294054 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:05.294022    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89937d9b-eafb-4d38-a02b-484bfa26876f-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-a5b3f0-84b6ff667d-skmlp\" (UID: \"89937d9b-eafb-4d38-a02b-484bfa26876f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a5b3f0-84b6ff667d-skmlp"
Oct 30 13:51:05.294760 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:05.294725    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89937d9b-eafb-4d38-a02b-484bfa26876f-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-a5b3f0-84b6ff667d-skmlp\" (UID: \"89937d9b-eafb-4d38-a02b-484bfa26876f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a5b3f0-84b6ff667d-skmlp"
Oct 30 13:51:05.296433 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:05.296413    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89937d9b-eafb-4d38-a02b-484bfa26876f-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-a5b3f0-84b6ff667d-skmlp\" (UID: \"89937d9b-eafb-4d38-a02b-484bfa26876f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a5b3f0-84b6ff667d-skmlp"
Oct 30 13:51:05.296523 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:05.296451    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89937d9b-eafb-4d38-a02b-484bfa26876f-metrics-certs\") pod \"maas-api-e2e-worker-single-main-a5b3f0-84b6ff667d-skmlp\" (UID: \"89937d9b-eafb-4d38-a02b-484bfa26876f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a5b3f0-84b6ff667d-skmlp"
Oct 30 13:51:05.325167 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:05.325136    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddts2\" (UniqueName: \"kubernetes.io/projected/89937d9b-eafb-4d38-a02b-484bfa26876f-kube-api-access-ddts2\") pod \"maas-api-e2e-worker-single-main-a5b3f0-84b6ff667d-skmlp\" (UID: \"89937d9b-eafb-4d38-a02b-484bfa26876f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a5b3f0-84b6ff667d-skmlp"
Oct 30 13:51:05.439193 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:05.439090    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a5b3f0-84b6ff667d-skmlp"
Oct 30 13:51:05.453829 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:05.453800    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a5b3f0-c9d8565db4826q"]
Oct 30 13:51:05.457661 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:05.457641    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a5b3f0-c9d8565db4826q"
Oct 30 13:51:05.480684 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:05.479453    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a5b3f0-c9d8565db4826q"]
Oct 30 13:51:05.482076 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:05.482028    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-20fc86de-dockercfg-rh7f8\""
Oct 30 13:51:05.482228 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:05.482211    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-a5b3f0\""
Oct 30 13:51:05.498418 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:05.498047    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gdst\" (UniqueName: \"kubernetes.io/projected/84430325-7922-4f93-bcd7-616ebddc1a2d-kube-api-access-8gdst\") pod \"payload-processing-e2e-worker-single-main-a5b3f0-c9d8565db4826q\" (UID: \"84430325-7922-4f93-bcd7-616ebddc1a2d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a5b3f0-c9d8565db4826q"
Oct 30 13:51:05.498418 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:05.498167    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84430325-7922-4f93-bcd7-616ebddc1a2d-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-a5b3f0-c9d8565db4826q\" (UID: \"84430325-7922-4f93-bcd7-616ebddc1a2d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a5b3f0-c9d8565db4826q"
Oct 30 13:51:05.528500 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:05.528467    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a5b3f0-76587mqgpt"]
Oct 30 13:51:05.532607 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:05.532575    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a5b3f0-76587mqgpt"
Oct 30 13:51:05.567336 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:05.567305    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a5b3f0-76587mqgpt"]
Oct 30 13:51:05.613648 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:05.609031    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cb7160f-e10f-4f3c-8327-65d508ccc923-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-a5b3f0-76587mqgpt\" (UID: \"9cb7160f-e10f-4f3c-8327-65d508ccc923\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a5b3f0-76587mqgpt"
Oct 30 13:51:05.613648 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:05.609195    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gdst\" (UniqueName: \"kubernetes.io/projected/84430325-7922-4f93-bcd7-616ebddc1a2d-kube-api-access-8gdst\") pod \"payload-processing-e2e-worker-single-main-a5b3f0-c9d8565db4826q\" (UID: \"84430325-7922-4f93-bcd7-616ebddc1a2d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a5b3f0-c9d8565db4826q"
Oct 30 13:51:05.616763 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:05.616671    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84430325-7922-4f93-bcd7-616ebddc1a2d-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-a5b3f0-c9d8565db4826q\" (UID: \"84430325-7922-4f93-bcd7-616ebddc1a2d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a5b3f0-c9d8565db4826q"
Oct 30 13:51:05.616991 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:05.616863    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvbz5\" (UniqueName: \"kubernetes.io/projected/9cb7160f-e10f-4f3c-8327-65d508ccc923-kube-api-access-gvbz5\") pod \"payload-pre-processing-e2e-worker-single-main-a5b3f0-76587mqgpt\" (UID: \"9cb7160f-e10f-4f3c-8327-65d508ccc923\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a5b3f0-76587mqgpt"
Oct 30 13:51:05.617697 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:05.617557    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84430325-7922-4f93-bcd7-616ebddc1a2d-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-a5b3f0-c9d8565db4826q\" (UID: \"84430325-7922-4f93-bcd7-616ebddc1a2d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a5b3f0-c9d8565db4826q"
Oct 30 13:51:05.632931 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:05.632898    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gdst\" (UniqueName: \"kubernetes.io/projected/84430325-7922-4f93-bcd7-616ebddc1a2d-kube-api-access-8gdst\") pod \"payload-processing-e2e-worker-single-main-a5b3f0-c9d8565db4826q\" (UID: \"84430325-7922-4f93-bcd7-616ebddc1a2d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a5b3f0-c9d8565db4826q"
Oct 30 13:51:05.718539 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:05.718458    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvbz5\" (UniqueName: \"kubernetes.io/projected/9cb7160f-e10f-4f3c-8327-65d508ccc923-kube-api-access-gvbz5\") pod \"payload-pre-processing-e2e-worker-single-main-a5b3f0-76587mqgpt\" (UID: \"9cb7160f-e10f-4f3c-8327-65d508ccc923\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a5b3f0-76587mqgpt"
Oct 30 13:51:05.718539 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:05.718520    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cb7160f-e10f-4f3c-8327-65d508ccc923-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-a5b3f0-76587mqgpt\" (UID: \"9cb7160f-e10f-4f3c-8327-65d508ccc923\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a5b3f0-76587mqgpt"
Oct 30 13:51:05.719332 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:05.719303    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cb7160f-e10f-4f3c-8327-65d508ccc923-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-a5b3f0-76587mqgpt\" (UID: \"9cb7160f-e10f-4f3c-8327-65d508ccc923\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a5b3f0-76587mqgpt"
Oct 30 13:51:05.742340 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:05.742315    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvbz5\" (UniqueName: \"kubernetes.io/projected/9cb7160f-e10f-4f3c-8327-65d508ccc923-kube-api-access-gvbz5\") pod \"payload-pre-processing-e2e-worker-single-main-a5b3f0-76587mqgpt\" (UID: \"9cb7160f-e10f-4f3c-8327-65d508ccc923\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a5b3f0-76587mqgpt"
Oct 30 13:51:05.748407 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:05.748372    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a5b3f0-84b6ff667d-skmlp"]
Oct 30 13:51:05.750659 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:51:05.750621    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89937d9b_eafb_4d38_a02b_484bfa26876f.slice/crio-332bad891fe74329d6e7e283b940199baa549bdb2886254478be479ecb763c5b WatchSource:0}: Error finding container 332bad891fe74329d6e7e283b940199baa549bdb2886254478be479ecb763c5b: Status 404 returned error can't find the container with id 332bad891fe74329d6e7e283b940199baa549bdb2886254478be479ecb763c5b
Oct 30 13:51:05.752150 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:05.752133    4364 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 13:51:05.777880 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:05.777855    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a5b3f0-c9d8565db4826q"
Oct 30 13:51:05.846789 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:05.846760    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a5b3f0-76587mqgpt"
Oct 30 13:51:06.021114 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:06.021025    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a5b3f0-84b6ff667d-skmlp" event={"ID":"89937d9b-eafb-4d38-a02b-484bfa26876f","Type":"ContainerStarted","Data":"332bad891fe74329d6e7e283b940199baa549bdb2886254478be479ecb763c5b"}
Oct 30 13:51:06.140875 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:06.140835    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a5b3f0-c9d8565db4826q"]
Oct 30 13:51:06.154251 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:51:06.154216    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84430325_7922_4f93_bcd7_616ebddc1a2d.slice/crio-86fbae921031cb4ef6afe84b6dd39c7e095ea112eb1650ffc01a567e7c1558b3 WatchSource:0}: Error finding container 86fbae921031cb4ef6afe84b6dd39c7e095ea112eb1650ffc01a567e7c1558b3: Status 404 returned error can't find the container with id 86fbae921031cb4ef6afe84b6dd39c7e095ea112eb1650ffc01a567e7c1558b3
Oct 30 13:51:06.202581 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:06.202555    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a5b3f0-76587mqgpt"]
Oct 30 13:51:06.206207 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:51:06.206184    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cb7160f_e10f_4f3c_8327_65d508ccc923.slice/crio-c57cf68a8704a5b9b5c9c1671f15065d019fee611cc653c2504e70aaafce6312 WatchSource:0}: Error finding container c57cf68a8704a5b9b5c9c1671f15065d019fee611cc653c2504e70aaafce6312: Status 404 returned error can't find the container with id c57cf68a8704a5b9b5c9c1671f15065d019fee611cc653c2504e70aaafce6312
Oct 30 13:51:07.027849 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:07.027804    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a5b3f0-84b6ff667d-skmlp" event={"ID":"89937d9b-eafb-4d38-a02b-484bfa26876f","Type":"ContainerStarted","Data":"3e2ae70acfd70ba8a4c5e520a9e4216724f8477b398ea4f7a640f2ef2d514862"}
Oct 30 13:51:07.028067 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:07.027925    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a5b3f0-84b6ff667d-skmlp"
Oct 30 13:51:07.029542 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:07.029503    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a5b3f0-c9d8565db4826q" event={"ID":"84430325-7922-4f93-bcd7-616ebddc1a2d","Type":"ContainerStarted","Data":"e1c7c4203e4ff74527aeb48e200f9f87b31f5b049c3deb84106b3b3921d16c3f"}
Oct 30 13:51:07.029683 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:07.029561    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a5b3f0-c9d8565db4826q" event={"ID":"84430325-7922-4f93-bcd7-616ebddc1a2d","Type":"ContainerStarted","Data":"86fbae921031cb4ef6afe84b6dd39c7e095ea112eb1650ffc01a567e7c1558b3"}
Oct 30 13:51:07.029683 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:07.029620    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a5b3f0-c9d8565db4826q"
Oct 30 13:51:07.031458 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:07.031423    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a5b3f0-76587mqgpt" event={"ID":"9cb7160f-e10f-4f3c-8327-65d508ccc923","Type":"ContainerStarted","Data":"fbc002f3193917d5cf56c5f9dd608f231ced24b3cf23bcca7a8b8194eaddd9da"}
Oct 30 13:51:07.031615 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:07.031483    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a5b3f0-76587mqgpt" event={"ID":"9cb7160f-e10f-4f3c-8327-65d508ccc923","Type":"ContainerStarted","Data":"c57cf68a8704a5b9b5c9c1671f15065d019fee611cc653c2504e70aaafce6312"}
Oct 30 13:51:07.031716 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:07.031695    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a5b3f0-76587mqgpt"
Oct 30 13:51:07.072086 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:07.072036    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a5b3f0-84b6ff667d-skmlp" podStartSLOduration=1.629841477 podStartE2EDuration="2.072023812s" totalImagesPullingTime="442.182335ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:51:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:51:07.070238047 +0000 UTC m=+3566.236602697" watchObservedRunningTime="2026-10-30 13:51:07.072023812 +0000 UTC m=+3566.238388487"
Oct 30 13:51:07.113372 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:07.113320    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a5b3f0-76587mqgpt" podStartSLOduration=2.11330713 podStartE2EDuration="2.11330713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:51:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:51:07.111160818 +0000 UTC m=+3566.277525465" watchObservedRunningTime="2026-10-30 13:51:07.11330713 +0000 UTC m=+3566.279671775"
Oct 30 13:51:07.154127 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:07.154068    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a5b3f0-c9d8565db4826q" podStartSLOduration=2.154053803 podStartE2EDuration="2.154053803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:51:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:51:07.151530697 +0000 UTC m=+3566.317895345" watchObservedRunningTime="2026-10-30 13:51:07.154053803 +0000 UTC m=+3566.320418450"
Oct 30 13:51:08.896371 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:08.896248    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:51:11.527944 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:51:11.527908    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:51:13.042774 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:13.042745    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a5b3f0-84b6ff667d-skmlp"
Oct 30 13:51:18.040445 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:18.040410    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a5b3f0-c9d8565db4826q"
Oct 30 13:51:18.040968 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:18.040492    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a5b3f0-76587mqgpt"
Oct 30 13:51:18.041736 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:18.041710    4364 patch_prober.go:29] interesting pod/e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.17:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:51:18.041833 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:18.041768    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9" podUID="3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.17:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:51:19.104051 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:19.102765    4364 generic.go:356] "Generic (PLEG): container finished" podID="3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72" containerID="79394241230f61cca0db87f08a70b5afc5f7849944214026ac93770ac02eb337" exitCode=0
Oct 30 13:51:19.104051 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:19.102841    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9" event={"ID":"3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72","Type":"ContainerDied","Data":"79394241230f61cca0db87f08a70b5afc5f7849944214026ac93770ac02eb337"}
Oct 30 13:51:19.104051 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:19.103321    4364 scope.go:122] "RemoveContainer" containerID="79394241230f61cca0db87f08a70b5afc5f7849944214026ac93770ac02eb337"
Oct 30 13:51:19.106166 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:19.106125    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:51:19.106322 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:19.106221    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:51:19.106322 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:19.106278    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:51:20.111268 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:20.111233    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9" event={"ID":"3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72","Type":"ContainerStarted","Data":"f90488186bf96a52b8e5bdabb12e974a6e48c76663ad272290c4152f487fb8dc"}
Oct 30 13:51:20.397631 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:20.397034    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9"
Oct 30 13:51:20.397631 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:20.397228    4364 patch_prober.go:29] interesting pod/e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused" start-of-body=
Oct 30 13:51:20.397631 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:20.397287    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9" podUID="3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused"
Oct 30 13:51:21.209625 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:21.207295    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-a5b3f0/simulator-single-main-a5b3f0-kserve-5d9bf8ff77-8cpqh"]
Oct 30 13:51:21.211044 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:21.211017    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-a5b3f0/simulator-single-main-a5b3f0-kserve-5d9bf8ff77-8cpqh"
Oct 30 13:51:21.218389 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:21.218332    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dc02570-e085-4987-94eb-ab9efda42c0c-model-cache\") pod \"simulator-single-main-a5b3f0-kserve-5d9bf8ff77-8cpqh\" (UID: \"0dc02570-e085-4987-94eb-ab9efda42c0c\") " pod="e2e-models-e2e-worker-single-main-a5b3f0/simulator-single-main-a5b3f0-kserve-5d9bf8ff77-8cpqh"
Oct 30 13:51:21.218644 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:21.218505    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-a5b3f0/simulator-single-main-a5b3f0-kserve-5d9bf8ff77-8cpqh"]
Oct 30 13:51:21.218855 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:21.218831    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvb47\" (UniqueName: \"kubernetes.io/projected/0dc02570-e085-4987-94eb-ab9efda42c0c-kube-api-access-dvb47\") pod \"simulator-single-main-a5b3f0-kserve-5d9bf8ff77-8cpqh\" (UID: \"0dc02570-e085-4987-94eb-ab9efda42c0c\") " pod="e2e-models-e2e-worker-single-main-a5b3f0/simulator-single-main-a5b3f0-kserve-5d9bf8ff77-8cpqh"
Oct 30 13:51:21.218959 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:21.218895    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dc02570-e085-4987-94eb-ab9efda42c0c-tls-certs\") pod \"simulator-single-main-a5b3f0-kserve-5d9bf8ff77-8cpqh\" (UID: \"0dc02570-e085-4987-94eb-ab9efda42c0c\") " pod="e2e-models-e2e-worker-single-main-a5b3f0/simulator-single-main-a5b3f0-kserve-5d9bf8ff77-8cpqh"
Oct 30 13:51:21.219045 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:21.219030    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dc02570-e085-4987-94eb-ab9efda42c0c-tmp-dir\") pod \"simulator-single-main-a5b3f0-kserve-5d9bf8ff77-8cpqh\" (UID: \"0dc02570-e085-4987-94eb-ab9efda42c0c\") " pod="e2e-models-e2e-worker-single-main-a5b3f0/simulator-single-main-a5b3f0-kserve-5d9bf8ff77-8cpqh"
Oct 30 13:51:21.219251 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:21.219110    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0dc02570-e085-4987-94eb-ab9efda42c0c-home\") pod \"simulator-single-main-a5b3f0-kserve-5d9bf8ff77-8cpqh\" (UID: \"0dc02570-e085-4987-94eb-ab9efda42c0c\") " pod="e2e-models-e2e-worker-single-main-a5b3f0/simulator-single-main-a5b3f0-kserve-5d9bf8ff77-8cpqh"
Oct 30 13:51:21.219251 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:21.219142    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dc02570-e085-4987-94eb-ab9efda42c0c-dshm\") pod \"simulator-single-main-a5b3f0-kserve-5d9bf8ff77-8cpqh\" (UID: \"0dc02570-e085-4987-94eb-ab9efda42c0c\") " pod="e2e-models-e2e-worker-single-main-a5b3f0/simulator-single-main-a5b3f0-kserve-5d9bf8ff77-8cpqh"
Oct 30 13:51:21.226695 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:21.226673    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-a5b3f0\"/\"kube-root-ca.crt\""
Oct 30 13:51:21.226793 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:21.226676    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-a5b3f0\"/\"openshift-service-ca.crt\""
Oct 30 13:51:21.238477 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:21.238452    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-a5b3f0\"/\"default-dockercfg-nz7nc\""
Oct 30 13:51:21.238477 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:21.238465    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-a5b3f0\"/\"simulator-single-main-a5b3f0-kserve-self-signed-certs\""
Oct 30 13:51:21.319664 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:21.319587    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvb47\" (UniqueName: \"kubernetes.io/projected/0dc02570-e085-4987-94eb-ab9efda42c0c-kube-api-access-dvb47\") pod \"simulator-single-main-a5b3f0-kserve-5d9bf8ff77-8cpqh\" (UID: \"0dc02570-e085-4987-94eb-ab9efda42c0c\") " pod="e2e-models-e2e-worker-single-main-a5b3f0/simulator-single-main-a5b3f0-kserve-5d9bf8ff77-8cpqh"
Oct 30 13:51:21.319857 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:21.319703    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dc02570-e085-4987-94eb-ab9efda42c0c-tls-certs\") pod \"simulator-single-main-a5b3f0-kserve-5d9bf8ff77-8cpqh\" (UID: \"0dc02570-e085-4987-94eb-ab9efda42c0c\") " pod="e2e-models-e2e-worker-single-main-a5b3f0/simulator-single-main-a5b3f0-kserve-5d9bf8ff77-8cpqh"
Oct 30 13:51:21.319857 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:21.319828    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dc02570-e085-4987-94eb-ab9efda42c0c-tmp-dir\") pod \"simulator-single-main-a5b3f0-kserve-5d9bf8ff77-8cpqh\" (UID: \"0dc02570-e085-4987-94eb-ab9efda42c0c\") " pod="e2e-models-e2e-worker-single-main-a5b3f0/simulator-single-main-a5b3f0-kserve-5d9bf8ff77-8cpqh"
Oct 30 13:51:21.319988 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:21.319877    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0dc02570-e085-4987-94eb-ab9efda42c0c-home\") pod \"simulator-single-main-a5b3f0-kserve-5d9bf8ff77-8cpqh\" (UID: \"0dc02570-e085-4987-94eb-ab9efda42c0c\") " pod="e2e-models-e2e-worker-single-main-a5b3f0/simulator-single-main-a5b3f0-kserve-5d9bf8ff77-8cpqh"
Oct 30 13:51:21.319988 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:21.319905    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dc02570-e085-4987-94eb-ab9efda42c0c-dshm\") pod \"simulator-single-main-a5b3f0-kserve-5d9bf8ff77-8cpqh\" (UID: \"0dc02570-e085-4987-94eb-ab9efda42c0c\") " pod="e2e-models-e2e-worker-single-main-a5b3f0/simulator-single-main-a5b3f0-kserve-5d9bf8ff77-8cpqh"
Oct 30 13:51:21.319988 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:21.319947    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dc02570-e085-4987-94eb-ab9efda42c0c-model-cache\") pod \"simulator-single-main-a5b3f0-kserve-5d9bf8ff77-8cpqh\" (UID: \"0dc02570-e085-4987-94eb-ab9efda42c0c\") " pod="e2e-models-e2e-worker-single-main-a5b3f0/simulator-single-main-a5b3f0-kserve-5d9bf8ff77-8cpqh"
Oct 30 13:51:21.320444 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:21.320419    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0dc02570-e085-4987-94eb-ab9efda42c0c-home\") pod \"simulator-single-main-a5b3f0-kserve-5d9bf8ff77-8cpqh\" (UID: \"0dc02570-e085-4987-94eb-ab9efda42c0c\") " pod="e2e-models-e2e-worker-single-main-a5b3f0/simulator-single-main-a5b3f0-kserve-5d9bf8ff77-8cpqh"
Oct 30 13:51:21.320560 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:21.320495    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dc02570-e085-4987-94eb-ab9efda42c0c-tmp-dir\") pod \"simulator-single-main-a5b3f0-kserve-5d9bf8ff77-8cpqh\" (UID: \"0dc02570-e085-4987-94eb-ab9efda42c0c\") " pod="e2e-models-e2e-worker-single-main-a5b3f0/simulator-single-main-a5b3f0-kserve-5d9bf8ff77-8cpqh"
Oct 30 13:51:21.320560 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:21.320531    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dc02570-e085-4987-94eb-ab9efda42c0c-model-cache\") pod \"simulator-single-main-a5b3f0-kserve-5d9bf8ff77-8cpqh\" (UID: \"0dc02570-e085-4987-94eb-ab9efda42c0c\") " pod="e2e-models-e2e-worker-single-main-a5b3f0/simulator-single-main-a5b3f0-kserve-5d9bf8ff77-8cpqh"
Oct 30 13:51:21.322229 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:21.322206    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dc02570-e085-4987-94eb-ab9efda42c0c-dshm\") pod \"simulator-single-main-a5b3f0-kserve-5d9bf8ff77-8cpqh\" (UID: \"0dc02570-e085-4987-94eb-ab9efda42c0c\") " pod="e2e-models-e2e-worker-single-main-a5b3f0/simulator-single-main-a5b3f0-kserve-5d9bf8ff77-8cpqh"
Oct 30 13:51:21.322362 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:21.322310    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dc02570-e085-4987-94eb-ab9efda42c0c-tls-certs\") pod \"simulator-single-main-a5b3f0-kserve-5d9bf8ff77-8cpqh\" (UID: \"0dc02570-e085-4987-94eb-ab9efda42c0c\") " pod="e2e-models-e2e-worker-single-main-a5b3f0/simulator-single-main-a5b3f0-kserve-5d9bf8ff77-8cpqh"
Oct 30 13:51:21.349162 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:21.349136    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvb47\" (UniqueName: \"kubernetes.io/projected/0dc02570-e085-4987-94eb-ab9efda42c0c-kube-api-access-dvb47\") pod \"simulator-single-main-a5b3f0-kserve-5d9bf8ff77-8cpqh\" (UID: \"0dc02570-e085-4987-94eb-ab9efda42c0c\") " pod="e2e-models-e2e-worker-single-main-a5b3f0/simulator-single-main-a5b3f0-kserve-5d9bf8ff77-8cpqh"
Oct 30 13:51:21.401621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:21.400986    4364 patch_prober.go:29] interesting pod/e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused" start-of-body=
Oct 30 13:51:21.401621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:21.401049    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9" podUID="3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused"
Oct 30 13:51:21.526231 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:21.526141    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-a5b3f0/simulator-single-main-a5b3f0-kserve-5d9bf8ff77-8cpqh"
Oct 30 13:51:21.835124 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:21.835025    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-a5b3f0/simulator-single-main-a5b3f0-kserve-5d9bf8ff77-8cpqh"]
Oct 30 13:51:21.837494 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:51:21.837469    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0dc02570_e085_4987_94eb_ab9efda42c0c.slice/crio-9539fdaf3d62661d9d5b0b3eb2db1d5bbf680ec13e38b48979504f1392981d49 WatchSource:0}: Error finding container 9539fdaf3d62661d9d5b0b3eb2db1d5bbf680ec13e38b48979504f1392981d49: Status 404 returned error can't find the container with id 9539fdaf3d62661d9d5b0b3eb2db1d5bbf680ec13e38b48979504f1392981d49
Oct 30 13:51:22.028899 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:22.028866    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-q8tpb"]
Oct 30 13:51:22.031563 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:22.031536    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-q8tpb"]
Oct 30 13:51:22.124568 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:22.124535    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-a5b3f0/simulator-single-main-a5b3f0-kserve-5d9bf8ff77-8cpqh" event={"ID":"0dc02570-e085-4987-94eb-ab9efda42c0c","Type":"ContainerStarted","Data":"71d589a01109d4e83c614416bf6a8d44fffc76b3f340d08c1c684223cd6609c4"}
Oct 30 13:51:22.124783 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:22.124575    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-a5b3f0/simulator-single-main-a5b3f0-kserve-5d9bf8ff77-8cpqh" event={"ID":"0dc02570-e085-4987-94eb-ab9efda42c0c","Type":"ContainerStarted","Data":"9539fdaf3d62661d9d5b0b3eb2db1d5bbf680ec13e38b48979504f1392981d49"}
Oct 30 13:51:22.167127 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:22.167076    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-a5b3f0/simulator-single-main-a5b3f0-kserve-5d9bf8ff77-8cpqh" podStartSLOduration=1.167063201 podStartE2EDuration="1.167063201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:51:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:51:22.165326294 +0000 UTC m=+3581.331690943" watchObservedRunningTime="2026-10-30 13:51:22.167063201 +0000 UTC m=+3581.333427876"
Oct 30 13:51:22.401699 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:22.401024    4364 patch_prober.go:29] interesting pod/e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused" start-of-body=
Oct 30 13:51:22.401699 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:22.401085    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9" podUID="3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused"
Oct 30 13:51:22.526795 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:22.526753    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-a5b3f0/simulator-single-main-a5b3f0-kserve-5d9bf8ff77-8cpqh"
Oct 30 13:51:22.710819 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:22.710730    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:51:22.969859 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:22.969776    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ff3f00b-a2fc-461f-a23c-32cb08dc3225" path="/var/lib/kubelet/pods/2ff3f00b-a2fc-461f-a23c-32cb08dc3225/volumes"
Oct 30 13:51:23.407027 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:23.406985    4364 patch_prober.go:29] interesting pod/e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused" start-of-body=
Oct 30 13:51:23.407576 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:23.407048    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9" podUID="3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused"
Oct 30 13:51:24.393623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:24.392436    4364 patch_prober.go:29] interesting pod/e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused" start-of-body=
Oct 30 13:51:24.393623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:24.392493    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9" podUID="3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused"
Oct 30 13:51:25.717504 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:25.717466    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9"
Oct 30 13:51:25.718058 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:25.718038    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9"
Oct 30 13:51:25.718992 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:25.718971    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9"
Oct 30 13:51:31.527159 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:31.527125    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-a5b3f0/simulator-single-main-a5b3f0-kserve-5d9bf8ff77-8cpqh"
Oct 30 13:51:31.540128 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:31.540105    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-a5b3f0/simulator-single-main-a5b3f0-kserve-5d9bf8ff77-8cpqh"
Oct 30 13:51:32.192639 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:32.192585    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-a5b3f0/simulator-single-main-a5b3f0-kserve-5d9bf8ff77-8cpqh"
Oct 30 13:51:39.354540 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:39.354504    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:51:41.504077 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:51:41.504035    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:51:44.707786 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:44.707755    4364 scope.go:122] "RemoveContainer" containerID="bb11a36c05ed116a3f64048a5436fab4282d194913ef93584d3936c5cca0ce5c"
Oct 30 13:51:44.717140 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:44.717109    4364 scope.go:122] "RemoveContainer" containerID="25462c8cbbac98743555fed75e826c6285c38522c87bf1cdfcc666362151effb"
Oct 30 13:51:44.728582 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:44.728559    4364 scope.go:122] "RemoveContainer" containerID="240741113a5723174ed33765ab2a09e654b914880a7a568895a5227855d95d10"
Oct 30 13:51:44.737280 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:44.737260    4364 scope.go:122] "RemoveContainer" containerID="5dcfcba21d591edafd1f1aa0b6dd31105dd8af3311840cba91ec7ad9c6744150"
Oct 30 13:51:57.084307 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:51:57.084268    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:52:01.672251 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:01.672212    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-a5b3f0/simulator-single-main-a5b3f0-kserve-5d9bf8ff77-8cpqh"]
Oct 30 13:52:07.256975 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:07.256937    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a5b3f0-c9d8565db4826q"]
Oct 30 13:52:07.287073 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:07.287041    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a5b3f0-76587mqgpt"]
Oct 30 13:52:08.012885 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:08.012849    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a5b3f0-84b6ff667d-skmlp"]
Oct 30 13:52:08.013137 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:08.013091    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a5b3f0-84b6ff667d-skmlp" podUID="89937d9b-eafb-4d38-a02b-484bfa26876f" containerName="maas-api" containerID="cri-o://3e2ae70acfd70ba8a4c5e520a9e4216724f8477b398ea4f7a640f2ef2d514862" gracePeriod=30
Oct 30 13:52:08.037325 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:08.037283    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a5b3f0-84b6ff667d-skmlp" podUID="89937d9b-eafb-4d38-a02b-484bfa26876f" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.1.18:8443/health\": dial tcp 10.133.1.18:8443: connect: connection refused"
Oct 30 13:52:08.407117 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:08.406039    4364 generic.go:356] "Generic (PLEG): container finished" podID="89937d9b-eafb-4d38-a02b-484bfa26876f" containerID="3e2ae70acfd70ba8a4c5e520a9e4216724f8477b398ea4f7a640f2ef2d514862" exitCode=0
Oct 30 13:52:08.407117 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:08.406089    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a5b3f0-84b6ff667d-skmlp" event={"ID":"89937d9b-eafb-4d38-a02b-484bfa26876f","Type":"ContainerDied","Data":"3e2ae70acfd70ba8a4c5e520a9e4216724f8477b398ea4f7a640f2ef2d514862"}
Oct 30 13:52:08.483559 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:08.483531    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a5b3f0-84b6ff667d-skmlp"
Oct 30 13:52:08.605740 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:08.605657    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89937d9b-eafb-4d38-a02b-484bfa26876f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/89937d9b-eafb-4d38-a02b-484bfa26876f-metrics-certs\") pod \"89937d9b-eafb-4d38-a02b-484bfa26876f\" (UID: \"89937d9b-eafb-4d38-a02b-484bfa26876f\") "
Oct 30 13:52:08.605924 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:08.605748    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89937d9b-eafb-4d38-a02b-484bfa26876f-kube-api-access-ddts2\" (UniqueName: \"kubernetes.io/projected/89937d9b-eafb-4d38-a02b-484bfa26876f-kube-api-access-ddts2\") pod \"89937d9b-eafb-4d38-a02b-484bfa26876f\" (UID: \"89937d9b-eafb-4d38-a02b-484bfa26876f\") "
Oct 30 13:52:08.605924 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:08.605789    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89937d9b-eafb-4d38-a02b-484bfa26876f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89937d9b-eafb-4d38-a02b-484bfa26876f-openshift-service-ca\") pod \"89937d9b-eafb-4d38-a02b-484bfa26876f\" (UID: \"89937d9b-eafb-4d38-a02b-484bfa26876f\") "
Oct 30 13:52:08.605924 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:08.605808    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89937d9b-eafb-4d38-a02b-484bfa26876f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89937d9b-eafb-4d38-a02b-484bfa26876f-maas-api-tls\") pod \"89937d9b-eafb-4d38-a02b-484bfa26876f\" (UID: \"89937d9b-eafb-4d38-a02b-484bfa26876f\") "
Oct 30 13:52:08.606223 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:08.606190    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89937d9b-eafb-4d38-a02b-484bfa26876f-openshift-service-ca" pod "89937d9b-eafb-4d38-a02b-484bfa26876f" (UID: "89937d9b-eafb-4d38-a02b-484bfa26876f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:52:08.608009 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:08.607978    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89937d9b-eafb-4d38-a02b-484bfa26876f-maas-api-tls" pod "89937d9b-eafb-4d38-a02b-484bfa26876f" (UID: "89937d9b-eafb-4d38-a02b-484bfa26876f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:52:08.608112 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:08.608059    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89937d9b-eafb-4d38-a02b-484bfa26876f-kube-api-access-ddts2" pod "89937d9b-eafb-4d38-a02b-484bfa26876f" (UID: "89937d9b-eafb-4d38-a02b-484bfa26876f"). InnerVolumeSpecName "kube-api-access-ddts2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:52:08.608112 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:08.608095    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89937d9b-eafb-4d38-a02b-484bfa26876f-metrics-certs" pod "89937d9b-eafb-4d38-a02b-484bfa26876f" (UID: "89937d9b-eafb-4d38-a02b-484bfa26876f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:52:08.707089 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:08.707053    4364 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89937d9b-eafb-4d38-a02b-484bfa26876f-openshift-service-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:52:08.707089 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:08.707083    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89937d9b-eafb-4d38-a02b-484bfa26876f-maas-api-tls\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:52:08.707089 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:08.707092    4364 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89937d9b-eafb-4d38-a02b-484bfa26876f-metrics-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:52:08.707340 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:08.707102    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddts2\" (UniqueName: \"kubernetes.io/projected/89937d9b-eafb-4d38-a02b-484bfa26876f-kube-api-access-ddts2\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:52:09.420716 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:09.420673    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a5b3f0-84b6ff667d-skmlp" event={"ID":"89937d9b-eafb-4d38-a02b-484bfa26876f","Type":"ContainerDied","Data":"332bad891fe74329d6e7e283b940199baa549bdb2886254478be479ecb763c5b"}
Oct 30 13:52:09.421174 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:09.420693    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a5b3f0-84b6ff667d-skmlp"
Oct 30 13:52:09.421174 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:09.420780    4364 scope.go:122] "RemoveContainer" containerID="3e2ae70acfd70ba8a4c5e520a9e4216724f8477b398ea4f7a640f2ef2d514862"
Oct 30 13:52:09.473241 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:09.473208    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a5b3f0-84b6ff667d-skmlp"]
Oct 30 13:52:09.485484 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:09.485456    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a5b3f0-84b6ff667d-skmlp"]
Oct 30 13:52:10.314019 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:10.313984    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:52:10.969578 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:10.969547    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89937d9b-eafb-4d38-a02b-484bfa26876f" path="/var/lib/kubelet/pods/89937d9b-eafb-4d38-a02b-484bfa26876f/volumes"
Oct 30 13:52:11.504706 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:52:11.504670    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:52:12.258059 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:12.257995    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a5b3f0-c9d8565db4826q" podUID="84430325-7922-4f93-bcd7-616ebddc1a2d" containerName="payload-processing" containerID="cri-o://e1c7c4203e4ff74527aeb48e200f9f87b31f5b049c3deb84106b3b3921d16c3f" gracePeriod=25
Oct 30 13:52:12.287591 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:12.287549    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a5b3f0-76587mqgpt" podUID="9cb7160f-e10f-4f3c-8327-65d508ccc923" containerName="payload-pre-processing" containerID="cri-o://fbc002f3193917d5cf56c5f9dd608f231ced24b3cf23bcca7a8b8194eaddd9da" gracePeriod=25
Oct 30 13:52:14.029060 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:14.029026    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-l5b9d"]
Oct 30 13:52:14.040977 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:14.040946    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-l5b9d"]
Oct 30 13:52:14.969102 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:14.969071    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d96f9cb-87aa-476c-be7e-e22ccb69c755" path="/var/lib/kubelet/pods/9d96f9cb-87aa-476c-be7e-e22ccb69c755/volumes"
Oct 30 13:52:15.658268 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:15.658226    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9"]
Oct 30 13:52:15.658717 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:15.658498    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9" podUID="3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72" containerName="istio-proxy" containerID="cri-o://f90488186bf96a52b8e5bdabb12e974a6e48c76663ad272290c4152f487fb8dc" gracePeriod=30
Oct 30 13:52:16.692763 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:16.692697    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-a5b3f0/simulator-single-main-a5b3f0-kserve-5d9bf8ff77-8cpqh" podUID="0dc02570-e085-4987-94eb-ab9efda42c0c" containerName="main" containerID="cri-o://71d589a01109d4e83c614416bf6a8d44fffc76b3f340d08c1c684223cd6609c4" gracePeriod=45
Oct 30 13:52:16.908638 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:16.908592    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-65lbb"]
Oct 30 13:52:16.921068 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:16.921039    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-65lbb"]
Oct 30 13:52:16.951565 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:16.951510    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-a5b3f0/simulator-single-main-a5b3f0-kserve-5d9bf8ff77-8cpqh"
Oct 30 13:52:16.958136 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:16.958108    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ps4mr"]
Oct 30 13:52:16.958580 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:16.958560    4364 cpu_manager.go:414] "containerMap: removing container" podUID="0dc02570-e085-4987-94eb-ab9efda42c0c" containerName="main"
Oct 30 13:52:16.958580 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:16.958577    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dc02570-e085-4987-94eb-ab9efda42c0c" containerName="main"
Oct 30 13:52:16.958839 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:16.958593    4364 cpu_manager.go:414] "containerMap: removing container" podUID="89937d9b-eafb-4d38-a02b-484bfa26876f" containerName="maas-api"
Oct 30 13:52:16.958839 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:16.958610    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89937d9b-eafb-4d38-a02b-484bfa26876f" containerName="maas-api"
Oct 30 13:52:16.958839 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:16.958688    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dc02570-e085-4987-94eb-ab9efda42c0c" containerName="main"
Oct 30 13:52:16.958839 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:16.958697    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="89937d9b-eafb-4d38-a02b-484bfa26876f" containerName="maas-api"
Oct 30 13:52:16.961578 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:16.961562    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ps4mr"
Oct 30 13:52:16.968583 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:16.968561    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2641d4ad-4a62-41d9-9081-7dece03c3ae7" path="/var/lib/kubelet/pods/2641d4ad-4a62-41d9-9081-7dece03c3ae7/volumes"
Oct 30 13:52:16.977417 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:16.977394    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:52:16.977564 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:16.977530    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:52:16.977564 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:16.977535    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:52:16.977710 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:16.977661    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\""
Oct 30 13:52:16.981224 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:16.981202    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ps4mr"]
Oct 30 13:52:17.098874 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:17.098843    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0dc02570-e085-4987-94eb-ab9efda42c0c-tls-certs\" (UniqueName: \"kubernetes.io/secret/0dc02570-e085-4987-94eb-ab9efda42c0c-tls-certs\") pod \"0dc02570-e085-4987-94eb-ab9efda42c0c\" (UID: \"0dc02570-e085-4987-94eb-ab9efda42c0c\") "
Oct 30 13:52:17.099060 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:17.098947    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dc02570-e085-4987-94eb-ab9efda42c0c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dc02570-e085-4987-94eb-ab9efda42c0c-model-cache\") pod \"0dc02570-e085-4987-94eb-ab9efda42c0c\" (UID: \"0dc02570-e085-4987-94eb-ab9efda42c0c\") "
Oct 30 13:52:17.099060 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:17.098995    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dc02570-e085-4987-94eb-ab9efda42c0c-kube-api-access-dvb47\" (UniqueName: \"kubernetes.io/projected/0dc02570-e085-4987-94eb-ab9efda42c0c-kube-api-access-dvb47\") pod \"0dc02570-e085-4987-94eb-ab9efda42c0c\" (UID: \"0dc02570-e085-4987-94eb-ab9efda42c0c\") "
Oct 30 13:52:17.099060 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:17.099020    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dc02570-e085-4987-94eb-ab9efda42c0c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dc02570-e085-4987-94eb-ab9efda42c0c-dshm\") pod \"0dc02570-e085-4987-94eb-ab9efda42c0c\" (UID: \"0dc02570-e085-4987-94eb-ab9efda42c0c\") "
Oct 30 13:52:17.099188 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:17.099066    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dc02570-e085-4987-94eb-ab9efda42c0c-home\" (UniqueName: \"kubernetes.io/empty-dir/0dc02570-e085-4987-94eb-ab9efda42c0c-home\") pod \"0dc02570-e085-4987-94eb-ab9efda42c0c\" (UID: \"0dc02570-e085-4987-94eb-ab9efda42c0c\") "
Oct 30 13:52:17.099188 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:17.099113    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dc02570-e085-4987-94eb-ab9efda42c0c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dc02570-e085-4987-94eb-ab9efda42c0c-tmp-dir\") pod \"0dc02570-e085-4987-94eb-ab9efda42c0c\" (UID: \"0dc02570-e085-4987-94eb-ab9efda42c0c\") "
Oct 30 13:52:17.099273 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:17.099211    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dc02570-e085-4987-94eb-ab9efda42c0c-model-cache" pod "0dc02570-e085-4987-94eb-ab9efda42c0c" (UID: "0dc02570-e085-4987-94eb-ab9efda42c0c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:52:17.099344 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:17.099320    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvvzb\" (UniqueName: \"kubernetes.io/projected/7a7ba96d-104e-4933-b40d-13127dc2f8fb-kube-api-access-zvvzb\") pod \"maas-api-revoke-sub-3d4156f1-ps4mr\" (UID: \"7a7ba96d-104e-4933-b40d-13127dc2f8fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ps4mr"
Oct 30 13:52:17.099403 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:17.099368    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dc02570-e085-4987-94eb-ab9efda42c0c-home" pod "0dc02570-e085-4987-94eb-ab9efda42c0c" (UID: "0dc02570-e085-4987-94eb-ab9efda42c0c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:52:17.099464 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:17.099418    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dc02570-e085-4987-94eb-ab9efda42c0c-tmp-dir" pod "0dc02570-e085-4987-94eb-ab9efda42c0c" (UID: "0dc02570-e085-4987-94eb-ab9efda42c0c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:52:17.099587 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:17.099566    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a7ba96d-104e-4933-b40d-13127dc2f8fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ps4mr\" (UID: \"7a7ba96d-104e-4933-b40d-13127dc2f8fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ps4mr"
Oct 30 13:52:17.099722 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:17.099705    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a7ba96d-104e-4933-b40d-13127dc2f8fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ps4mr\" (UID: \"7a7ba96d-104e-4933-b40d-13127dc2f8fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ps4mr"
Oct 30 13:52:17.099853 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:17.099839    4364 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dc02570-e085-4987-94eb-ab9efda42c0c-model-cache\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:52:17.099891 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:17.099859    4364 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0dc02570-e085-4987-94eb-ab9efda42c0c-home\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:52:17.099891 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:17.099875    4364 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dc02570-e085-4987-94eb-ab9efda42c0c-tmp-dir\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:52:17.101246 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:17.101214    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dc02570-e085-4987-94eb-ab9efda42c0c-kube-api-access-dvb47" pod "0dc02570-e085-4987-94eb-ab9efda42c0c" (UID: "0dc02570-e085-4987-94eb-ab9efda42c0c"). InnerVolumeSpecName "kube-api-access-dvb47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:52:17.101464 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:17.101444    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dc02570-e085-4987-94eb-ab9efda42c0c-tls-certs" pod "0dc02570-e085-4987-94eb-ab9efda42c0c" (UID: "0dc02570-e085-4987-94eb-ab9efda42c0c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:52:17.101537 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:17.101519    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dc02570-e085-4987-94eb-ab9efda42c0c-dshm" pod "0dc02570-e085-4987-94eb-ab9efda42c0c" (UID: "0dc02570-e085-4987-94eb-ab9efda42c0c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:52:17.201055 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:17.201015    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvvzb\" (UniqueName: \"kubernetes.io/projected/7a7ba96d-104e-4933-b40d-13127dc2f8fb-kube-api-access-zvvzb\") pod \"maas-api-revoke-sub-3d4156f1-ps4mr\" (UID: \"7a7ba96d-104e-4933-b40d-13127dc2f8fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ps4mr"
Oct 30 13:52:17.201231 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:17.201145    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a7ba96d-104e-4933-b40d-13127dc2f8fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ps4mr\" (UID: \"7a7ba96d-104e-4933-b40d-13127dc2f8fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ps4mr"
Oct 30 13:52:17.201231 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:17.201209    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a7ba96d-104e-4933-b40d-13127dc2f8fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ps4mr\" (UID: \"7a7ba96d-104e-4933-b40d-13127dc2f8fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ps4mr"
Oct 30 13:52:17.201369 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:17.201302    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvb47\" (UniqueName: \"kubernetes.io/projected/0dc02570-e085-4987-94eb-ab9efda42c0c-kube-api-access-dvb47\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:52:17.201369 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:17.201318    4364 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dc02570-e085-4987-94eb-ab9efda42c0c-dshm\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:52:17.201369 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:17.201335    4364 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dc02570-e085-4987-94eb-ab9efda42c0c-tls-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:52:17.201849 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:17.201800    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a7ba96d-104e-4933-b40d-13127dc2f8fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ps4mr\" (UID: \"7a7ba96d-104e-4933-b40d-13127dc2f8fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ps4mr"
Oct 30 13:52:17.203458 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:17.203440    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a7ba96d-104e-4933-b40d-13127dc2f8fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ps4mr\" (UID: \"7a7ba96d-104e-4933-b40d-13127dc2f8fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ps4mr"
Oct 30 13:52:17.228252 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:17.228220    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvvzb\" (UniqueName: \"kubernetes.io/projected/7a7ba96d-104e-4933-b40d-13127dc2f8fb-kube-api-access-zvvzb\") pod \"maas-api-revoke-sub-3d4156f1-ps4mr\" (UID: \"7a7ba96d-104e-4933-b40d-13127dc2f8fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ps4mr"
Oct 30 13:52:17.274800 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:17.274771    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ps4mr"
Oct 30 13:52:17.466375 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:17.466282    4364 generic.go:356] "Generic (PLEG): container finished" podID="0dc02570-e085-4987-94eb-ab9efda42c0c" containerID="71d589a01109d4e83c614416bf6a8d44fffc76b3f340d08c1c684223cd6609c4" exitCode=0
Oct 30 13:52:17.466375 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:17.466327    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-a5b3f0/simulator-single-main-a5b3f0-kserve-5d9bf8ff77-8cpqh"
Oct 30 13:52:17.466375 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:17.466361    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-a5b3f0/simulator-single-main-a5b3f0-kserve-5d9bf8ff77-8cpqh" event={"ID":"0dc02570-e085-4987-94eb-ab9efda42c0c","Type":"ContainerDied","Data":"71d589a01109d4e83c614416bf6a8d44fffc76b3f340d08c1c684223cd6609c4"}
Oct 30 13:52:17.466706 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:17.466413    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-a5b3f0/simulator-single-main-a5b3f0-kserve-5d9bf8ff77-8cpqh" event={"ID":"0dc02570-e085-4987-94eb-ab9efda42c0c","Type":"ContainerDied","Data":"9539fdaf3d62661d9d5b0b3eb2db1d5bbf680ec13e38b48979504f1392981d49"}
Oct 30 13:52:17.466706 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:17.466439    4364 scope.go:122] "RemoveContainer" containerID="71d589a01109d4e83c614416bf6a8d44fffc76b3f340d08c1c684223cd6609c4"
Oct 30 13:52:17.475756 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:17.475735    4364 scope.go:122] "RemoveContainer" containerID="71d589a01109d4e83c614416bf6a8d44fffc76b3f340d08c1c684223cd6609c4"
Oct 30 13:52:17.475996 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:52:17.475976    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71d589a01109d4e83c614416bf6a8d44fffc76b3f340d08c1c684223cd6609c4\": container with ID starting with 71d589a01109d4e83c614416bf6a8d44fffc76b3f340d08c1c684223cd6609c4 not found: ID does not exist" containerID="71d589a01109d4e83c614416bf6a8d44fffc76b3f340d08c1c684223cd6609c4"
Oct 30 13:52:17.476060 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:17.476002    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71d589a01109d4e83c614416bf6a8d44fffc76b3f340d08c1c684223cd6609c4"} err="failed to get container status \"71d589a01109d4e83c614416bf6a8d44fffc76b3f340d08c1c684223cd6609c4\": rpc error: code = NotFound desc = could not find container \"71d589a01109d4e83c614416bf6a8d44fffc76b3f340d08c1c684223cd6609c4\": container with ID starting with 71d589a01109d4e83c614416bf6a8d44fffc76b3f340d08c1c684223cd6609c4 not found: ID does not exist"
Oct 30 13:52:17.556519 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:17.556488    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-a5b3f0/simulator-single-main-a5b3f0-kserve-5d9bf8ff77-8cpqh"]
Oct 30 13:52:17.586633 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:17.586582    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-a5b3f0/simulator-single-main-a5b3f0-kserve-5d9bf8ff77-8cpqh"]
Oct 30 13:52:17.606767 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:17.606740    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ps4mr"]
Oct 30 13:52:17.610002 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:52:17.609968    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a7ba96d_104e_4933_b40d_13127dc2f8fb.slice/crio-7409f1e5e1a5761c9021978f7b63f2b29b9fffca6ba5a36572fbad8aaac1522a WatchSource:0}: Error finding container 7409f1e5e1a5761c9021978f7b63f2b29b9fffca6ba5a36572fbad8aaac1522a: Status 404 returned error can't find the container with id 7409f1e5e1a5761c9021978f7b63f2b29b9fffca6ba5a36572fbad8aaac1522a
Oct 30 13:52:18.038567 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:18.038536    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a5b3f0-76587mqgpt" podUID="9cb7160f-e10f-4f3c-8327-65d508ccc923" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:52:18.038921 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:18.038536    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a5b3f0-c9d8565db4826q" podUID="84430325-7922-4f93-bcd7-616ebddc1a2d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:52:18.473871 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:18.473840    4364 generic.go:356] "Generic (PLEG): container finished" podID="7a7ba96d-104e-4933-b40d-13127dc2f8fb" containerID="18f243fcd2277e1f178457e1fe65fb8af437d57cfbf1a1655b91be3893066c25" exitCode=0
Oct 30 13:52:18.474055 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:18.473927    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ps4mr" event={"ID":"7a7ba96d-104e-4933-b40d-13127dc2f8fb","Type":"ContainerDied","Data":"18f243fcd2277e1f178457e1fe65fb8af437d57cfbf1a1655b91be3893066c25"}
Oct 30 13:52:18.474055 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:18.473982    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ps4mr" event={"ID":"7a7ba96d-104e-4933-b40d-13127dc2f8fb","Type":"ContainerStarted","Data":"7409f1e5e1a5761c9021978f7b63f2b29b9fffca6ba5a36572fbad8aaac1522a"}
Oct 30 13:52:18.968713 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:18.968681    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dc02570-e085-4987-94eb-ab9efda42c0c" path="/var/lib/kubelet/pods/0dc02570-e085-4987-94eb-ab9efda42c0c/volumes"
Oct 30 13:52:19.615298 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:19.615275    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ps4mr"
Oct 30 13:52:19.726043 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:19.726012    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a7ba96d-104e-4933-b40d-13127dc2f8fb-kube-api-access-zvvzb\" (UniqueName: \"kubernetes.io/projected/7a7ba96d-104e-4933-b40d-13127dc2f8fb-kube-api-access-zvvzb\") pod \"7a7ba96d-104e-4933-b40d-13127dc2f8fb\" (UID: \"7a7ba96d-104e-4933-b40d-13127dc2f8fb\") "
Oct 30 13:52:19.726227 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:19.726075    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a7ba96d-104e-4933-b40d-13127dc2f8fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a7ba96d-104e-4933-b40d-13127dc2f8fb-maas-api-openshift-ca\") pod \"7a7ba96d-104e-4933-b40d-13127dc2f8fb\" (UID: \"7a7ba96d-104e-4933-b40d-13127dc2f8fb\") "
Oct 30 13:52:19.726227 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:19.726128    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a7ba96d-104e-4933-b40d-13127dc2f8fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a7ba96d-104e-4933-b40d-13127dc2f8fb-maas-api-xks-ca\") pod \"7a7ba96d-104e-4933-b40d-13127dc2f8fb\" (UID: \"7a7ba96d-104e-4933-b40d-13127dc2f8fb\") "
Oct 30 13:52:19.726441 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:19.726417    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a7ba96d-104e-4933-b40d-13127dc2f8fb-maas-api-openshift-ca" pod "7a7ba96d-104e-4933-b40d-13127dc2f8fb" (UID: "7a7ba96d-104e-4933-b40d-13127dc2f8fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:52:19.726529 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:19.726516    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a7ba96d-104e-4933-b40d-13127dc2f8fb-maas-api-openshift-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:52:19.728076 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:19.728048    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a7ba96d-104e-4933-b40d-13127dc2f8fb-maas-api-xks-ca" pod "7a7ba96d-104e-4933-b40d-13127dc2f8fb" (UID: "7a7ba96d-104e-4933-b40d-13127dc2f8fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:52:19.728200 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:19.728128    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a7ba96d-104e-4933-b40d-13127dc2f8fb-kube-api-access-zvvzb" pod "7a7ba96d-104e-4933-b40d-13127dc2f8fb" (UID: "7a7ba96d-104e-4933-b40d-13127dc2f8fb"). InnerVolumeSpecName "kube-api-access-zvvzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:52:19.827939 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:19.827915    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a7ba96d-104e-4933-b40d-13127dc2f8fb-maas-api-xks-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:52:19.827939 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:19.827940    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvvzb\" (UniqueName: \"kubernetes.io/projected/7a7ba96d-104e-4933-b40d-13127dc2f8fb-kube-api-access-zvvzb\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:52:20.493607 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:20.493561    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ps4mr"
Oct 30 13:52:20.493799 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:20.493585    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ps4mr" event={"ID":"7a7ba96d-104e-4933-b40d-13127dc2f8fb","Type":"ContainerDied","Data":"7409f1e5e1a5761c9021978f7b63f2b29b9fffca6ba5a36572fbad8aaac1522a"}
Oct 30 13:52:20.493799 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:20.493683    4364 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7409f1e5e1a5761c9021978f7b63f2b29b9fffca6ba5a36572fbad8aaac1522a"
Oct 30 13:52:20.912759 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:20.912737    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9"
Oct 30 13:52:21.041454 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:21.041386    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72-kube-api-access-htmcl\" (UniqueName: \"kubernetes.io/projected/3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72-kube-api-access-htmcl\") pod \"3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72\" (UID: \"3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72\") "
Oct 30 13:52:21.041454 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:21.041436    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72-istio-podinfo\") pod \"3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72\" (UID: \"3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72\") "
Oct 30 13:52:21.041733 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:21.041459    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72-istio-envoy\") pod \"3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72\" (UID: \"3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72\") "
Oct 30 13:52:21.041733 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:21.041526    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72-credential-socket\") pod \"3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72\" (UID: \"3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72\") "
Oct 30 13:52:21.041733 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:21.041579    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72-istiod-ca-cert\") pod \"3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72\" (UID: \"3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72\") "
Oct 30 13:52:21.041733 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:21.041664    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72-workload-certs\") pod \"3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72\" (UID: \"3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72\") "
Oct 30 13:52:21.041733 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:21.041708    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72-istio-data\") pod \"3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72\" (UID: \"3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72\") "
Oct 30 13:52:21.042020 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:21.041901    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72-credential-socket" pod "3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72" (UID: "3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:52:21.042020 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:21.041969    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72-istio-token\" (UniqueName: \"kubernetes.io/projected/3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72-istio-token\") pod \"3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72\" (UID: \"3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72\") "
Oct 30 13:52:21.042020 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:21.041992    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72-istio-data" pod "3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72" (UID: "3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:52:21.042020 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:21.042006    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72-workload-certs" pod "3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72" (UID: "3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:52:21.042248 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:21.042036    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72-workload-socket\") pod \"3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72\" (UID: \"3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72\") "
Oct 30 13:52:21.042248 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:21.042070    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72-istiod-ca-cert" pod "3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72" (UID: "3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:52:21.042430 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:21.042407    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72-workload-socket" pod "3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72" (UID: "3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:52:21.042498 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:21.042408    4364 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72-workload-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:52:21.042498 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:21.042452    4364 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72-istio-data\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:52:21.042498 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:21.042469    4364 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72-credential-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:52:21.042498 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:21.042487    4364 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72-istiod-ca-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:52:21.043768 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:21.043742    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72-kube-api-access-htmcl" pod "3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72" (UID: "3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72"). InnerVolumeSpecName "kube-api-access-htmcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:52:21.043872 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:21.043759    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72-istio-podinfo" pod "3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72" (UID: "3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:52:21.043872 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:21.043782    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72-istio-envoy" pod "3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72" (UID: "3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:52:21.043981 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:21.043933    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72-istio-token" pod "3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72" (UID: "3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:52:21.142948 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:21.142917    4364 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72-istio-token\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:52:21.142948 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:21.142944    4364 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72-workload-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:52:21.143102 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:21.142957    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htmcl\" (UniqueName: \"kubernetes.io/projected/3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72-kube-api-access-htmcl\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:52:21.143102 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:21.142971    4364 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72-istio-podinfo\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:52:21.143102 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:21.142983    4364 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72-istio-envoy\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:52:21.500319 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:21.500289    4364 generic.go:356] "Generic (PLEG): container finished" podID="3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72" containerID="f90488186bf96a52b8e5bdabb12e974a6e48c76663ad272290c4152f487fb8dc" exitCode=0
Oct 30 13:52:21.500495 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:21.500350    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9"
Oct 30 13:52:21.500495 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:21.500371    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9" event={"ID":"3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72","Type":"ContainerDied","Data":"f90488186bf96a52b8e5bdabb12e974a6e48c76663ad272290c4152f487fb8dc"}
Oct 30 13:52:21.500495 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:21.500430    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9" event={"ID":"3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72","Type":"ContainerDied","Data":"30674910959f3ed2e5d1f7c5d2b3faf71913a7549df48791dc4133534df678e0"}
Oct 30 13:52:21.500495 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:21.500445    4364 scope.go:122] "RemoveContainer" containerID="f90488186bf96a52b8e5bdabb12e974a6e48c76663ad272290c4152f487fb8dc"
Oct 30 13:52:21.509781 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:21.509767    4364 scope.go:122] "RemoveContainer" containerID="79394241230f61cca0db87f08a70b5afc5f7849944214026ac93770ac02eb337"
Oct 30 13:52:21.517937 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:21.517921    4364 scope.go:122] "RemoveContainer" containerID="f90488186bf96a52b8e5bdabb12e974a6e48c76663ad272290c4152f487fb8dc"
Oct 30 13:52:21.518192 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:52:21.518168    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f90488186bf96a52b8e5bdabb12e974a6e48c76663ad272290c4152f487fb8dc\": container with ID starting with f90488186bf96a52b8e5bdabb12e974a6e48c76663ad272290c4152f487fb8dc not found: ID does not exist" containerID="f90488186bf96a52b8e5bdabb12e974a6e48c76663ad272290c4152f487fb8dc"
Oct 30 13:52:21.518294 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:21.518199    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f90488186bf96a52b8e5bdabb12e974a6e48c76663ad272290c4152f487fb8dc"} err="failed to get container status \"f90488186bf96a52b8e5bdabb12e974a6e48c76663ad272290c4152f487fb8dc\": rpc error: code = NotFound desc = could not find container \"f90488186bf96a52b8e5bdabb12e974a6e48c76663ad272290c4152f487fb8dc\": container with ID starting with f90488186bf96a52b8e5bdabb12e974a6e48c76663ad272290c4152f487fb8dc not found: ID does not exist"
Oct 30 13:52:21.518294 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:21.518222    4364 scope.go:122] "RemoveContainer" containerID="79394241230f61cca0db87f08a70b5afc5f7849944214026ac93770ac02eb337"
Oct 30 13:52:21.518463 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:52:21.518443    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79394241230f61cca0db87f08a70b5afc5f7849944214026ac93770ac02eb337\": container with ID starting with 79394241230f61cca0db87f08a70b5afc5f7849944214026ac93770ac02eb337 not found: ID does not exist" containerID="79394241230f61cca0db87f08a70b5afc5f7849944214026ac93770ac02eb337"
Oct 30 13:52:21.518507 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:21.518470    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79394241230f61cca0db87f08a70b5afc5f7849944214026ac93770ac02eb337"} err="failed to get container status \"79394241230f61cca0db87f08a70b5afc5f7849944214026ac93770ac02eb337\": rpc error: code = NotFound desc = could not find container \"79394241230f61cca0db87f08a70b5afc5f7849944214026ac93770ac02eb337\": container with ID starting with 79394241230f61cca0db87f08a70b5afc5f7849944214026ac93770ac02eb337 not found: ID does not exist"
Oct 30 13:52:21.566710 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:21.566687    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9"]
Oct 30 13:52:21.578750 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:21.578724    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-a5b3f0-openshift-default-6cccdfb5cctp4z9"]
Oct 30 13:52:22.968194 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:22.968160    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72" path="/var/lib/kubelet/pods/3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72/volumes"
Oct 30 13:52:28.037981 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:28.037902    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a5b3f0-c9d8565db4826q" podUID="84430325-7922-4f93-bcd7-616ebddc1a2d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:52:28.038374 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:28.037902    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a5b3f0-76587mqgpt" podUID="9cb7160f-e10f-4f3c-8327-65d508ccc923" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:52:28.991333 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:28.991301    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:52:32.535869 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:32.535841    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a5b3f0-76587mqgpt"
Oct 30 13:52:32.556259 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:32.556237    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a5b3f0-c9d8565db4826q"
Oct 30 13:52:32.591665 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:32.584117    4364 generic.go:356] "Generic (PLEG): container finished" podID="9cb7160f-e10f-4f3c-8327-65d508ccc923" containerID="fbc002f3193917d5cf56c5f9dd608f231ced24b3cf23bcca7a8b8194eaddd9da" exitCode=0
Oct 30 13:52:32.591665 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:32.584373    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a5b3f0-76587mqgpt" event={"ID":"9cb7160f-e10f-4f3c-8327-65d508ccc923","Type":"ContainerDied","Data":"fbc002f3193917d5cf56c5f9dd608f231ced24b3cf23bcca7a8b8194eaddd9da"}
Oct 30 13:52:32.591665 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:32.584444    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a5b3f0-76587mqgpt" event={"ID":"9cb7160f-e10f-4f3c-8327-65d508ccc923","Type":"ContainerDied","Data":"c57cf68a8704a5b9b5c9c1671f15065d019fee611cc653c2504e70aaafce6312"}
Oct 30 13:52:32.591665 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:32.584477    4364 scope.go:122] "RemoveContainer" containerID="fbc002f3193917d5cf56c5f9dd608f231ced24b3cf23bcca7a8b8194eaddd9da"
Oct 30 13:52:32.591665 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:32.584674    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a5b3f0-76587mqgpt"
Oct 30 13:52:32.603085 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:32.603055    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cb7160f-e10f-4f3c-8327-65d508ccc923-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cb7160f-e10f-4f3c-8327-65d508ccc923-plugins-config-volume\") pod \"9cb7160f-e10f-4f3c-8327-65d508ccc923\" (UID: \"9cb7160f-e10f-4f3c-8327-65d508ccc923\") "
Oct 30 13:52:32.603584 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:32.603326    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84430325-7922-4f93-bcd7-616ebddc1a2d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84430325-7922-4f93-bcd7-616ebddc1a2d-plugins-config-volume\") pod \"84430325-7922-4f93-bcd7-616ebddc1a2d\" (UID: \"84430325-7922-4f93-bcd7-616ebddc1a2d\") "
Oct 30 13:52:32.603831 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:32.603800    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cb7160f-e10f-4f3c-8327-65d508ccc923-kube-api-access-gvbz5\" (UniqueName: \"kubernetes.io/projected/9cb7160f-e10f-4f3c-8327-65d508ccc923-kube-api-access-gvbz5\") pod \"9cb7160f-e10f-4f3c-8327-65d508ccc923\" (UID: \"9cb7160f-e10f-4f3c-8327-65d508ccc923\") "
Oct 30 13:52:32.604478 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:32.604452    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84430325-7922-4f93-bcd7-616ebddc1a2d-kube-api-access-8gdst\" (UniqueName: \"kubernetes.io/projected/84430325-7922-4f93-bcd7-616ebddc1a2d-kube-api-access-8gdst\") pod \"84430325-7922-4f93-bcd7-616ebddc1a2d\" (UID: \"84430325-7922-4f93-bcd7-616ebddc1a2d\") "
Oct 30 13:52:32.607356 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:32.607324    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cb7160f-e10f-4f3c-8327-65d508ccc923-plugins-config-volume" pod "9cb7160f-e10f-4f3c-8327-65d508ccc923" (UID: "9cb7160f-e10f-4f3c-8327-65d508ccc923"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:52:32.608776 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:32.608750    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84430325-7922-4f93-bcd7-616ebddc1a2d-plugins-config-volume" pod "84430325-7922-4f93-bcd7-616ebddc1a2d" (UID: "84430325-7922-4f93-bcd7-616ebddc1a2d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:52:32.611401 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:32.611369    4364 generic.go:356] "Generic (PLEG): container finished" podID="84430325-7922-4f93-bcd7-616ebddc1a2d" containerID="e1c7c4203e4ff74527aeb48e200f9f87b31f5b049c3deb84106b3b3921d16c3f" exitCode=0
Oct 30 13:52:32.611508 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:32.611488    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a5b3f0-c9d8565db4826q" event={"ID":"84430325-7922-4f93-bcd7-616ebddc1a2d","Type":"ContainerDied","Data":"e1c7c4203e4ff74527aeb48e200f9f87b31f5b049c3deb84106b3b3921d16c3f"}
Oct 30 13:52:32.611627 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:32.611594    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a5b3f0-c9d8565db4826q" event={"ID":"84430325-7922-4f93-bcd7-616ebddc1a2d","Type":"ContainerDied","Data":"86fbae921031cb4ef6afe84b6dd39c7e095ea112eb1650ffc01a567e7c1558b3"}
Oct 30 13:52:32.611754 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:32.611736    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a5b3f0-c9d8565db4826q"
Oct 30 13:52:32.626440 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:32.626412    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84430325-7922-4f93-bcd7-616ebddc1a2d-kube-api-access-8gdst" pod "84430325-7922-4f93-bcd7-616ebddc1a2d" (UID: "84430325-7922-4f93-bcd7-616ebddc1a2d"). InnerVolumeSpecName "kube-api-access-8gdst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:52:32.629657 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:32.625791    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cb7160f-e10f-4f3c-8327-65d508ccc923-kube-api-access-gvbz5" pod "9cb7160f-e10f-4f3c-8327-65d508ccc923" (UID: "9cb7160f-e10f-4f3c-8327-65d508ccc923"). InnerVolumeSpecName "kube-api-access-gvbz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:52:32.650643 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:32.650590    4364 scope.go:122] "RemoveContainer" containerID="fbc002f3193917d5cf56c5f9dd608f231ced24b3cf23bcca7a8b8194eaddd9da"
Oct 30 13:52:32.651250 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:52:32.651022    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbc002f3193917d5cf56c5f9dd608f231ced24b3cf23bcca7a8b8194eaddd9da\": container with ID starting with fbc002f3193917d5cf56c5f9dd608f231ced24b3cf23bcca7a8b8194eaddd9da not found: ID does not exist" containerID="fbc002f3193917d5cf56c5f9dd608f231ced24b3cf23bcca7a8b8194eaddd9da"
Oct 30 13:52:32.651250 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:32.651061    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbc002f3193917d5cf56c5f9dd608f231ced24b3cf23bcca7a8b8194eaddd9da"} err="failed to get container status \"fbc002f3193917d5cf56c5f9dd608f231ced24b3cf23bcca7a8b8194eaddd9da\": rpc error: code = NotFound desc = could not find container \"fbc002f3193917d5cf56c5f9dd608f231ced24b3cf23bcca7a8b8194eaddd9da\": container with ID starting with fbc002f3193917d5cf56c5f9dd608f231ced24b3cf23bcca7a8b8194eaddd9da not found: ID does not exist"
Oct 30 13:52:32.651250 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:32.651085    4364 scope.go:122] "RemoveContainer" containerID="e1c7c4203e4ff74527aeb48e200f9f87b31f5b049c3deb84106b3b3921d16c3f"
Oct 30 13:52:32.662989 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:32.662960    4364 scope.go:122] "RemoveContainer" containerID="e1c7c4203e4ff74527aeb48e200f9f87b31f5b049c3deb84106b3b3921d16c3f"
Oct 30 13:52:32.663355 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:52:32.663319    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1c7c4203e4ff74527aeb48e200f9f87b31f5b049c3deb84106b3b3921d16c3f\": container with ID starting with e1c7c4203e4ff74527aeb48e200f9f87b31f5b049c3deb84106b3b3921d16c3f not found: ID does not exist" containerID="e1c7c4203e4ff74527aeb48e200f9f87b31f5b049c3deb84106b3b3921d16c3f"
Oct 30 13:52:32.663459 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:32.663356    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1c7c4203e4ff74527aeb48e200f9f87b31f5b049c3deb84106b3b3921d16c3f"} err="failed to get container status \"e1c7c4203e4ff74527aeb48e200f9f87b31f5b049c3deb84106b3b3921d16c3f\": rpc error: code = NotFound desc = could not find container \"e1c7c4203e4ff74527aeb48e200f9f87b31f5b049c3deb84106b3b3921d16c3f\": container with ID starting with e1c7c4203e4ff74527aeb48e200f9f87b31f5b049c3deb84106b3b3921d16c3f not found: ID does not exist"
Oct 30 13:52:32.707064 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:32.707032    4364 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cb7160f-e10f-4f3c-8327-65d508ccc923-plugins-config-volume\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:52:32.707064 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:32.707060    4364 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84430325-7922-4f93-bcd7-616ebddc1a2d-plugins-config-volume\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:52:32.707064 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:32.707070    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvbz5\" (UniqueName: \"kubernetes.io/projected/9cb7160f-e10f-4f3c-8327-65d508ccc923-kube-api-access-gvbz5\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:52:32.707304 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:32.707079    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gdst\" (UniqueName: \"kubernetes.io/projected/84430325-7922-4f93-bcd7-616ebddc1a2d-kube-api-access-8gdst\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:52:32.934708 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:32.934678    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a5b3f0-76587mqgpt"]
Oct 30 13:52:32.946512 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:32.946486    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a5b3f0-76587mqgpt"]
Oct 30 13:52:32.968919 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:32.968895    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cb7160f-e10f-4f3c-8327-65d508ccc923" path="/var/lib/kubelet/pods/9cb7160f-e10f-4f3c-8327-65d508ccc923/volumes"
Oct 30 13:52:32.973624 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:32.973586    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a5b3f0-c9d8565db4826q"]
Oct 30 13:52:32.977794 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:32.977774    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a5b3f0-c9d8565db4826q"]
Oct 30 13:52:34.307163 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:34.307132    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:52:34.969056 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:34.969024    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84430325-7922-4f93-bcd7-616ebddc1a2d" path="/var/lib/kubelet/pods/84430325-7922-4f93-bcd7-616ebddc1a2d/volumes"
Oct 30 13:52:41.503800 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:52:41.503758    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:52:44.769860 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:52:44.769821    4364 scope.go:122] "RemoveContainer" containerID="2093bc5327f48b99611ee4bd6d24a26df4d82e3bb3f868c9c6a5a0ca6bd30a0c"
Oct 30 13:53:11.506875 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:53:11.506832    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:53:41.504299 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:53:41.504258    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:53:44.810946 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:53:44.810909    4364 scope.go:122] "RemoveContainer" containerID="7bfa11d58ba7a4a5758e480cb1c8ddc5de8629c372936fc40290f7c81d45f2cc"
Oct 30 13:54:01.740623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:01.740516    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:54:05.775510 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:05.775469    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-k2ft9"]
Oct 30 13:54:05.775939 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:05.775907    4364 cpu_manager.go:414] "containerMap: removing container" podUID="3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72" containerName="istio-proxy"
Oct 30 13:54:05.775939 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:05.775921    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72" containerName="istio-proxy"
Oct 30 13:54:05.775939 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:05.775929    4364 cpu_manager.go:414] "containerMap: removing container" podUID="9cb7160f-e10f-4f3c-8327-65d508ccc923" containerName="payload-pre-processing"
Oct 30 13:54:05.775939 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:05.775935    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cb7160f-e10f-4f3c-8327-65d508ccc923" containerName="payload-pre-processing"
Oct 30 13:54:05.776099 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:05.775949    4364 cpu_manager.go:414] "containerMap: removing container" podUID="3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72" containerName="istio-proxy"
Oct 30 13:54:05.776099 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:05.775956    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72" containerName="istio-proxy"
Oct 30 13:54:05.776099 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:05.775969    4364 cpu_manager.go:414] "containerMap: removing container" podUID="7a7ba96d-104e-4933-b40d-13127dc2f8fb" containerName="revoke-keys"
Oct 30 13:54:05.776099 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:05.775974    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a7ba96d-104e-4933-b40d-13127dc2f8fb" containerName="revoke-keys"
Oct 30 13:54:05.776099 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:05.775983    4364 cpu_manager.go:414] "containerMap: removing container" podUID="84430325-7922-4f93-bcd7-616ebddc1a2d" containerName="payload-processing"
Oct 30 13:54:05.776099 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:05.775987    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84430325-7922-4f93-bcd7-616ebddc1a2d" containerName="payload-processing"
Oct 30 13:54:05.776099 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:05.776044    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a7ba96d-104e-4933-b40d-13127dc2f8fb" containerName="revoke-keys"
Oct 30 13:54:05.776099 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:05.776052    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="84430325-7922-4f93-bcd7-616ebddc1a2d" containerName="payload-processing"
Oct 30 13:54:05.776099 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:05.776064    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72" containerName="istio-proxy"
Oct 30 13:54:05.776099 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:05.776069    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cb7160f-e10f-4f3c-8327-65d508ccc923" containerName="payload-pre-processing"
Oct 30 13:54:05.776099 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:05.776075    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f2c06e3-2c7f-4eaa-8153-6fd66eb78f72" containerName="istio-proxy"
Oct 30 13:54:05.779303 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:05.779284    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-k2ft9"
Oct 30 13:54:05.795079 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:05.795051    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:54:05.795214 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:05.795160    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:54:05.807471 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:05.807449    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:54:05.807689 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:05.807567    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\""
Oct 30 13:54:05.810037 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:05.810015    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-k2ft9"]
Oct 30 13:54:05.918842 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:05.918815    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht422\" (UniqueName: \"kubernetes.io/projected/17a1d683-7bf3-4c03-8223-2ca23778288c-kube-api-access-ht422\") pod \"maas-api-revoke-sub-d576e68b-k2ft9\" (UID: \"17a1d683-7bf3-4c03-8223-2ca23778288c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-k2ft9"
Oct 30 13:54:05.918963 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:05.918850    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17a1d683-7bf3-4c03-8223-2ca23778288c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-k2ft9\" (UID: \"17a1d683-7bf3-4c03-8223-2ca23778288c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-k2ft9"
Oct 30 13:54:05.918963 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:05.918933    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17a1d683-7bf3-4c03-8223-2ca23778288c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-k2ft9\" (UID: \"17a1d683-7bf3-4c03-8223-2ca23778288c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-k2ft9"
Oct 30 13:54:06.020321 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:06.020297    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht422\" (UniqueName: \"kubernetes.io/projected/17a1d683-7bf3-4c03-8223-2ca23778288c-kube-api-access-ht422\") pod \"maas-api-revoke-sub-d576e68b-k2ft9\" (UID: \"17a1d683-7bf3-4c03-8223-2ca23778288c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-k2ft9"
Oct 30 13:54:06.020430 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:06.020328    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17a1d683-7bf3-4c03-8223-2ca23778288c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-k2ft9\" (UID: \"17a1d683-7bf3-4c03-8223-2ca23778288c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-k2ft9"
Oct 30 13:54:06.020482 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:06.020449    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17a1d683-7bf3-4c03-8223-2ca23778288c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-k2ft9\" (UID: \"17a1d683-7bf3-4c03-8223-2ca23778288c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-k2ft9"
Oct 30 13:54:06.021051 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:06.021029    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17a1d683-7bf3-4c03-8223-2ca23778288c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-k2ft9\" (UID: \"17a1d683-7bf3-4c03-8223-2ca23778288c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-k2ft9"
Oct 30 13:54:06.022713 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:06.022691    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17a1d683-7bf3-4c03-8223-2ca23778288c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-k2ft9\" (UID: \"17a1d683-7bf3-4c03-8223-2ca23778288c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-k2ft9"
Oct 30 13:54:06.043565 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:06.043510    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht422\" (UniqueName: \"kubernetes.io/projected/17a1d683-7bf3-4c03-8223-2ca23778288c-kube-api-access-ht422\") pod \"maas-api-revoke-sub-d576e68b-k2ft9\" (UID: \"17a1d683-7bf3-4c03-8223-2ca23778288c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-k2ft9"
Oct 30 13:54:06.092320 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:06.092296    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-k2ft9"
Oct 30 13:54:06.388529 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:06.388473    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-k2ft9"]
Oct 30 13:54:06.390722 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:54:06.390695    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17a1d683_7bf3_4c03_8223_2ca23778288c.slice/crio-8d85670b9898e19073533e5ce1bff939c73975db818a0ebbbde74c33f62cb361 WatchSource:0}: Error finding container 8d85670b9898e19073533e5ce1bff939c73975db818a0ebbbde74c33f62cb361: Status 404 returned error can't find the container with id 8d85670b9898e19073533e5ce1bff939c73975db818a0ebbbde74c33f62cb361
Oct 30 13:54:07.166360 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:07.166319    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-k2ft9" event={"ID":"17a1d683-7bf3-4c03-8223-2ca23778288c","Type":"ContainerStarted","Data":"757305002e433f193ff118f678eebe04b0ac0724825d9553324b7006a0a1d4aa"}
Oct 30 13:54:07.166814 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:07.166410    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-k2ft9" event={"ID":"17a1d683-7bf3-4c03-8223-2ca23778288c","Type":"ContainerStarted","Data":"8d85670b9898e19073533e5ce1bff939c73975db818a0ebbbde74c33f62cb361"}
Oct 30 13:54:07.212933 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:07.212882    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-k2ft9" podStartSLOduration=2.212866345 podStartE2EDuration="2.212866345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:54:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:54:07.210505529 +0000 UTC m=+3746.376870178" watchObservedRunningTime="2026-10-30 13:54:07.212866345 +0000 UTC m=+3746.379230993"
Oct 30 13:54:08.173513 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:08.173469    4364 generic.go:356] "Generic (PLEG): container finished" podID="17a1d683-7bf3-4c03-8223-2ca23778288c" containerID="757305002e433f193ff118f678eebe04b0ac0724825d9553324b7006a0a1d4aa" exitCode=0
Oct 30 13:54:08.173966 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:08.173555    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-k2ft9" event={"ID":"17a1d683-7bf3-4c03-8223-2ca23778288c","Type":"ContainerDied","Data":"757305002e433f193ff118f678eebe04b0ac0724825d9553324b7006a0a1d4aa"}
Oct 30 13:54:09.329997 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:09.329964    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-k2ft9"
Oct 30 13:54:09.452270 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:09.452172    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17a1d683-7bf3-4c03-8223-2ca23778288c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17a1d683-7bf3-4c03-8223-2ca23778288c-maas-api-openshift-ca\") pod \"17a1d683-7bf3-4c03-8223-2ca23778288c\" (UID: \"17a1d683-7bf3-4c03-8223-2ca23778288c\") "
Oct 30 13:54:09.452468 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:09.452282    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17a1d683-7bf3-4c03-8223-2ca23778288c-kube-api-access-ht422\" (UniqueName: \"kubernetes.io/projected/17a1d683-7bf3-4c03-8223-2ca23778288c-kube-api-access-ht422\") pod \"17a1d683-7bf3-4c03-8223-2ca23778288c\" (UID: \"17a1d683-7bf3-4c03-8223-2ca23778288c\") "
Oct 30 13:54:09.452468 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:09.452350    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17a1d683-7bf3-4c03-8223-2ca23778288c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17a1d683-7bf3-4c03-8223-2ca23778288c-maas-api-xks-ca\") pod \"17a1d683-7bf3-4c03-8223-2ca23778288c\" (UID: \"17a1d683-7bf3-4c03-8223-2ca23778288c\") "
Oct 30 13:54:09.452642 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:09.452571    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17a1d683-7bf3-4c03-8223-2ca23778288c-maas-api-openshift-ca" pod "17a1d683-7bf3-4c03-8223-2ca23778288c" (UID: "17a1d683-7bf3-4c03-8223-2ca23778288c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:54:09.452718 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:09.452704    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17a1d683-7bf3-4c03-8223-2ca23778288c-maas-api-openshift-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:54:09.454524 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:09.454492    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17a1d683-7bf3-4c03-8223-2ca23778288c-maas-api-xks-ca" pod "17a1d683-7bf3-4c03-8223-2ca23778288c" (UID: "17a1d683-7bf3-4c03-8223-2ca23778288c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:54:09.454650 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:09.454585    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17a1d683-7bf3-4c03-8223-2ca23778288c-kube-api-access-ht422" pod "17a1d683-7bf3-4c03-8223-2ca23778288c" (UID: "17a1d683-7bf3-4c03-8223-2ca23778288c"). InnerVolumeSpecName "kube-api-access-ht422". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:54:09.554122 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:09.554071    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ht422\" (UniqueName: \"kubernetes.io/projected/17a1d683-7bf3-4c03-8223-2ca23778288c-kube-api-access-ht422\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:54:09.554122 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:09.554114    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17a1d683-7bf3-4c03-8223-2ca23778288c-maas-api-xks-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:54:10.189799 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:10.189765    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-k2ft9" event={"ID":"17a1d683-7bf3-4c03-8223-2ca23778288c","Type":"ContainerDied","Data":"8d85670b9898e19073533e5ce1bff939c73975db818a0ebbbde74c33f62cb361"}
Oct 30 13:54:10.190018 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:10.189803    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-k2ft9"
Oct 30 13:54:10.190018 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:10.189827    4364 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d85670b9898e19073533e5ce1bff939c73975db818a0ebbbde74c33f62cb361"
Oct 30 13:54:11.503258 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:54:11.503215    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:54:16.899878 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:16.899834    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:54:22.586096 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:22.586051    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-58w2x"]
Oct 30 13:54:22.586526 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:22.586517    4364 cpu_manager.go:414] "containerMap: removing container" podUID="17a1d683-7bf3-4c03-8223-2ca23778288c" containerName="revoke-keys"
Oct 30 13:54:22.586578 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:22.586530    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17a1d683-7bf3-4c03-8223-2ca23778288c" containerName="revoke-keys"
Oct 30 13:54:22.586637 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:22.586592    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="17a1d683-7bf3-4c03-8223-2ca23778288c" containerName="revoke-keys"
Oct 30 13:54:22.589711 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:22.589693    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-58w2x"
Oct 30 13:54:22.615544 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:22.615502    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvxzs\" (UniqueName: \"kubernetes.io/projected/aba25037-7144-4ba0-9a1a-081243f9445c-kube-api-access-dvxzs\") pod \"maas-api-revoke-sub-d2bc93bc-58w2x\" (UID: \"aba25037-7144-4ba0-9a1a-081243f9445c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-58w2x"
Oct 30 13:54:22.615761 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:22.615656    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aba25037-7144-4ba0-9a1a-081243f9445c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-58w2x\" (UID: \"aba25037-7144-4ba0-9a1a-081243f9445c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-58w2x"
Oct 30 13:54:22.615761 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:22.615687    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aba25037-7144-4ba0-9a1a-081243f9445c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-58w2x\" (UID: \"aba25037-7144-4ba0-9a1a-081243f9445c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-58w2x"
Oct 30 13:54:22.617401 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:22.617369    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:54:22.617578 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:22.617430    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:54:22.617578 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:22.617442    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:54:22.617578 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:22.617369    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\""
Oct 30 13:54:22.619992 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:22.619962    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-58w2x"]
Oct 30 13:54:22.716919 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:22.716877    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aba25037-7144-4ba0-9a1a-081243f9445c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-58w2x\" (UID: \"aba25037-7144-4ba0-9a1a-081243f9445c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-58w2x"
Oct 30 13:54:22.716919 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:22.716922    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aba25037-7144-4ba0-9a1a-081243f9445c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-58w2x\" (UID: \"aba25037-7144-4ba0-9a1a-081243f9445c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-58w2x"
Oct 30 13:54:22.717151 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:22.716966    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvxzs\" (UniqueName: \"kubernetes.io/projected/aba25037-7144-4ba0-9a1a-081243f9445c-kube-api-access-dvxzs\") pod \"maas-api-revoke-sub-d2bc93bc-58w2x\" (UID: \"aba25037-7144-4ba0-9a1a-081243f9445c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-58w2x"
Oct 30 13:54:22.717741 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:22.717721    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aba25037-7144-4ba0-9a1a-081243f9445c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-58w2x\" (UID: \"aba25037-7144-4ba0-9a1a-081243f9445c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-58w2x"
Oct 30 13:54:22.719388 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:22.719362    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aba25037-7144-4ba0-9a1a-081243f9445c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-58w2x\" (UID: \"aba25037-7144-4ba0-9a1a-081243f9445c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-58w2x"
Oct 30 13:54:22.740913 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:22.740882    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvxzs\" (UniqueName: \"kubernetes.io/projected/aba25037-7144-4ba0-9a1a-081243f9445c-kube-api-access-dvxzs\") pod \"maas-api-revoke-sub-d2bc93bc-58w2x\" (UID: \"aba25037-7144-4ba0-9a1a-081243f9445c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-58w2x"
Oct 30 13:54:22.903155 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:22.903118    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-58w2x"
Oct 30 13:54:23.203030 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:23.202898    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-58w2x"]
Oct 30 13:54:23.205770 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:54:23.205739    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaba25037_7144_4ba0_9a1a_081243f9445c.slice/crio-d5735006bd4a36e8c838bd39bddb206b5cdfce3580bc215803334cd42632bd41 WatchSource:0}: Error finding container d5735006bd4a36e8c838bd39bddb206b5cdfce3580bc215803334cd42632bd41: Status 404 returned error can't find the container with id d5735006bd4a36e8c838bd39bddb206b5cdfce3580bc215803334cd42632bd41
Oct 30 13:54:23.268394 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:23.268361    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-58w2x" event={"ID":"aba25037-7144-4ba0-9a1a-081243f9445c","Type":"ContainerStarted","Data":"d5735006bd4a36e8c838bd39bddb206b5cdfce3580bc215803334cd42632bd41"}
Oct 30 13:54:24.275530 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:24.275437    4364 generic.go:356] "Generic (PLEG): container finished" podID="aba25037-7144-4ba0-9a1a-081243f9445c" containerID="17d3ff853f810588d54bd8dc902300eb3feb0b0f97b59b5abef9a536d9f082c0" exitCode=0
Oct 30 13:54:24.275928 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:24.275528    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-58w2x" event={"ID":"aba25037-7144-4ba0-9a1a-081243f9445c","Type":"ContainerDied","Data":"17d3ff853f810588d54bd8dc902300eb3feb0b0f97b59b5abef9a536d9f082c0"}
Oct 30 13:54:25.420127 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:25.420093    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-58w2x"
Oct 30 13:54:25.444880 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:25.444832    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aba25037-7144-4ba0-9a1a-081243f9445c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aba25037-7144-4ba0-9a1a-081243f9445c-maas-api-openshift-ca\") pod \"aba25037-7144-4ba0-9a1a-081243f9445c\" (UID: \"aba25037-7144-4ba0-9a1a-081243f9445c\") "
Oct 30 13:54:25.445055 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:25.444917    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aba25037-7144-4ba0-9a1a-081243f9445c-kube-api-access-dvxzs\" (UniqueName: \"kubernetes.io/projected/aba25037-7144-4ba0-9a1a-081243f9445c-kube-api-access-dvxzs\") pod \"aba25037-7144-4ba0-9a1a-081243f9445c\" (UID: \"aba25037-7144-4ba0-9a1a-081243f9445c\") "
Oct 30 13:54:25.445055 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:25.444947    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aba25037-7144-4ba0-9a1a-081243f9445c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aba25037-7144-4ba0-9a1a-081243f9445c-maas-api-xks-ca\") pod \"aba25037-7144-4ba0-9a1a-081243f9445c\" (UID: \"aba25037-7144-4ba0-9a1a-081243f9445c\") "
Oct 30 13:54:25.445280 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:25.445254    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aba25037-7144-4ba0-9a1a-081243f9445c-maas-api-openshift-ca" pod "aba25037-7144-4ba0-9a1a-081243f9445c" (UID: "aba25037-7144-4ba0-9a1a-081243f9445c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:54:25.447161 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:25.447132    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aba25037-7144-4ba0-9a1a-081243f9445c-maas-api-xks-ca" pod "aba25037-7144-4ba0-9a1a-081243f9445c" (UID: "aba25037-7144-4ba0-9a1a-081243f9445c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:54:25.447258 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:25.447210    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aba25037-7144-4ba0-9a1a-081243f9445c-kube-api-access-dvxzs" pod "aba25037-7144-4ba0-9a1a-081243f9445c" (UID: "aba25037-7144-4ba0-9a1a-081243f9445c"). InnerVolumeSpecName "kube-api-access-dvxzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:54:25.546185 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:25.546098    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aba25037-7144-4ba0-9a1a-081243f9445c-maas-api-openshift-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:54:25.546185 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:25.546129    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvxzs\" (UniqueName: \"kubernetes.io/projected/aba25037-7144-4ba0-9a1a-081243f9445c-kube-api-access-dvxzs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:54:25.546185 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:25.546141    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aba25037-7144-4ba0-9a1a-081243f9445c-maas-api-xks-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:54:26.287018 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:26.286989    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-58w2x"
Oct 30 13:54:26.287018 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:26.287007    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-58w2x" event={"ID":"aba25037-7144-4ba0-9a1a-081243f9445c","Type":"ContainerDied","Data":"d5735006bd4a36e8c838bd39bddb206b5cdfce3580bc215803334cd42632bd41"}
Oct 30 13:54:26.287292 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:26.287062    4364 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5735006bd4a36e8c838bd39bddb206b5cdfce3580bc215803334cd42632bd41"
Oct 30 13:54:34.739845 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:34.739804    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4"]
Oct 30 13:54:34.740346 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:34.740239    4364 cpu_manager.go:414] "containerMap: removing container" podUID="aba25037-7144-4ba0-9a1a-081243f9445c" containerName="revoke-keys"
Oct 30 13:54:34.740346 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:34.740251    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aba25037-7144-4ba0-9a1a-081243f9445c" containerName="revoke-keys"
Oct 30 13:54:34.740346 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:34.740336    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="aba25037-7144-4ba0-9a1a-081243f9445c" containerName="revoke-keys"
Oct 30 13:54:34.743663 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:34.743642    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4"
Oct 30 13:54:34.762088 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:34.762059    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-d61c7a-openshift-default-dockercfg-ntmm9\""
Oct 30 13:54:34.776521 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:34.776490    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4"]
Oct 30 13:54:34.837037 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:34.837000    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g44c\" (UniqueName: \"kubernetes.io/projected/03a318c2-ba1f-4c34-a2f4-8b258f71eee3-kube-api-access-2g44c\") pod \"e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4\" (UID: \"03a318c2-ba1f-4c34-a2f4-8b258f71eee3\") " pod="openshift-ingress/e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4"
Oct 30 13:54:34.837240 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:34.837063    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03a318c2-ba1f-4c34-a2f4-8b258f71eee3-istio-podinfo\") pod \"e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4\" (UID: \"03a318c2-ba1f-4c34-a2f4-8b258f71eee3\") " pod="openshift-ingress/e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4"
Oct 30 13:54:34.837240 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:34.837162    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03a318c2-ba1f-4c34-a2f4-8b258f71eee3-istiod-ca-cert\") pod \"e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4\" (UID: \"03a318c2-ba1f-4c34-a2f4-8b258f71eee3\") " pod="openshift-ingress/e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4"
Oct 30 13:54:34.837240 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:34.837196    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a318c2-ba1f-4c34-a2f4-8b258f71eee3-workload-socket\") pod \"e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4\" (UID: \"03a318c2-ba1f-4c34-a2f4-8b258f71eee3\") " pod="openshift-ingress/e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4"
Oct 30 13:54:34.837240 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:34.837214    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03a318c2-ba1f-4c34-a2f4-8b258f71eee3-istio-envoy\") pod \"e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4\" (UID: \"03a318c2-ba1f-4c34-a2f4-8b258f71eee3\") " pod="openshift-ingress/e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4"
Oct 30 13:54:34.837392 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:34.837363    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03a318c2-ba1f-4c34-a2f4-8b258f71eee3-istio-token\") pod \"e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4\" (UID: \"03a318c2-ba1f-4c34-a2f4-8b258f71eee3\") " pod="openshift-ingress/e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4"
Oct 30 13:54:34.837487 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:34.837471    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a318c2-ba1f-4c34-a2f4-8b258f71eee3-credential-socket\") pod \"e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4\" (UID: \"03a318c2-ba1f-4c34-a2f4-8b258f71eee3\") " pod="openshift-ingress/e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4"
Oct 30 13:54:34.837549 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:34.837500    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03a318c2-ba1f-4c34-a2f4-8b258f71eee3-istio-data\") pod \"e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4\" (UID: \"03a318c2-ba1f-4c34-a2f4-8b258f71eee3\") " pod="openshift-ingress/e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4"
Oct 30 13:54:34.837590 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:34.837561    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03a318c2-ba1f-4c34-a2f4-8b258f71eee3-workload-certs\") pod \"e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4\" (UID: \"03a318c2-ba1f-4c34-a2f4-8b258f71eee3\") " pod="openshift-ingress/e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4"
Oct 30 13:54:34.938530 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:34.938488    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03a318c2-ba1f-4c34-a2f4-8b258f71eee3-istio-podinfo\") pod \"e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4\" (UID: \"03a318c2-ba1f-4c34-a2f4-8b258f71eee3\") " pod="openshift-ingress/e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4"
Oct 30 13:54:34.938766 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:34.938538    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03a318c2-ba1f-4c34-a2f4-8b258f71eee3-istiod-ca-cert\") pod \"e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4\" (UID: \"03a318c2-ba1f-4c34-a2f4-8b258f71eee3\") " pod="openshift-ingress/e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4"
Oct 30 13:54:34.938766 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:34.938560    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a318c2-ba1f-4c34-a2f4-8b258f71eee3-workload-socket\") pod \"e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4\" (UID: \"03a318c2-ba1f-4c34-a2f4-8b258f71eee3\") " pod="openshift-ingress/e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4"
Oct 30 13:54:34.938766 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:34.938574    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03a318c2-ba1f-4c34-a2f4-8b258f71eee3-istio-envoy\") pod \"e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4\" (UID: \"03a318c2-ba1f-4c34-a2f4-8b258f71eee3\") " pod="openshift-ingress/e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4"
Oct 30 13:54:34.938766 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:34.938630    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03a318c2-ba1f-4c34-a2f4-8b258f71eee3-istio-token\") pod \"e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4\" (UID: \"03a318c2-ba1f-4c34-a2f4-8b258f71eee3\") " pod="openshift-ingress/e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4"
Oct 30 13:54:34.938766 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:34.938699    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a318c2-ba1f-4c34-a2f4-8b258f71eee3-credential-socket\") pod \"e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4\" (UID: \"03a318c2-ba1f-4c34-a2f4-8b258f71eee3\") " pod="openshift-ingress/e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4"
Oct 30 13:54:34.938766 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:34.938726    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03a318c2-ba1f-4c34-a2f4-8b258f71eee3-istio-data\") pod \"e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4\" (UID: \"03a318c2-ba1f-4c34-a2f4-8b258f71eee3\") " pod="openshift-ingress/e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4"
Oct 30 13:54:34.938766 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:34.938767    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03a318c2-ba1f-4c34-a2f4-8b258f71eee3-workload-certs\") pod \"e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4\" (UID: \"03a318c2-ba1f-4c34-a2f4-8b258f71eee3\") " pod="openshift-ingress/e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4"
Oct 30 13:54:34.939169 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:34.938836    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g44c\" (UniqueName: \"kubernetes.io/projected/03a318c2-ba1f-4c34-a2f4-8b258f71eee3-kube-api-access-2g44c\") pod \"e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4\" (UID: \"03a318c2-ba1f-4c34-a2f4-8b258f71eee3\") " pod="openshift-ingress/e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4"
Oct 30 13:54:34.939169 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:34.939099    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a318c2-ba1f-4c34-a2f4-8b258f71eee3-workload-socket\") pod \"e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4\" (UID: \"03a318c2-ba1f-4c34-a2f4-8b258f71eee3\") " pod="openshift-ingress/e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4"
Oct 30 13:54:34.939169 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:34.939126    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03a318c2-ba1f-4c34-a2f4-8b258f71eee3-istio-data\") pod \"e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4\" (UID: \"03a318c2-ba1f-4c34-a2f4-8b258f71eee3\") " pod="openshift-ingress/e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4"
Oct 30 13:54:34.939281 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:34.939246    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03a318c2-ba1f-4c34-a2f4-8b258f71eee3-workload-certs\") pod \"e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4\" (UID: \"03a318c2-ba1f-4c34-a2f4-8b258f71eee3\") " pod="openshift-ingress/e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4"
Oct 30 13:54:34.939446 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:34.939421    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03a318c2-ba1f-4c34-a2f4-8b258f71eee3-istiod-ca-cert\") pod \"e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4\" (UID: \"03a318c2-ba1f-4c34-a2f4-8b258f71eee3\") " pod="openshift-ingress/e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4"
Oct 30 13:54:34.939501 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:34.939424    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a318c2-ba1f-4c34-a2f4-8b258f71eee3-credential-socket\") pod \"e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4\" (UID: \"03a318c2-ba1f-4c34-a2f4-8b258f71eee3\") " pod="openshift-ingress/e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4"
Oct 30 13:54:34.941667 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:34.941633    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03a318c2-ba1f-4c34-a2f4-8b258f71eee3-istio-envoy\") pod \"e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4\" (UID: \"03a318c2-ba1f-4c34-a2f4-8b258f71eee3\") " pod="openshift-ingress/e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4"
Oct 30 13:54:34.941790 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:34.941770    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03a318c2-ba1f-4c34-a2f4-8b258f71eee3-istio-podinfo\") pod \"e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4\" (UID: \"03a318c2-ba1f-4c34-a2f4-8b258f71eee3\") " pod="openshift-ingress/e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4"
Oct 30 13:54:34.962848 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:34.962822    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03a318c2-ba1f-4c34-a2f4-8b258f71eee3-istio-token\") pod \"e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4\" (UID: \"03a318c2-ba1f-4c34-a2f4-8b258f71eee3\") " pod="openshift-ingress/e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4"
Oct 30 13:54:34.963062 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:34.963038    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g44c\" (UniqueName: \"kubernetes.io/projected/03a318c2-ba1f-4c34-a2f4-8b258f71eee3-kube-api-access-2g44c\") pod \"e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4\" (UID: \"03a318c2-ba1f-4c34-a2f4-8b258f71eee3\") " pod="openshift-ingress/e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4"
Oct 30 13:54:35.065105 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:35.064976    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4"
Oct 30 13:54:35.393951 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:35.393922    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4"]
Oct 30 13:54:35.396067 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:54:35.396032    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03a318c2_ba1f_4c34_a2f4_8b258f71eee3.slice/crio-3f24a340e942e12a4066962c7d6ff2cc9bee324d36f29054da08bf8de805686d WatchSource:0}: Error finding container 3f24a340e942e12a4066962c7d6ff2cc9bee324d36f29054da08bf8de805686d: Status 404 returned error can't find the container with id 3f24a340e942e12a4066962c7d6ff2cc9bee324d36f29054da08bf8de805686d
Oct 30 13:54:35.398480 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:35.398445    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:54:35.398571 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:35.398514    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:54:35.398571 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:35.398543    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:54:36.347335 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:36.347296    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4" event={"ID":"03a318c2-ba1f-4c34-a2f4-8b258f71eee3","Type":"ContainerStarted","Data":"0772dfd437e8f077a7036aa72ba2520bee2255ff3f8c4cf92f6921e5a37a9b7e"}
Oct 30 13:54:36.347792 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:36.347354    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4" event={"ID":"03a318c2-ba1f-4c34-a2f4-8b258f71eee3","Type":"ContainerStarted","Data":"3f24a340e942e12a4066962c7d6ff2cc9bee324d36f29054da08bf8de805686d"}
Oct 30 13:54:36.407493 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:36.407435    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4" podStartSLOduration=2.407421386 podStartE2EDuration="2.407421386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:54:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:54:36.406912827 +0000 UTC m=+3775.573277475" watchObservedRunningTime="2026-10-30 13:54:36.407421386 +0000 UTC m=+3775.573786034"
Oct 30 13:54:37.065801 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:37.065765    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4"
Oct 30 13:54:37.070775 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:37.070747    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4"
Oct 30 13:54:37.353312 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:37.353222    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4"
Oct 30 13:54:37.354326 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:37.354305    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4"
Oct 30 13:54:40.186063 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:40.186024    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d61c7a-66cb874f59-qftdb"]
Oct 30 13:54:40.190086 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:40.190062    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d61c7a-66cb874f59-qftdb"
Oct 30 13:54:40.206494 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:40.206460    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 13:54:40.208540 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:40.208496    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-d61c7a-serving-cert\""
Oct 30 13:54:40.220460 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:40.219062    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-zw9vx\""
Oct 30 13:54:40.220460 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:40.219864    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:54:40.220460 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:40.219965    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:54:40.224618 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:40.224574    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d61c7a-66cb874f59-qftdb"]
Oct 30 13:54:40.294777 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:40.294740    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12b8f29c-158a-4493-9296-2ccd04a10846-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-d61c7a-66cb874f59-qftdb\" (UID: \"12b8f29c-158a-4493-9296-2ccd04a10846\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d61c7a-66cb874f59-qftdb"
Oct 30 13:54:40.294992 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:40.294836    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12b8f29c-158a-4493-9296-2ccd04a10846-metrics-certs\") pod \"maas-api-e2e-worker-single-main-d61c7a-66cb874f59-qftdb\" (UID: \"12b8f29c-158a-4493-9296-2ccd04a10846\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d61c7a-66cb874f59-qftdb"
Oct 30 13:54:40.294992 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:40.294921    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4snm4\" (UniqueName: \"kubernetes.io/projected/12b8f29c-158a-4493-9296-2ccd04a10846-kube-api-access-4snm4\") pod \"maas-api-e2e-worker-single-main-d61c7a-66cb874f59-qftdb\" (UID: \"12b8f29c-158a-4493-9296-2ccd04a10846\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d61c7a-66cb874f59-qftdb"
Oct 30 13:54:40.295094 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:40.295051    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12b8f29c-158a-4493-9296-2ccd04a10846-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-d61c7a-66cb874f59-qftdb\" (UID: \"12b8f29c-158a-4493-9296-2ccd04a10846\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d61c7a-66cb874f59-qftdb"
Oct 30 13:54:40.396565 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:40.396527    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12b8f29c-158a-4493-9296-2ccd04a10846-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-d61c7a-66cb874f59-qftdb\" (UID: \"12b8f29c-158a-4493-9296-2ccd04a10846\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d61c7a-66cb874f59-qftdb"
Oct 30 13:54:40.396794 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:40.396573    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12b8f29c-158a-4493-9296-2ccd04a10846-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-d61c7a-66cb874f59-qftdb\" (UID: \"12b8f29c-158a-4493-9296-2ccd04a10846\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d61c7a-66cb874f59-qftdb"
Oct 30 13:54:40.396794 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:40.396662    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12b8f29c-158a-4493-9296-2ccd04a10846-metrics-certs\") pod \"maas-api-e2e-worker-single-main-d61c7a-66cb874f59-qftdb\" (UID: \"12b8f29c-158a-4493-9296-2ccd04a10846\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d61c7a-66cb874f59-qftdb"
Oct 30 13:54:40.396794 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:40.396695    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4snm4\" (UniqueName: \"kubernetes.io/projected/12b8f29c-158a-4493-9296-2ccd04a10846-kube-api-access-4snm4\") pod \"maas-api-e2e-worker-single-main-d61c7a-66cb874f59-qftdb\" (UID: \"12b8f29c-158a-4493-9296-2ccd04a10846\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d61c7a-66cb874f59-qftdb"
Oct 30 13:54:40.397438 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:40.397408    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12b8f29c-158a-4493-9296-2ccd04a10846-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-d61c7a-66cb874f59-qftdb\" (UID: \"12b8f29c-158a-4493-9296-2ccd04a10846\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d61c7a-66cb874f59-qftdb"
Oct 30 13:54:40.399292 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:40.399256    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12b8f29c-158a-4493-9296-2ccd04a10846-metrics-certs\") pod \"maas-api-e2e-worker-single-main-d61c7a-66cb874f59-qftdb\" (UID: \"12b8f29c-158a-4493-9296-2ccd04a10846\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d61c7a-66cb874f59-qftdb"
Oct 30 13:54:40.399401 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:40.399296    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12b8f29c-158a-4493-9296-2ccd04a10846-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-d61c7a-66cb874f59-qftdb\" (UID: \"12b8f29c-158a-4493-9296-2ccd04a10846\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d61c7a-66cb874f59-qftdb"
Oct 30 13:54:40.421054 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:40.421021    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4snm4\" (UniqueName: \"kubernetes.io/projected/12b8f29c-158a-4493-9296-2ccd04a10846-kube-api-access-4snm4\") pod \"maas-api-e2e-worker-single-main-d61c7a-66cb874f59-qftdb\" (UID: \"12b8f29c-158a-4493-9296-2ccd04a10846\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d61c7a-66cb874f59-qftdb"
Oct 30 13:54:40.504228 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:40.504137    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d61c7a-66cb874f59-qftdb"
Oct 30 13:54:40.597010 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:40.596967    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-d61c7a-66b6589849tbc9"]
Oct 30 13:54:40.602525 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:40.602496    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d61c7a-66b6589849tbc9"
Oct 30 13:54:40.616578 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:40.616545    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-d61c7a-66b6589849tbc9"]
Oct 30 13:54:40.619448 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:40.619420    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-d61c7a\""
Oct 30 13:54:40.619936 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:40.619911    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-b53875af-dockercfg-qt62l\""
Oct 30 13:54:40.651706 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:40.651671    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d61c7a-78657kht4h"]
Oct 30 13:54:40.655344 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:40.655319    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d61c7a-78657kht4h"
Oct 30 13:54:40.672200 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:40.672170    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d61c7a-78657kht4h"]
Oct 30 13:54:40.699039 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:40.698978    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e345a8d5-e10a-44ee-824b-9966bac27061-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-d61c7a-66b6589849tbc9\" (UID: \"e345a8d5-e10a-44ee-824b-9966bac27061\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-d61c7a-66b6589849tbc9"
Oct 30 13:54:40.699369 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:40.699336    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s876x\" (UniqueName: \"kubernetes.io/projected/e345a8d5-e10a-44ee-824b-9966bac27061-kube-api-access-s876x\") pod \"payload-processing-e2e-worker-single-main-d61c7a-66b6589849tbc9\" (UID: \"e345a8d5-e10a-44ee-824b-9966bac27061\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-d61c7a-66b6589849tbc9"
Oct 30 13:54:40.801042 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:40.800949    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxs6q\" (UniqueName: \"kubernetes.io/projected/d7df9595-a3d4-4793-a26d-403deda052dd-kube-api-access-zxs6q\") pod \"payload-pre-processing-e2e-worker-single-main-d61c7a-78657kht4h\" (UID: \"d7df9595-a3d4-4793-a26d-403deda052dd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d61c7a-78657kht4h"
Oct 30 13:54:40.801042 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:40.801027    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s876x\" (UniqueName: \"kubernetes.io/projected/e345a8d5-e10a-44ee-824b-9966bac27061-kube-api-access-s876x\") pod \"payload-processing-e2e-worker-single-main-d61c7a-66b6589849tbc9\" (UID: \"e345a8d5-e10a-44ee-824b-9966bac27061\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-d61c7a-66b6589849tbc9"
Oct 30 13:54:40.801265 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:40.801111    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e345a8d5-e10a-44ee-824b-9966bac27061-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-d61c7a-66b6589849tbc9\" (UID: \"e345a8d5-e10a-44ee-824b-9966bac27061\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-d61c7a-66b6589849tbc9"
Oct 30 13:54:40.801265 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:40.801157    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7df9595-a3d4-4793-a26d-403deda052dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-d61c7a-78657kht4h\" (UID: \"d7df9595-a3d4-4793-a26d-403deda052dd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d61c7a-78657kht4h"
Oct 30 13:54:40.801942 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:40.801921    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e345a8d5-e10a-44ee-824b-9966bac27061-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-d61c7a-66b6589849tbc9\" (UID: \"e345a8d5-e10a-44ee-824b-9966bac27061\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-d61c7a-66b6589849tbc9"
Oct 30 13:54:40.820525 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:40.820504    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d61c7a-66cb874f59-qftdb"]
Oct 30 13:54:40.822244 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:54:40.822203    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12b8f29c_158a_4493_9296_2ccd04a10846.slice/crio-140f389f705e0a1bf834926a8633f9f8ddce12a0ba8618e65ba1769f2f2b7d8e WatchSource:0}: Error finding container 140f389f705e0a1bf834926a8633f9f8ddce12a0ba8618e65ba1769f2f2b7d8e: Status 404 returned error can't find the container with id 140f389f705e0a1bf834926a8633f9f8ddce12a0ba8618e65ba1769f2f2b7d8e
Oct 30 13:54:40.825447 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:40.825424    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s876x\" (UniqueName: \"kubernetes.io/projected/e345a8d5-e10a-44ee-824b-9966bac27061-kube-api-access-s876x\") pod \"payload-processing-e2e-worker-single-main-d61c7a-66b6589849tbc9\" (UID: \"e345a8d5-e10a-44ee-824b-9966bac27061\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-d61c7a-66b6589849tbc9"
Oct 30 13:54:40.902249 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:40.902206    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxs6q\" (UniqueName: \"kubernetes.io/projected/d7df9595-a3d4-4793-a26d-403deda052dd-kube-api-access-zxs6q\") pod \"payload-pre-processing-e2e-worker-single-main-d61c7a-78657kht4h\" (UID: \"d7df9595-a3d4-4793-a26d-403deda052dd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d61c7a-78657kht4h"
Oct 30 13:54:40.902461 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:40.902307    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7df9595-a3d4-4793-a26d-403deda052dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-d61c7a-78657kht4h\" (UID: \"d7df9595-a3d4-4793-a26d-403deda052dd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d61c7a-78657kht4h"
Oct 30 13:54:40.903102 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:40.903083    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7df9595-a3d4-4793-a26d-403deda052dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-d61c7a-78657kht4h\" (UID: \"d7df9595-a3d4-4793-a26d-403deda052dd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d61c7a-78657kht4h"
Oct 30 13:54:40.917251 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:40.917213    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d61c7a-66b6589849tbc9"
Oct 30 13:54:40.925743 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:40.925716    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxs6q\" (UniqueName: \"kubernetes.io/projected/d7df9595-a3d4-4793-a26d-403deda052dd-kube-api-access-zxs6q\") pod \"payload-pre-processing-e2e-worker-single-main-d61c7a-78657kht4h\" (UID: \"d7df9595-a3d4-4793-a26d-403deda052dd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d61c7a-78657kht4h"
Oct 30 13:54:40.969866 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:40.968920    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d61c7a-78657kht4h"
Oct 30 13:54:41.239147 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:41.239010    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-d61c7a-66b6589849tbc9"]
Oct 30 13:54:41.242144 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:54:41.242114    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode345a8d5_e10a_44ee_824b_9966bac27061.slice/crio-89c4c65f0965986fd4a9ddca1125e07c772159ebb1fadbba555b651d8556367f WatchSource:0}: Error finding container 89c4c65f0965986fd4a9ddca1125e07c772159ebb1fadbba555b651d8556367f: Status 404 returned error can't find the container with id 89c4c65f0965986fd4a9ddca1125e07c772159ebb1fadbba555b651d8556367f
Oct 30 13:54:41.291710 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:41.291681    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d61c7a-78657kht4h"]
Oct 30 13:54:41.294364 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:54:41.294326    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7df9595_a3d4_4793_a26d_403deda052dd.slice/crio-156aee45236682c1a5500c72bbd4f069c8ed9c4a45f4bb196fe91e127b704475 WatchSource:0}: Error finding container 156aee45236682c1a5500c72bbd4f069c8ed9c4a45f4bb196fe91e127b704475: Status 404 returned error can't find the container with id 156aee45236682c1a5500c72bbd4f069c8ed9c4a45f4bb196fe91e127b704475
Oct 30 13:54:41.377902 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:41.377862    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d61c7a-78657kht4h" event={"ID":"d7df9595-a3d4-4793-a26d-403deda052dd","Type":"ContainerStarted","Data":"156aee45236682c1a5500c72bbd4f069c8ed9c4a45f4bb196fe91e127b704475"}
Oct 30 13:54:41.380086 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:41.380053    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d61c7a-66cb874f59-qftdb" event={"ID":"12b8f29c-158a-4493-9296-2ccd04a10846","Type":"ContainerStarted","Data":"9dc4f33e039bb953b0fca65db5fb49763d24b74bd6a417e36c9fa28f4806aeb5"}
Oct 30 13:54:41.380213 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:41.380112    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d61c7a-66cb874f59-qftdb" event={"ID":"12b8f29c-158a-4493-9296-2ccd04a10846","Type":"ContainerStarted","Data":"140f389f705e0a1bf834926a8633f9f8ddce12a0ba8618e65ba1769f2f2b7d8e"}
Oct 30 13:54:41.380277 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:41.380262    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d61c7a-66cb874f59-qftdb"
Oct 30 13:54:41.381749 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:41.381724    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d61c7a-66b6589849tbc9" event={"ID":"e345a8d5-e10a-44ee-824b-9966bac27061","Type":"ContainerStarted","Data":"2e58969a381fd50f2f4f723ae4b714ae5c739d2d7a47842bdc4762dd4a3ea7ec"}
Oct 30 13:54:41.381855 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:41.381773    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d61c7a-66b6589849tbc9" event={"ID":"e345a8d5-e10a-44ee-824b-9966bac27061","Type":"ContainerStarted","Data":"89c4c65f0965986fd4a9ddca1125e07c772159ebb1fadbba555b651d8556367f"}
Oct 30 13:54:41.381855 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:41.381805    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d61c7a-66b6589849tbc9"
Oct 30 13:54:41.424487 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:41.424388    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d61c7a-66cb874f59-qftdb" podStartSLOduration=1.113214871 podStartE2EDuration="1.42437204s" totalImagesPullingTime="311.157169ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:54:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:54:41.422360457 +0000 UTC m=+3780.588725104" watchObservedRunningTime="2026-10-30 13:54:41.42437204 +0000 UTC m=+3780.590736688"
Oct 30 13:54:41.473501 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:41.473444    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d61c7a-66b6589849tbc9" podStartSLOduration=1.473428958 podStartE2EDuration="1.473428958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:54:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:54:41.471617178 +0000 UTC m=+3780.637981816" watchObservedRunningTime="2026-10-30 13:54:41.473428958 +0000 UTC m=+3780.639793602"
Oct 30 13:54:41.504927 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:54:41.504890    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:54:42.389851 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:42.389817    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d61c7a-78657kht4h" event={"ID":"d7df9595-a3d4-4793-a26d-403deda052dd","Type":"ContainerStarted","Data":"5028c5db2b3a58fc67e4dd46314ffeff8ee0bc98a0d163158f3bdbfa3a2c3b0c"}
Oct 30 13:54:42.390309 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:42.389894    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d61c7a-78657kht4h"
Oct 30 13:54:42.431886 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:42.431838    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d61c7a-78657kht4h" podStartSLOduration=2.431824742 podStartE2EDuration="2.431824742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:54:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:54:42.430707234 +0000 UTC m=+3781.597071884" watchObservedRunningTime="2026-10-30 13:54:42.431824742 +0000 UTC m=+3781.598189390"
Oct 30 13:54:43.883470 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:43.883432    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:54:47.407763 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:47.407731    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d61c7a-66cb874f59-qftdb"
Oct 30 13:54:52.393782 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:52.393740    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d61c7a-66b6589849tbc9"
Oct 30 13:54:53.395844 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:53.395798    4364 patch_prober.go:29] interesting pod/e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.25:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:54:53.396319 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:53.395868    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4" podUID="03a318c2-ba1f-4c34-a2f4-8b258f71eee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.25:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:54:53.399914 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:53.399890    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d61c7a-78657kht4h"
Oct 30 13:54:54.462917 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:54.462884    4364 generic.go:356] "Generic (PLEG): container finished" podID="03a318c2-ba1f-4c34-a2f4-8b258f71eee3" containerID="0772dfd437e8f077a7036aa72ba2520bee2255ff3f8c4cf92f6921e5a37a9b7e" exitCode=0
Oct 30 13:54:54.463370 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:54.462961    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4" event={"ID":"03a318c2-ba1f-4c34-a2f4-8b258f71eee3","Type":"ContainerDied","Data":"0772dfd437e8f077a7036aa72ba2520bee2255ff3f8c4cf92f6921e5a37a9b7e"}
Oct 30 13:54:54.463370 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:54.463359    4364 scope.go:122] "RemoveContainer" containerID="0772dfd437e8f077a7036aa72ba2520bee2255ff3f8c4cf92f6921e5a37a9b7e"
Oct 30 13:54:54.464700 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:54.464672    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:54:54.464769 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:54.464741    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:54:54.464810 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:54.464773    4364 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 13:54:55.472654 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:55.472591    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4" event={"ID":"03a318c2-ba1f-4c34-a2f4-8b258f71eee3","Type":"ContainerStarted","Data":"e24c02f9e4442ae3a58611ea7aa34787b39f8f15ba027924ac217d0476639626"}
Oct 30 13:54:55.928970 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:55.928931    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-d61c7a/simulator-single-main-d61c7a-kserve-975b88bd-w8cmx"]
Oct 30 13:54:55.931664 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:55.931634    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-d61c7a/simulator-single-main-d61c7a-kserve-975b88bd-w8cmx"
Oct 30 13:54:55.954178 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:55.954140    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-d61c7a\"/\"kube-root-ca.crt\""
Oct 30 13:54:55.954178 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:55.954174    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-d61c7a\"/\"simulator-single-main-d61c7a-kserve-self-signed-certs\""
Oct 30 13:54:55.964917 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:55.964885    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-d61c7a\"/\"openshift-service-ca.crt\""
Oct 30 13:54:55.964917 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:55.964894    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-d61c7a\"/\"default-dockercfg-hv7sq\""
Oct 30 13:54:55.967937 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:55.967910    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-d61c7a/simulator-single-main-d61c7a-kserve-975b88bd-w8cmx"]
Oct 30 13:54:56.006755 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:56.006711    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/340a6502-4312-4a36-883c-a9c32e7d0c99-tmp-dir\") pod \"simulator-single-main-d61c7a-kserve-975b88bd-w8cmx\" (UID: \"340a6502-4312-4a36-883c-a9c32e7d0c99\") " pod="e2e-models-e2e-worker-single-main-d61c7a/simulator-single-main-d61c7a-kserve-975b88bd-w8cmx"
Oct 30 13:54:56.007125 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:56.007095    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/340a6502-4312-4a36-883c-a9c32e7d0c99-tls-certs\") pod \"simulator-single-main-d61c7a-kserve-975b88bd-w8cmx\" (UID: \"340a6502-4312-4a36-883c-a9c32e7d0c99\") " pod="e2e-models-e2e-worker-single-main-d61c7a/simulator-single-main-d61c7a-kserve-975b88bd-w8cmx"
Oct 30 13:54:56.007340 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:56.007161    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/340a6502-4312-4a36-883c-a9c32e7d0c99-dshm\") pod \"simulator-single-main-d61c7a-kserve-975b88bd-w8cmx\" (UID: \"340a6502-4312-4a36-883c-a9c32e7d0c99\") " pod="e2e-models-e2e-worker-single-main-d61c7a/simulator-single-main-d61c7a-kserve-975b88bd-w8cmx"
Oct 30 13:54:56.007340 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:56.007185    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/340a6502-4312-4a36-883c-a9c32e7d0c99-model-cache\") pod \"simulator-single-main-d61c7a-kserve-975b88bd-w8cmx\" (UID: \"340a6502-4312-4a36-883c-a9c32e7d0c99\") " pod="e2e-models-e2e-worker-single-main-d61c7a/simulator-single-main-d61c7a-kserve-975b88bd-w8cmx"
Oct 30 13:54:56.007340 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:56.007247    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/340a6502-4312-4a36-883c-a9c32e7d0c99-home\") pod \"simulator-single-main-d61c7a-kserve-975b88bd-w8cmx\" (UID: \"340a6502-4312-4a36-883c-a9c32e7d0c99\") " pod="e2e-models-e2e-worker-single-main-d61c7a/simulator-single-main-d61c7a-kserve-975b88bd-w8cmx"
Oct 30 13:54:56.007340 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:56.007309    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4bsl\" (UniqueName: \"kubernetes.io/projected/340a6502-4312-4a36-883c-a9c32e7d0c99-kube-api-access-q4bsl\") pod \"simulator-single-main-d61c7a-kserve-975b88bd-w8cmx\" (UID: \"340a6502-4312-4a36-883c-a9c32e7d0c99\") " pod="e2e-models-e2e-worker-single-main-d61c7a/simulator-single-main-d61c7a-kserve-975b88bd-w8cmx"
Oct 30 13:54:56.065838 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:56.065804    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4"
Oct 30 13:54:56.066163 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:56.066131    4364 patch_prober.go:29] interesting pod/e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused" start-of-body=
Oct 30 13:54:56.066311 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:56.066183    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4" podUID="03a318c2-ba1f-4c34-a2f4-8b258f71eee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused"
Oct 30 13:54:56.108441 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:56.108401    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/340a6502-4312-4a36-883c-a9c32e7d0c99-tmp-dir\") pod \"simulator-single-main-d61c7a-kserve-975b88bd-w8cmx\" (UID: \"340a6502-4312-4a36-883c-a9c32e7d0c99\") " pod="e2e-models-e2e-worker-single-main-d61c7a/simulator-single-main-d61c7a-kserve-975b88bd-w8cmx"
Oct 30 13:54:56.108667 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:56.108506    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/340a6502-4312-4a36-883c-a9c32e7d0c99-tls-certs\") pod \"simulator-single-main-d61c7a-kserve-975b88bd-w8cmx\" (UID: \"340a6502-4312-4a36-883c-a9c32e7d0c99\") " pod="e2e-models-e2e-worker-single-main-d61c7a/simulator-single-main-d61c7a-kserve-975b88bd-w8cmx"
Oct 30 13:54:56.108667 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:56.108550    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/340a6502-4312-4a36-883c-a9c32e7d0c99-dshm\") pod \"simulator-single-main-d61c7a-kserve-975b88bd-w8cmx\" (UID: \"340a6502-4312-4a36-883c-a9c32e7d0c99\") " pod="e2e-models-e2e-worker-single-main-d61c7a/simulator-single-main-d61c7a-kserve-975b88bd-w8cmx"
Oct 30 13:54:56.108667 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:56.108574    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/340a6502-4312-4a36-883c-a9c32e7d0c99-model-cache\") pod \"simulator-single-main-d61c7a-kserve-975b88bd-w8cmx\" (UID: \"340a6502-4312-4a36-883c-a9c32e7d0c99\") " pod="e2e-models-e2e-worker-single-main-d61c7a/simulator-single-main-d61c7a-kserve-975b88bd-w8cmx"
Oct 30 13:54:56.108797 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:56.108665    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/340a6502-4312-4a36-883c-a9c32e7d0c99-home\") pod \"simulator-single-main-d61c7a-kserve-975b88bd-w8cmx\" (UID: \"340a6502-4312-4a36-883c-a9c32e7d0c99\") " pod="e2e-models-e2e-worker-single-main-d61c7a/simulator-single-main-d61c7a-kserve-975b88bd-w8cmx"
Oct 30 13:54:56.108797 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:56.108724    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4bsl\" (UniqueName: \"kubernetes.io/projected/340a6502-4312-4a36-883c-a9c32e7d0c99-kube-api-access-q4bsl\") pod \"simulator-single-main-d61c7a-kserve-975b88bd-w8cmx\" (UID: \"340a6502-4312-4a36-883c-a9c32e7d0c99\") " pod="e2e-models-e2e-worker-single-main-d61c7a/simulator-single-main-d61c7a-kserve-975b88bd-w8cmx"
Oct 30 13:54:56.108951 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:56.108929    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/340a6502-4312-4a36-883c-a9c32e7d0c99-tmp-dir\") pod \"simulator-single-main-d61c7a-kserve-975b88bd-w8cmx\" (UID: \"340a6502-4312-4a36-883c-a9c32e7d0c99\") " pod="e2e-models-e2e-worker-single-main-d61c7a/simulator-single-main-d61c7a-kserve-975b88bd-w8cmx"
Oct 30 13:54:56.109323 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:56.109287    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/340a6502-4312-4a36-883c-a9c32e7d0c99-home\") pod \"simulator-single-main-d61c7a-kserve-975b88bd-w8cmx\" (UID: \"340a6502-4312-4a36-883c-a9c32e7d0c99\") " pod="e2e-models-e2e-worker-single-main-d61c7a/simulator-single-main-d61c7a-kserve-975b88bd-w8cmx"
Oct 30 13:54:56.109463 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:56.109371    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/340a6502-4312-4a36-883c-a9c32e7d0c99-model-cache\") pod \"simulator-single-main-d61c7a-kserve-975b88bd-w8cmx\" (UID: \"340a6502-4312-4a36-883c-a9c32e7d0c99\") " pod="e2e-models-e2e-worker-single-main-d61c7a/simulator-single-main-d61c7a-kserve-975b88bd-w8cmx"
Oct 30 13:54:56.111200 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:56.111180    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/340a6502-4312-4a36-883c-a9c32e7d0c99-dshm\") pod \"simulator-single-main-d61c7a-kserve-975b88bd-w8cmx\" (UID: \"340a6502-4312-4a36-883c-a9c32e7d0c99\") " pod="e2e-models-e2e-worker-single-main-d61c7a/simulator-single-main-d61c7a-kserve-975b88bd-w8cmx"
Oct 30 13:54:56.111577 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:56.111561    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/340a6502-4312-4a36-883c-a9c32e7d0c99-tls-certs\") pod \"simulator-single-main-d61c7a-kserve-975b88bd-w8cmx\" (UID: \"340a6502-4312-4a36-883c-a9c32e7d0c99\") " pod="e2e-models-e2e-worker-single-main-d61c7a/simulator-single-main-d61c7a-kserve-975b88bd-w8cmx"
Oct 30 13:54:56.134868 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:56.134835    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4bsl\" (UniqueName: \"kubernetes.io/projected/340a6502-4312-4a36-883c-a9c32e7d0c99-kube-api-access-q4bsl\") pod \"simulator-single-main-d61c7a-kserve-975b88bd-w8cmx\" (UID: \"340a6502-4312-4a36-883c-a9c32e7d0c99\") " pod="e2e-models-e2e-worker-single-main-d61c7a/simulator-single-main-d61c7a-kserve-975b88bd-w8cmx"
Oct 30 13:54:56.247090 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:56.246992    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-d61c7a/simulator-single-main-d61c7a-kserve-975b88bd-w8cmx"
Oct 30 13:54:56.605528 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:56.605427    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-d61c7a/simulator-single-main-d61c7a-kserve-975b88bd-w8cmx"]
Oct 30 13:54:56.609166 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:54:56.609135    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod340a6502_4312_4a36_883c_a9c32e7d0c99.slice/crio-8ef6bed14190e1f1633992b8df12772be686897adee77cf3e307341d8ca889bd WatchSource:0}: Error finding container 8ef6bed14190e1f1633992b8df12772be686897adee77cf3e307341d8ca889bd: Status 404 returned error can't find the container with id 8ef6bed14190e1f1633992b8df12772be686897adee77cf3e307341d8ca889bd
Oct 30 13:54:57.065731 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:57.065693    4364 patch_prober.go:29] interesting pod/e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused" start-of-body=
Oct 30 13:54:57.065941 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:57.065752    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4" podUID="03a318c2-ba1f-4c34-a2f4-8b258f71eee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused"
Oct 30 13:54:57.507058 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:57.507011    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-d61c7a/simulator-single-main-d61c7a-kserve-975b88bd-w8cmx" event={"ID":"340a6502-4312-4a36-883c-a9c32e7d0c99","Type":"ContainerStarted","Data":"5f8683cc67b4e3cb9399977dced85a26082a014bdd5dffa447fdeb95735cbba0"}
Oct 30 13:54:57.507287 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:57.507068    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-d61c7a/simulator-single-main-d61c7a-kserve-975b88bd-w8cmx" event={"ID":"340a6502-4312-4a36-883c-a9c32e7d0c99","Type":"ContainerStarted","Data":"8ef6bed14190e1f1633992b8df12772be686897adee77cf3e307341d8ca889bd"}
Oct 30 13:54:57.550516 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:57.550455    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-d61c7a/simulator-single-main-d61c7a-kserve-975b88bd-w8cmx" podStartSLOduration=2.550440666 podStartE2EDuration="2.550440666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:54:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:54:57.54936686 +0000 UTC m=+3796.715731513" watchObservedRunningTime="2026-10-30 13:54:57.550440666 +0000 UTC m=+3796.716805314"
Oct 30 13:54:58.066317 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:58.066276    4364 patch_prober.go:29] interesting pod/e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused" start-of-body=
Oct 30 13:54:58.066764 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:58.066340    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4" podUID="03a318c2-ba1f-4c34-a2f4-8b258f71eee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused"
Oct 30 13:54:58.247642 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:58.247584    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-d61c7a/simulator-single-main-d61c7a-kserve-975b88bd-w8cmx"
Oct 30 13:54:58.410616 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:58.410572    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:54:59.065486 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:59.065446    4364 patch_prober.go:29] interesting pod/e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused" start-of-body=
Oct 30 13:54:59.065722 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:54:59.065504    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4" podUID="03a318c2-ba1f-4c34-a2f4-8b258f71eee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused"
Oct 30 13:55:00.065513 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:00.065469    4364 patch_prober.go:29] interesting pod/e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused" start-of-body=
Oct 30 13:55:00.066050 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:00.065530    4364 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4" podUID="03a318c2-ba1f-4c34-a2f4-8b258f71eee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused"
Oct 30 13:55:01.112150 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:01.112108    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4"
Oct 30 13:55:01.112925 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:01.112905    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4"
Oct 30 13:55:01.114240 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:01.114206    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4"
Oct 30 13:55:06.247233 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:06.247197    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-d61c7a/simulator-single-main-d61c7a-kserve-975b88bd-w8cmx"
Oct 30 13:55:06.259593 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:06.259567    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-d61c7a/simulator-single-main-d61c7a-kserve-975b88bd-w8cmx"
Oct 30 13:55:06.568956 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:06.568870    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-d61c7a/simulator-single-main-d61c7a-kserve-975b88bd-w8cmx"
Oct 30 13:55:11.505274 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:55:11.505234    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:55:13.385241 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:13.385202    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:55:27.165082 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:27.165002    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:55:31.257299 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:31.257257    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-d61c7a/simulator-single-main-d61c7a-kserve-975b88bd-w8cmx"]
Oct 30 13:55:36.852510 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:36.852477    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-d61c7a-66b6589849tbc9"]
Oct 30 13:55:36.882371 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:36.882339    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d61c7a-78657kht4h"]
Oct 30 13:55:37.617747 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:37.617713    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d61c7a-66cb874f59-qftdb"]
Oct 30 13:55:37.617955 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:37.617931    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d61c7a-66cb874f59-qftdb" podUID="12b8f29c-158a-4493-9296-2ccd04a10846" containerName="maas-api" containerID="cri-o://9dc4f33e039bb953b0fca65db5fb49763d24b74bd6a417e36c9fa28f4806aeb5" gracePeriod=30
Oct 30 13:55:38.099874 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:38.099848    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d61c7a-66cb874f59-qftdb"
Oct 30 13:55:38.157061 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:38.156987    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12b8f29c-158a-4493-9296-2ccd04a10846-metrics-certs\" (UniqueName: \"kubernetes.io/secret/12b8f29c-158a-4493-9296-2ccd04a10846-metrics-certs\") pod \"12b8f29c-158a-4493-9296-2ccd04a10846\" (UID: \"12b8f29c-158a-4493-9296-2ccd04a10846\") "
Oct 30 13:55:38.157250 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:38.157113    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12b8f29c-158a-4493-9296-2ccd04a10846-kube-api-access-4snm4\" (UniqueName: \"kubernetes.io/projected/12b8f29c-158a-4493-9296-2ccd04a10846-kube-api-access-4snm4\") pod \"12b8f29c-158a-4493-9296-2ccd04a10846\" (UID: \"12b8f29c-158a-4493-9296-2ccd04a10846\") "
Oct 30 13:55:38.157250 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:38.157169    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12b8f29c-158a-4493-9296-2ccd04a10846-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12b8f29c-158a-4493-9296-2ccd04a10846-maas-api-tls\") pod \"12b8f29c-158a-4493-9296-2ccd04a10846\" (UID: \"12b8f29c-158a-4493-9296-2ccd04a10846\") "
Oct 30 13:55:38.157250 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:38.157196    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12b8f29c-158a-4493-9296-2ccd04a10846-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12b8f29c-158a-4493-9296-2ccd04a10846-openshift-service-ca\") pod \"12b8f29c-158a-4493-9296-2ccd04a10846\" (UID: \"12b8f29c-158a-4493-9296-2ccd04a10846\") "
Oct 30 13:55:38.157558 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:38.157527    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12b8f29c-158a-4493-9296-2ccd04a10846-openshift-service-ca" pod "12b8f29c-158a-4493-9296-2ccd04a10846" (UID: "12b8f29c-158a-4493-9296-2ccd04a10846"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:55:38.157732 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:38.157703    4364 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12b8f29c-158a-4493-9296-2ccd04a10846-openshift-service-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:55:38.159191 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:38.159165    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12b8f29c-158a-4493-9296-2ccd04a10846-metrics-certs" pod "12b8f29c-158a-4493-9296-2ccd04a10846" (UID: "12b8f29c-158a-4493-9296-2ccd04a10846"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:55:38.159722 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:38.159700    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12b8f29c-158a-4493-9296-2ccd04a10846-maas-api-tls" pod "12b8f29c-158a-4493-9296-2ccd04a10846" (UID: "12b8f29c-158a-4493-9296-2ccd04a10846"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:55:38.159817 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:38.159732    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12b8f29c-158a-4493-9296-2ccd04a10846-kube-api-access-4snm4" pod "12b8f29c-158a-4493-9296-2ccd04a10846" (UID: "12b8f29c-158a-4493-9296-2ccd04a10846"). InnerVolumeSpecName "kube-api-access-4snm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:55:38.258614 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:38.258561    4364 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12b8f29c-158a-4493-9296-2ccd04a10846-metrics-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:55:38.258614 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:38.258593    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4snm4\" (UniqueName: \"kubernetes.io/projected/12b8f29c-158a-4493-9296-2ccd04a10846-kube-api-access-4snm4\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:55:38.258814 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:38.258623    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12b8f29c-158a-4493-9296-2ccd04a10846-maas-api-tls\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:55:38.738183 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:38.738142    4364 generic.go:356] "Generic (PLEG): container finished" podID="12b8f29c-158a-4493-9296-2ccd04a10846" containerID="9dc4f33e039bb953b0fca65db5fb49763d24b74bd6a417e36c9fa28f4806aeb5" exitCode=0
Oct 30 13:55:38.738396 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:38.738190    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d61c7a-66cb874f59-qftdb"
Oct 30 13:55:38.738396 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:38.738219    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d61c7a-66cb874f59-qftdb" event={"ID":"12b8f29c-158a-4493-9296-2ccd04a10846","Type":"ContainerDied","Data":"9dc4f33e039bb953b0fca65db5fb49763d24b74bd6a417e36c9fa28f4806aeb5"}
Oct 30 13:55:38.738396 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:38.738306    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d61c7a-66cb874f59-qftdb" event={"ID":"12b8f29c-158a-4493-9296-2ccd04a10846","Type":"ContainerDied","Data":"140f389f705e0a1bf834926a8633f9f8ddce12a0ba8618e65ba1769f2f2b7d8e"}
Oct 30 13:55:38.738396 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:38.738344    4364 scope.go:122] "RemoveContainer" containerID="9dc4f33e039bb953b0fca65db5fb49763d24b74bd6a417e36c9fa28f4806aeb5"
Oct 30 13:55:38.748261 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:38.748244    4364 scope.go:122] "RemoveContainer" containerID="9dc4f33e039bb953b0fca65db5fb49763d24b74bd6a417e36c9fa28f4806aeb5"
Oct 30 13:55:38.748514 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:55:38.748487    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dc4f33e039bb953b0fca65db5fb49763d24b74bd6a417e36c9fa28f4806aeb5\": container with ID starting with 9dc4f33e039bb953b0fca65db5fb49763d24b74bd6a417e36c9fa28f4806aeb5 not found: ID does not exist" containerID="9dc4f33e039bb953b0fca65db5fb49763d24b74bd6a417e36c9fa28f4806aeb5"
Oct 30 13:55:38.748572 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:38.748515    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dc4f33e039bb953b0fca65db5fb49763d24b74bd6a417e36c9fa28f4806aeb5"} err="failed to get container status \"9dc4f33e039bb953b0fca65db5fb49763d24b74bd6a417e36c9fa28f4806aeb5\": rpc error: code = NotFound desc = could not find container \"9dc4f33e039bb953b0fca65db5fb49763d24b74bd6a417e36c9fa28f4806aeb5\": container with ID starting with 9dc4f33e039bb953b0fca65db5fb49763d24b74bd6a417e36c9fa28f4806aeb5 not found: ID does not exist"
Oct 30 13:55:38.789934 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:38.789908    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d61c7a-66cb874f59-qftdb"]
Oct 30 13:55:38.802262 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:38.802237    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d61c7a-66cb874f59-qftdb"]
Oct 30 13:55:38.969298 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:38.969265    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12b8f29c-158a-4493-9296-2ccd04a10846" path="/var/lib/kubelet/pods/12b8f29c-158a-4493-9296-2ccd04a10846/volumes"
Oct 30 13:55:39.866570 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:39.866531    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:55:41.504935 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:55:41.504890    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:55:41.853731 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:41.853641    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d61c7a-66b6589849tbc9" podUID="e345a8d5-e10a-44ee-824b-9966bac27061" containerName="payload-processing" containerID="cri-o://2e58969a381fd50f2f4f723ae4b714ae5c739d2d7a47842bdc4762dd4a3ea7ec" gracePeriod=25
Oct 30 13:55:41.883098 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:41.883041    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d61c7a-78657kht4h" podUID="d7df9595-a3d4-4793-a26d-403deda052dd" containerName="payload-pre-processing" containerID="cri-o://5028c5db2b3a58fc67e4dd46314ffeff8ee0bc98a0d163158f3bdbfa3a2c3b0c" gracePeriod=25
Oct 30 13:55:42.391613 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:42.391562    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d61c7a-66b6589849tbc9" podUID="e345a8d5-e10a-44ee-824b-9966bac27061" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:55:43.398702 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:43.398660    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d61c7a-78657kht4h" podUID="d7df9595-a3d4-4793-a26d-403deda052dd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:55:45.842145 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:45.842099    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4"]
Oct 30 13:55:45.842576 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:45.842394    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4" podUID="03a318c2-ba1f-4c34-a2f4-8b258f71eee3" containerName="istio-proxy" containerID="cri-o://e24c02f9e4442ae3a58611ea7aa34787b39f8f15ba027924ac217d0476639626" gracePeriod=30
Oct 30 13:55:46.115124 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:46.115035    4364 patch_prober.go:29] interesting pod/e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Oct 30 13:55:46.115124 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:46.115111    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4" podUID="03a318c2-ba1f-4c34-a2f4-8b258f71eee3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:55:46.277707 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:46.277660    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-d61c7a/simulator-single-main-d61c7a-kserve-975b88bd-w8cmx" podUID="340a6502-4312-4a36-883c-a9c32e7d0c99" containerName="main" containerID="cri-o://5f8683cc67b4e3cb9399977dced85a26082a014bdd5dffa447fdeb95735cbba0" gracePeriod=45
Oct 30 13:55:46.536027 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:46.536002    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-d61c7a/simulator-single-main-d61c7a-kserve-975b88bd-w8cmx"
Oct 30 13:55:46.641850 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:46.641812    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/340a6502-4312-4a36-883c-a9c32e7d0c99-dshm\" (UniqueName: \"kubernetes.io/empty-dir/340a6502-4312-4a36-883c-a9c32e7d0c99-dshm\") pod \"340a6502-4312-4a36-883c-a9c32e7d0c99\" (UID: \"340a6502-4312-4a36-883c-a9c32e7d0c99\") "
Oct 30 13:55:46.642054 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:46.641873    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/340a6502-4312-4a36-883c-a9c32e7d0c99-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/340a6502-4312-4a36-883c-a9c32e7d0c99-model-cache\") pod \"340a6502-4312-4a36-883c-a9c32e7d0c99\" (UID: \"340a6502-4312-4a36-883c-a9c32e7d0c99\") "
Oct 30 13:55:46.642054 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:46.641911    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/340a6502-4312-4a36-883c-a9c32e7d0c99-home\" (UniqueName: \"kubernetes.io/empty-dir/340a6502-4312-4a36-883c-a9c32e7d0c99-home\") pod \"340a6502-4312-4a36-883c-a9c32e7d0c99\" (UID: \"340a6502-4312-4a36-883c-a9c32e7d0c99\") "
Oct 30 13:55:46.642054 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:46.641939    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/340a6502-4312-4a36-883c-a9c32e7d0c99-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/340a6502-4312-4a36-883c-a9c32e7d0c99-tmp-dir\") pod \"340a6502-4312-4a36-883c-a9c32e7d0c99\" (UID: \"340a6502-4312-4a36-883c-a9c32e7d0c99\") "
Oct 30 13:55:46.642054 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:46.641988    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/340a6502-4312-4a36-883c-a9c32e7d0c99-tls-certs\" (UniqueName: \"kubernetes.io/secret/340a6502-4312-4a36-883c-a9c32e7d0c99-tls-certs\") pod \"340a6502-4312-4a36-883c-a9c32e7d0c99\" (UID: \"340a6502-4312-4a36-883c-a9c32e7d0c99\") "
Oct 30 13:55:46.642299 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:46.642095    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/340a6502-4312-4a36-883c-a9c32e7d0c99-kube-api-access-q4bsl\" (UniqueName: \"kubernetes.io/projected/340a6502-4312-4a36-883c-a9c32e7d0c99-kube-api-access-q4bsl\") pod \"340a6502-4312-4a36-883c-a9c32e7d0c99\" (UID: \"340a6502-4312-4a36-883c-a9c32e7d0c99\") "
Oct 30 13:55:46.642299 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:46.642151    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/340a6502-4312-4a36-883c-a9c32e7d0c99-home" pod "340a6502-4312-4a36-883c-a9c32e7d0c99" (UID: "340a6502-4312-4a36-883c-a9c32e7d0c99"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:46.642299 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:46.642218    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/340a6502-4312-4a36-883c-a9c32e7d0c99-model-cache" pod "340a6502-4312-4a36-883c-a9c32e7d0c99" (UID: "340a6502-4312-4a36-883c-a9c32e7d0c99"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:46.642478 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:46.642336    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/340a6502-4312-4a36-883c-a9c32e7d0c99-tmp-dir" pod "340a6502-4312-4a36-883c-a9c32e7d0c99" (UID: "340a6502-4312-4a36-883c-a9c32e7d0c99"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:46.642667 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:46.642642    4364 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/340a6502-4312-4a36-883c-a9c32e7d0c99-model-cache\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:55:46.642780 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:46.642670    4364 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/340a6502-4312-4a36-883c-a9c32e7d0c99-home\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:55:46.642780 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:46.642687    4364 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/340a6502-4312-4a36-883c-a9c32e7d0c99-tmp-dir\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:55:46.644183 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:46.644152    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/340a6502-4312-4a36-883c-a9c32e7d0c99-dshm" pod "340a6502-4312-4a36-883c-a9c32e7d0c99" (UID: "340a6502-4312-4a36-883c-a9c32e7d0c99"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:46.644292 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:46.644211    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/340a6502-4312-4a36-883c-a9c32e7d0c99-tls-certs" pod "340a6502-4312-4a36-883c-a9c32e7d0c99" (UID: "340a6502-4312-4a36-883c-a9c32e7d0c99"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:55:46.644349 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:46.644309    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/340a6502-4312-4a36-883c-a9c32e7d0c99-kube-api-access-q4bsl" pod "340a6502-4312-4a36-883c-a9c32e7d0c99" (UID: "340a6502-4312-4a36-883c-a9c32e7d0c99"). InnerVolumeSpecName "kube-api-access-q4bsl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:55:46.743996 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:46.743963    4364 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/340a6502-4312-4a36-883c-a9c32e7d0c99-dshm\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:55:46.743996 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:46.743992    4364 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/340a6502-4312-4a36-883c-a9c32e7d0c99-tls-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:55:46.744210 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:46.744003    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4bsl\" (UniqueName: \"kubernetes.io/projected/340a6502-4312-4a36-883c-a9c32e7d0c99-kube-api-access-q4bsl\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:55:46.786008 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:46.785974    4364 generic.go:356] "Generic (PLEG): container finished" podID="340a6502-4312-4a36-883c-a9c32e7d0c99" containerID="5f8683cc67b4e3cb9399977dced85a26082a014bdd5dffa447fdeb95735cbba0" exitCode=0
Oct 30 13:55:46.786197 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:46.786023    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-d61c7a/simulator-single-main-d61c7a-kserve-975b88bd-w8cmx"
Oct 30 13:55:46.786197 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:46.786051    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-d61c7a/simulator-single-main-d61c7a-kserve-975b88bd-w8cmx" event={"ID":"340a6502-4312-4a36-883c-a9c32e7d0c99","Type":"ContainerDied","Data":"5f8683cc67b4e3cb9399977dced85a26082a014bdd5dffa447fdeb95735cbba0"}
Oct 30 13:55:46.786197 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:46.786093    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-d61c7a/simulator-single-main-d61c7a-kserve-975b88bd-w8cmx" event={"ID":"340a6502-4312-4a36-883c-a9c32e7d0c99","Type":"ContainerDied","Data":"8ef6bed14190e1f1633992b8df12772be686897adee77cf3e307341d8ca889bd"}
Oct 30 13:55:46.786197 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:46.786111    4364 scope.go:122] "RemoveContainer" containerID="5f8683cc67b4e3cb9399977dced85a26082a014bdd5dffa447fdeb95735cbba0"
Oct 30 13:55:46.797993 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:46.797892    4364 scope.go:122] "RemoveContainer" containerID="5f8683cc67b4e3cb9399977dced85a26082a014bdd5dffa447fdeb95735cbba0"
Oct 30 13:55:46.798425 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:55:46.798396    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f8683cc67b4e3cb9399977dced85a26082a014bdd5dffa447fdeb95735cbba0\": container with ID starting with 5f8683cc67b4e3cb9399977dced85a26082a014bdd5dffa447fdeb95735cbba0 not found: ID does not exist" containerID="5f8683cc67b4e3cb9399977dced85a26082a014bdd5dffa447fdeb95735cbba0"
Oct 30 13:55:46.798521 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:46.798435    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f8683cc67b4e3cb9399977dced85a26082a014bdd5dffa447fdeb95735cbba0"} err="failed to get container status \"5f8683cc67b4e3cb9399977dced85a26082a014bdd5dffa447fdeb95735cbba0\": rpc error: code = NotFound desc = could not find container \"5f8683cc67b4e3cb9399977dced85a26082a014bdd5dffa447fdeb95735cbba0\": container with ID starting with 5f8683cc67b4e3cb9399977dced85a26082a014bdd5dffa447fdeb95735cbba0 not found: ID does not exist"
Oct 30 13:55:46.837921 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:46.837888    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-d61c7a/simulator-single-main-d61c7a-kserve-975b88bd-w8cmx"]
Oct 30 13:55:46.847464 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:46.845920    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-d61c7a/simulator-single-main-d61c7a-kserve-975b88bd-w8cmx"]
Oct 30 13:55:46.968464 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:46.968387    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="340a6502-4312-4a36-883c-a9c32e7d0c99" path="/var/lib/kubelet/pods/340a6502-4312-4a36-883c-a9c32e7d0c99/volumes"
Oct 30 13:55:49.891612 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:49.891567    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-6a914e8f-kserve-7489c55769-mzrxr"]
Oct 30 13:55:49.892300 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:49.892280    4364 cpu_manager.go:414] "containerMap: removing container" podUID="340a6502-4312-4a36-883c-a9c32e7d0c99" containerName="main"
Oct 30 13:55:49.892350 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:49.892305    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="340a6502-4312-4a36-883c-a9c32e7d0c99" containerName="main"
Oct 30 13:55:49.892350 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:49.892323    4364 cpu_manager.go:414] "containerMap: removing container" podUID="12b8f29c-158a-4493-9296-2ccd04a10846" containerName="maas-api"
Oct 30 13:55:49.892350 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:49.892332    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12b8f29c-158a-4493-9296-2ccd04a10846" containerName="maas-api"
Oct 30 13:55:49.892456 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:49.892445    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="340a6502-4312-4a36-883c-a9c32e7d0c99" containerName="main"
Oct 30 13:55:49.892493 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:49.892461    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="12b8f29c-158a-4493-9296-2ccd04a10846" containerName="maas-api"
Oct 30 13:55:49.896018 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:49.895995    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-6a914e8f-kserve-7489c55769-mzrxr"
Oct 30 13:55:49.913623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:49.913580    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-6a914e8f-kserve-7489c55769-mzrxr"]
Oct 30 13:55:49.922684 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:49.922654    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-6a914e8f-kserve-self-signed-certs\""
Oct 30 13:55:49.978334 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:49.978300    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6e05360-004c-49e4-8359-7d56f2ba6b71-tmp-dir\") pod \"e2e-same-id-a-6a914e8f-kserve-7489c55769-mzrxr\" (UID: \"d6e05360-004c-49e4-8359-7d56f2ba6b71\") " pod="llm/e2e-same-id-a-6a914e8f-kserve-7489c55769-mzrxr"
Oct 30 13:55:49.978475 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:49.978344    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6e05360-004c-49e4-8359-7d56f2ba6b71-model-cache\") pod \"e2e-same-id-a-6a914e8f-kserve-7489c55769-mzrxr\" (UID: \"d6e05360-004c-49e4-8359-7d56f2ba6b71\") " pod="llm/e2e-same-id-a-6a914e8f-kserve-7489c55769-mzrxr"
Oct 30 13:55:49.978475 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:49.978422    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6e05360-004c-49e4-8359-7d56f2ba6b71-home\") pod \"e2e-same-id-a-6a914e8f-kserve-7489c55769-mzrxr\" (UID: \"d6e05360-004c-49e4-8359-7d56f2ba6b71\") " pod="llm/e2e-same-id-a-6a914e8f-kserve-7489c55769-mzrxr"
Oct 30 13:55:49.978592 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:49.978509    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6e05360-004c-49e4-8359-7d56f2ba6b71-tls-certs\") pod \"e2e-same-id-a-6a914e8f-kserve-7489c55769-mzrxr\" (UID: \"d6e05360-004c-49e4-8359-7d56f2ba6b71\") " pod="llm/e2e-same-id-a-6a914e8f-kserve-7489c55769-mzrxr"
Oct 30 13:55:49.978592 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:49.978567    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6e05360-004c-49e4-8359-7d56f2ba6b71-dshm\") pod \"e2e-same-id-a-6a914e8f-kserve-7489c55769-mzrxr\" (UID: \"d6e05360-004c-49e4-8359-7d56f2ba6b71\") " pod="llm/e2e-same-id-a-6a914e8f-kserve-7489c55769-mzrxr"
Oct 30 13:55:49.978592 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:49.978585    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwvq8\" (UniqueName: \"kubernetes.io/projected/d6e05360-004c-49e4-8359-7d56f2ba6b71-kube-api-access-wwvq8\") pod \"e2e-same-id-a-6a914e8f-kserve-7489c55769-mzrxr\" (UID: \"d6e05360-004c-49e4-8359-7d56f2ba6b71\") " pod="llm/e2e-same-id-a-6a914e8f-kserve-7489c55769-mzrxr"
Oct 30 13:55:50.079944 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:50.079905    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6e05360-004c-49e4-8359-7d56f2ba6b71-tmp-dir\") pod \"e2e-same-id-a-6a914e8f-kserve-7489c55769-mzrxr\" (UID: \"d6e05360-004c-49e4-8359-7d56f2ba6b71\") " pod="llm/e2e-same-id-a-6a914e8f-kserve-7489c55769-mzrxr"
Oct 30 13:55:50.080312 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:50.079971    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6e05360-004c-49e4-8359-7d56f2ba6b71-model-cache\") pod \"e2e-same-id-a-6a914e8f-kserve-7489c55769-mzrxr\" (UID: \"d6e05360-004c-49e4-8359-7d56f2ba6b71\") " pod="llm/e2e-same-id-a-6a914e8f-kserve-7489c55769-mzrxr"
Oct 30 13:55:50.080312 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:50.080011    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6e05360-004c-49e4-8359-7d56f2ba6b71-home\") pod \"e2e-same-id-a-6a914e8f-kserve-7489c55769-mzrxr\" (UID: \"d6e05360-004c-49e4-8359-7d56f2ba6b71\") " pod="llm/e2e-same-id-a-6a914e8f-kserve-7489c55769-mzrxr"
Oct 30 13:55:50.080312 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:50.080115    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6e05360-004c-49e4-8359-7d56f2ba6b71-tls-certs\") pod \"e2e-same-id-a-6a914e8f-kserve-7489c55769-mzrxr\" (UID: \"d6e05360-004c-49e4-8359-7d56f2ba6b71\") " pod="llm/e2e-same-id-a-6a914e8f-kserve-7489c55769-mzrxr"
Oct 30 13:55:50.080312 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:50.080165    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6e05360-004c-49e4-8359-7d56f2ba6b71-dshm\") pod \"e2e-same-id-a-6a914e8f-kserve-7489c55769-mzrxr\" (UID: \"d6e05360-004c-49e4-8359-7d56f2ba6b71\") " pod="llm/e2e-same-id-a-6a914e8f-kserve-7489c55769-mzrxr"
Oct 30 13:55:50.080312 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:50.080205    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwvq8\" (UniqueName: \"kubernetes.io/projected/d6e05360-004c-49e4-8359-7d56f2ba6b71-kube-api-access-wwvq8\") pod \"e2e-same-id-a-6a914e8f-kserve-7489c55769-mzrxr\" (UID: \"d6e05360-004c-49e4-8359-7d56f2ba6b71\") " pod="llm/e2e-same-id-a-6a914e8f-kserve-7489c55769-mzrxr"
Oct 30 13:55:50.080672 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:50.080593    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6e05360-004c-49e4-8359-7d56f2ba6b71-tmp-dir\") pod \"e2e-same-id-a-6a914e8f-kserve-7489c55769-mzrxr\" (UID: \"d6e05360-004c-49e4-8359-7d56f2ba6b71\") " pod="llm/e2e-same-id-a-6a914e8f-kserve-7489c55769-mzrxr"
Oct 30 13:55:50.080948 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:50.080921    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6e05360-004c-49e4-8359-7d56f2ba6b71-model-cache\") pod \"e2e-same-id-a-6a914e8f-kserve-7489c55769-mzrxr\" (UID: \"d6e05360-004c-49e4-8359-7d56f2ba6b71\") " pod="llm/e2e-same-id-a-6a914e8f-kserve-7489c55769-mzrxr"
Oct 30 13:55:50.081292 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:50.081269    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6e05360-004c-49e4-8359-7d56f2ba6b71-home\") pod \"e2e-same-id-a-6a914e8f-kserve-7489c55769-mzrxr\" (UID: \"d6e05360-004c-49e4-8359-7d56f2ba6b71\") " pod="llm/e2e-same-id-a-6a914e8f-kserve-7489c55769-mzrxr"
Oct 30 13:55:50.083048 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:50.082982    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6e05360-004c-49e4-8359-7d56f2ba6b71-dshm\") pod \"e2e-same-id-a-6a914e8f-kserve-7489c55769-mzrxr\" (UID: \"d6e05360-004c-49e4-8359-7d56f2ba6b71\") " pod="llm/e2e-same-id-a-6a914e8f-kserve-7489c55769-mzrxr"
Oct 30 13:55:50.083160 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:50.083087    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6e05360-004c-49e4-8359-7d56f2ba6b71-tls-certs\") pod \"e2e-same-id-a-6a914e8f-kserve-7489c55769-mzrxr\" (UID: \"d6e05360-004c-49e4-8359-7d56f2ba6b71\") " pod="llm/e2e-same-id-a-6a914e8f-kserve-7489c55769-mzrxr"
Oct 30 13:55:50.113094 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:50.113057    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwvq8\" (UniqueName: \"kubernetes.io/projected/d6e05360-004c-49e4-8359-7d56f2ba6b71-kube-api-access-wwvq8\") pod \"e2e-same-id-a-6a914e8f-kserve-7489c55769-mzrxr\" (UID: \"d6e05360-004c-49e4-8359-7d56f2ba6b71\") " pod="llm/e2e-same-id-a-6a914e8f-kserve-7489c55769-mzrxr"
Oct 30 13:55:50.208549 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:50.208469    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-6a914e8f-kserve-7489c55769-mzrxr"
Oct 30 13:55:50.511922 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:50.511826    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-6a914e8f-kserve-7489c55769-mzrxr"]
Oct 30 13:55:50.514682 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:55:50.514657    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd6e05360_004c_49e4_8359_7d56f2ba6b71.slice/crio-379ab5cbdd2eeebc3d9cefbe2a6462b4be1b57dbbf84ae58edcadb994114ae72 WatchSource:0}: Error finding container 379ab5cbdd2eeebc3d9cefbe2a6462b4be1b57dbbf84ae58edcadb994114ae72: Status 404 returned error can't find the container with id 379ab5cbdd2eeebc3d9cefbe2a6462b4be1b57dbbf84ae58edcadb994114ae72
Oct 30 13:55:50.810641 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:50.810509    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-6a914e8f-kserve-7489c55769-mzrxr" event={"ID":"d6e05360-004c-49e4-8359-7d56f2ba6b71","Type":"ContainerStarted","Data":"14c65847688573ddb25d4c75f97c4499bda17c917fc1d13e88d5ccc52c76f63a"}
Oct 30 13:55:50.810641 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:50.810562    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-6a914e8f-kserve-7489c55769-mzrxr" event={"ID":"d6e05360-004c-49e4-8359-7d56f2ba6b71","Type":"ContainerStarted","Data":"379ab5cbdd2eeebc3d9cefbe2a6462b4be1b57dbbf84ae58edcadb994114ae72"}
Oct 30 13:55:50.855953 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:50.855900    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-6a914e8f-kserve-7489c55769-mzrxr" podStartSLOduration=1.8558857899999999 podStartE2EDuration="1.85588579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:55:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:55:50.852972634 +0000 UTC m=+3850.019337297" watchObservedRunningTime="2026-10-30 13:55:50.85588579 +0000 UTC m=+3850.022250438"
Oct 30 13:55:51.109111 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:51.109082    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4"
Oct 30 13:55:51.191121 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:51.191092    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03a318c2-ba1f-4c34-a2f4-8b258f71eee3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a318c2-ba1f-4c34-a2f4-8b258f71eee3-credential-socket\") pod \"03a318c2-ba1f-4c34-a2f4-8b258f71eee3\" (UID: \"03a318c2-ba1f-4c34-a2f4-8b258f71eee3\") "
Oct 30 13:55:51.191301 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:51.191136    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03a318c2-ba1f-4c34-a2f4-8b258f71eee3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03a318c2-ba1f-4c34-a2f4-8b258f71eee3-istio-data\") pod \"03a318c2-ba1f-4c34-a2f4-8b258f71eee3\" (UID: \"03a318c2-ba1f-4c34-a2f4-8b258f71eee3\") "
Oct 30 13:55:51.191301 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:51.191175    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03a318c2-ba1f-4c34-a2f4-8b258f71eee3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03a318c2-ba1f-4c34-a2f4-8b258f71eee3-istiod-ca-cert\") pod \"03a318c2-ba1f-4c34-a2f4-8b258f71eee3\" (UID: \"03a318c2-ba1f-4c34-a2f4-8b258f71eee3\") "
Oct 30 13:55:51.191301 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:51.191241    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03a318c2-ba1f-4c34-a2f4-8b258f71eee3-istio-token\" (UniqueName: \"kubernetes.io/projected/03a318c2-ba1f-4c34-a2f4-8b258f71eee3-istio-token\") pod \"03a318c2-ba1f-4c34-a2f4-8b258f71eee3\" (UID: \"03a318c2-ba1f-4c34-a2f4-8b258f71eee3\") "
Oct 30 13:55:51.191476 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:51.191307    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03a318c2-ba1f-4c34-a2f4-8b258f71eee3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a318c2-ba1f-4c34-a2f4-8b258f71eee3-workload-socket\") pod \"03a318c2-ba1f-4c34-a2f4-8b258f71eee3\" (UID: \"03a318c2-ba1f-4c34-a2f4-8b258f71eee3\") "
Oct 30 13:55:51.191476 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:51.191351    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03a318c2-ba1f-4c34-a2f4-8b258f71eee3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03a318c2-ba1f-4c34-a2f4-8b258f71eee3-istio-envoy\") pod \"03a318c2-ba1f-4c34-a2f4-8b258f71eee3\" (UID: \"03a318c2-ba1f-4c34-a2f4-8b258f71eee3\") "
Oct 30 13:55:51.191476 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:51.191379    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03a318c2-ba1f-4c34-a2f4-8b258f71eee3-credential-socket" pod "03a318c2-ba1f-4c34-a2f4-8b258f71eee3" (UID: "03a318c2-ba1f-4c34-a2f4-8b258f71eee3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:51.191476 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:51.191401    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03a318c2-ba1f-4c34-a2f4-8b258f71eee3-kube-api-access-2g44c\" (UniqueName: \"kubernetes.io/projected/03a318c2-ba1f-4c34-a2f4-8b258f71eee3-kube-api-access-2g44c\") pod \"03a318c2-ba1f-4c34-a2f4-8b258f71eee3\" (UID: \"03a318c2-ba1f-4c34-a2f4-8b258f71eee3\") "
Oct 30 13:55:51.191476 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:51.191429    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/03a318c2-ba1f-4c34-a2f4-8b258f71eee3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03a318c2-ba1f-4c34-a2f4-8b258f71eee3-istio-podinfo\") pod \"03a318c2-ba1f-4c34-a2f4-8b258f71eee3\" (UID: \"03a318c2-ba1f-4c34-a2f4-8b258f71eee3\") "
Oct 30 13:55:51.191476 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:51.191463    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03a318c2-ba1f-4c34-a2f4-8b258f71eee3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03a318c2-ba1f-4c34-a2f4-8b258f71eee3-workload-certs\") pod \"03a318c2-ba1f-4c34-a2f4-8b258f71eee3\" (UID: \"03a318c2-ba1f-4c34-a2f4-8b258f71eee3\") "
Oct 30 13:55:51.191902 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:51.191485    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03a318c2-ba1f-4c34-a2f4-8b258f71eee3-istio-data" pod "03a318c2-ba1f-4c34-a2f4-8b258f71eee3" (UID: "03a318c2-ba1f-4c34-a2f4-8b258f71eee3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:51.191902 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:51.191692    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03a318c2-ba1f-4c34-a2f4-8b258f71eee3-workload-socket" pod "03a318c2-ba1f-4c34-a2f4-8b258f71eee3" (UID: "03a318c2-ba1f-4c34-a2f4-8b258f71eee3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:51.192019 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:51.191910    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03a318c2-ba1f-4c34-a2f4-8b258f71eee3-workload-certs" pod "03a318c2-ba1f-4c34-a2f4-8b258f71eee3" (UID: "03a318c2-ba1f-4c34-a2f4-8b258f71eee3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:51.192019 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:51.191914    4364 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a318c2-ba1f-4c34-a2f4-8b258f71eee3-credential-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:55:51.192019 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:51.191962    4364 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03a318c2-ba1f-4c34-a2f4-8b258f71eee3-istio-data\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:55:51.192019 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:51.191980    4364 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a318c2-ba1f-4c34-a2f4-8b258f71eee3-workload-socket\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:55:51.192224 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:51.192062    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03a318c2-ba1f-4c34-a2f4-8b258f71eee3-istiod-ca-cert" pod "03a318c2-ba1f-4c34-a2f4-8b258f71eee3" (UID: "03a318c2-ba1f-4c34-a2f4-8b258f71eee3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:55:51.194014 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:51.193987    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03a318c2-ba1f-4c34-a2f4-8b258f71eee3-istio-envoy" pod "03a318c2-ba1f-4c34-a2f4-8b258f71eee3" (UID: "03a318c2-ba1f-4c34-a2f4-8b258f71eee3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:51.194151 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:51.194023    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03a318c2-ba1f-4c34-a2f4-8b258f71eee3-istio-token" pod "03a318c2-ba1f-4c34-a2f4-8b258f71eee3" (UID: "03a318c2-ba1f-4c34-a2f4-8b258f71eee3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:55:51.194151 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:51.194066    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03a318c2-ba1f-4c34-a2f4-8b258f71eee3-istio-podinfo" pod "03a318c2-ba1f-4c34-a2f4-8b258f71eee3" (UID: "03a318c2-ba1f-4c34-a2f4-8b258f71eee3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:55:51.194304 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:51.194165    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03a318c2-ba1f-4c34-a2f4-8b258f71eee3-kube-api-access-2g44c" pod "03a318c2-ba1f-4c34-a2f4-8b258f71eee3" (UID: "03a318c2-ba1f-4c34-a2f4-8b258f71eee3"). InnerVolumeSpecName "kube-api-access-2g44c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:55:51.209299 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:51.209278    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-6a914e8f-kserve-7489c55769-mzrxr"
Oct 30 13:55:51.293212 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:51.293183    4364 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03a318c2-ba1f-4c34-a2f4-8b258f71eee3-istiod-ca-cert\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:55:51.293212 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:51.293209    4364 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03a318c2-ba1f-4c34-a2f4-8b258f71eee3-istio-token\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:55:51.293212 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:51.293218    4364 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03a318c2-ba1f-4c34-a2f4-8b258f71eee3-istio-envoy\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:55:51.293408 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:51.293227    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2g44c\" (UniqueName: \"kubernetes.io/projected/03a318c2-ba1f-4c34-a2f4-8b258f71eee3-kube-api-access-2g44c\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:55:51.293408 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:51.293236    4364 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03a318c2-ba1f-4c34-a2f4-8b258f71eee3-istio-podinfo\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:55:51.293408 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:51.293244    4364 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03a318c2-ba1f-4c34-a2f4-8b258f71eee3-workload-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:55:51.818709 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:51.818678    4364 generic.go:356] "Generic (PLEG): container finished" podID="03a318c2-ba1f-4c34-a2f4-8b258f71eee3" containerID="e24c02f9e4442ae3a58611ea7aa34787b39f8f15ba027924ac217d0476639626" exitCode=0
Oct 30 13:55:51.818893 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:51.818731    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4"
Oct 30 13:55:51.818893 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:51.818770    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4" event={"ID":"03a318c2-ba1f-4c34-a2f4-8b258f71eee3","Type":"ContainerDied","Data":"e24c02f9e4442ae3a58611ea7aa34787b39f8f15ba027924ac217d0476639626"}
Oct 30 13:55:51.818893 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:51.818822    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4" event={"ID":"03a318c2-ba1f-4c34-a2f4-8b258f71eee3","Type":"ContainerDied","Data":"3f24a340e942e12a4066962c7d6ff2cc9bee324d36f29054da08bf8de805686d"}
Oct 30 13:55:51.818893 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:51.818845    4364 scope.go:122] "RemoveContainer" containerID="e24c02f9e4442ae3a58611ea7aa34787b39f8f15ba027924ac217d0476639626"
Oct 30 13:55:51.829094 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:51.829074    4364 scope.go:122] "RemoveContainer" containerID="0772dfd437e8f077a7036aa72ba2520bee2255ff3f8c4cf92f6921e5a37a9b7e"
Oct 30 13:55:51.839077 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:51.839058    4364 scope.go:122] "RemoveContainer" containerID="e24c02f9e4442ae3a58611ea7aa34787b39f8f15ba027924ac217d0476639626"
Oct 30 13:55:51.839348 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:55:51.839327    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e24c02f9e4442ae3a58611ea7aa34787b39f8f15ba027924ac217d0476639626\": container with ID starting with e24c02f9e4442ae3a58611ea7aa34787b39f8f15ba027924ac217d0476639626 not found: ID does not exist" containerID="e24c02f9e4442ae3a58611ea7aa34787b39f8f15ba027924ac217d0476639626"
Oct 30 13:55:51.839423 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:51.839357    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e24c02f9e4442ae3a58611ea7aa34787b39f8f15ba027924ac217d0476639626"} err="failed to get container status \"e24c02f9e4442ae3a58611ea7aa34787b39f8f15ba027924ac217d0476639626\": rpc error: code = NotFound desc = could not find container \"e24c02f9e4442ae3a58611ea7aa34787b39f8f15ba027924ac217d0476639626\": container with ID starting with e24c02f9e4442ae3a58611ea7aa34787b39f8f15ba027924ac217d0476639626 not found: ID does not exist"
Oct 30 13:55:51.839423 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:51.839373    4364 scope.go:122] "RemoveContainer" containerID="0772dfd437e8f077a7036aa72ba2520bee2255ff3f8c4cf92f6921e5a37a9b7e"
Oct 30 13:55:51.839653 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:55:51.839630    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0772dfd437e8f077a7036aa72ba2520bee2255ff3f8c4cf92f6921e5a37a9b7e\": container with ID starting with 0772dfd437e8f077a7036aa72ba2520bee2255ff3f8c4cf92f6921e5a37a9b7e not found: ID does not exist" containerID="0772dfd437e8f077a7036aa72ba2520bee2255ff3f8c4cf92f6921e5a37a9b7e"
Oct 30 13:55:51.839710 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:51.839659    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0772dfd437e8f077a7036aa72ba2520bee2255ff3f8c4cf92f6921e5a37a9b7e"} err="failed to get container status \"0772dfd437e8f077a7036aa72ba2520bee2255ff3f8c4cf92f6921e5a37a9b7e\": rpc error: code = NotFound desc = could not find container \"0772dfd437e8f077a7036aa72ba2520bee2255ff3f8c4cf92f6921e5a37a9b7e\": container with ID starting with 0772dfd437e8f077a7036aa72ba2520bee2255ff3f8c4cf92f6921e5a37a9b7e not found: ID does not exist"
Oct 30 13:55:51.872146 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:51.872123    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4"]
Oct 30 13:55:51.884211 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:51.884188    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-d61c7a-openshift-default-5f774d9cdfgdhp4"]
Oct 30 13:55:51.988478 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:51.988444    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:55:52.391935 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:52.391892    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d61c7a-66b6589849tbc9" podUID="e345a8d5-e10a-44ee-824b-9966bac27061" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:55:52.973035 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:52.972984    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03a318c2-ba1f-4c34-a2f4-8b258f71eee3" path="/var/lib/kubelet/pods/03a318c2-ba1f-4c34-a2f4-8b258f71eee3/volumes"
Oct 30 13:55:53.399458 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:55:53.399410    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d61c7a-78657kht4h" podUID="d7df9595-a3d4-4793-a26d-403deda052dd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:56:00.209640 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:00.209590    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-6a914e8f-kserve-7489c55769-mzrxr"
Oct 30 13:56:00.222091 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:00.222066    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-6a914e8f-kserve-7489c55769-mzrxr"
Oct 30 13:56:01.017947 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:01.017909    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-6a914e8f-kserve-7489c55769-mzrxr"
Oct 30 13:56:01.893133 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:56:01.893102    4364 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode345a8d5_e10a_44ee_824b_9966bac27061.slice/crio-conmon-2e58969a381fd50f2f4f723ae4b714ae5c739d2d7a47842bdc4762dd4a3ea7ec.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 13:56:01.893465 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:56:01.893144    4364 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode345a8d5_e10a_44ee_824b_9966bac27061.slice/crio-conmon-2e58969a381fd50f2f4f723ae4b714ae5c739d2d7a47842bdc4762dd4a3ea7ec.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 13:56:02.025181 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:02.025140    4364 generic.go:356] "Generic (PLEG): container finished" podID="d7df9595-a3d4-4793-a26d-403deda052dd" containerID="5028c5db2b3a58fc67e4dd46314ffeff8ee0bc98a0d163158f3bdbfa3a2c3b0c" exitCode=0
Oct 30 13:56:02.025345 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:02.025311    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d61c7a-78657kht4h" event={"ID":"d7df9595-a3d4-4793-a26d-403deda052dd","Type":"ContainerDied","Data":"5028c5db2b3a58fc67e4dd46314ffeff8ee0bc98a0d163158f3bdbfa3a2c3b0c"}
Oct 30 13:56:02.027568 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:02.027538    4364 generic.go:356] "Generic (PLEG): container finished" podID="e345a8d5-e10a-44ee-824b-9966bac27061" containerID="2e58969a381fd50f2f4f723ae4b714ae5c739d2d7a47842bdc4762dd4a3ea7ec" exitCode=0
Oct 30 13:56:02.027776 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:02.027746    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d61c7a-66b6589849tbc9" event={"ID":"e345a8d5-e10a-44ee-824b-9966bac27061","Type":"ContainerDied","Data":"2e58969a381fd50f2f4f723ae4b714ae5c739d2d7a47842bdc4762dd4a3ea7ec"}
Oct 30 13:56:02.152419 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:02.152393    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d61c7a-66b6589849tbc9"
Oct 30 13:56:02.174264 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:02.174235    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d61c7a-78657kht4h"
Oct 30 13:56:02.241889 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:02.241859    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7df9595-a3d4-4793-a26d-403deda052dd-kube-api-access-zxs6q\" (UniqueName: \"kubernetes.io/projected/d7df9595-a3d4-4793-a26d-403deda052dd-kube-api-access-zxs6q\") pod \"d7df9595-a3d4-4793-a26d-403deda052dd\" (UID: \"d7df9595-a3d4-4793-a26d-403deda052dd\") "
Oct 30 13:56:02.242095 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:02.241940    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e345a8d5-e10a-44ee-824b-9966bac27061-kube-api-access-s876x\" (UniqueName: \"kubernetes.io/projected/e345a8d5-e10a-44ee-824b-9966bac27061-kube-api-access-s876x\") pod \"e345a8d5-e10a-44ee-824b-9966bac27061\" (UID: \"e345a8d5-e10a-44ee-824b-9966bac27061\") "
Oct 30 13:56:02.242095 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:02.242012    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e345a8d5-e10a-44ee-824b-9966bac27061-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e345a8d5-e10a-44ee-824b-9966bac27061-plugins-config-volume\") pod \"e345a8d5-e10a-44ee-824b-9966bac27061\" (UID: \"e345a8d5-e10a-44ee-824b-9966bac27061\") "
Oct 30 13:56:02.242095 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:02.242073    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7df9595-a3d4-4793-a26d-403deda052dd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7df9595-a3d4-4793-a26d-403deda052dd-plugins-config-volume\") pod \"d7df9595-a3d4-4793-a26d-403deda052dd\" (UID: \"d7df9595-a3d4-4793-a26d-403deda052dd\") "
Oct 30 13:56:02.242446 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:02.242417    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e345a8d5-e10a-44ee-824b-9966bac27061-plugins-config-volume" pod "e345a8d5-e10a-44ee-824b-9966bac27061" (UID: "e345a8d5-e10a-44ee-824b-9966bac27061"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:56:02.242559 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:02.242452    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7df9595-a3d4-4793-a26d-403deda052dd-plugins-config-volume" pod "d7df9595-a3d4-4793-a26d-403deda052dd" (UID: "d7df9595-a3d4-4793-a26d-403deda052dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:56:02.244153 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:02.244126    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e345a8d5-e10a-44ee-824b-9966bac27061-kube-api-access-s876x" pod "e345a8d5-e10a-44ee-824b-9966bac27061" (UID: "e345a8d5-e10a-44ee-824b-9966bac27061"). InnerVolumeSpecName "kube-api-access-s876x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:56:02.245203 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:02.245181    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7df9595-a3d4-4793-a26d-403deda052dd-kube-api-access-zxs6q" pod "d7df9595-a3d4-4793-a26d-403deda052dd" (UID: "d7df9595-a3d4-4793-a26d-403deda052dd"). InnerVolumeSpecName "kube-api-access-zxs6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:56:02.343311 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:02.343274    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxs6q\" (UniqueName: \"kubernetes.io/projected/d7df9595-a3d4-4793-a26d-403deda052dd-kube-api-access-zxs6q\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:56:02.343311 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:02.343305    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s876x\" (UniqueName: \"kubernetes.io/projected/e345a8d5-e10a-44ee-824b-9966bac27061-kube-api-access-s876x\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:56:02.343311 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:02.343316    4364 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e345a8d5-e10a-44ee-824b-9966bac27061-plugins-config-volume\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:56:02.343539 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:02.343325    4364 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7df9595-a3d4-4793-a26d-403deda052dd-plugins-config-volume\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:56:03.034104 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:03.034069    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d61c7a-66b6589849tbc9"
Oct 30 13:56:03.034523 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:03.034103    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d61c7a-66b6589849tbc9" event={"ID":"e345a8d5-e10a-44ee-824b-9966bac27061","Type":"ContainerDied","Data":"89c4c65f0965986fd4a9ddca1125e07c772159ebb1fadbba555b651d8556367f"}
Oct 30 13:56:03.034523 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:03.034165    4364 scope.go:122] "RemoveContainer" containerID="2e58969a381fd50f2f4f723ae4b714ae5c739d2d7a47842bdc4762dd4a3ea7ec"
Oct 30 13:56:03.035618 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:03.035582    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d61c7a-78657kht4h"
Oct 30 13:56:03.035694 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:03.035583    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d61c7a-78657kht4h" event={"ID":"d7df9595-a3d4-4793-a26d-403deda052dd","Type":"ContainerDied","Data":"156aee45236682c1a5500c72bbd4f069c8ed9c4a45f4bb196fe91e127b704475"}
Oct 30 13:56:03.045920 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:03.045860    4364 scope.go:122] "RemoveContainer" containerID="5028c5db2b3a58fc67e4dd46314ffeff8ee0bc98a0d163158f3bdbfa3a2c3b0c"
Oct 30 13:56:03.086782 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:03.086756    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-d61c7a-66b6589849tbc9"]
Oct 30 13:56:03.098900 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:03.098874    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-d61c7a-66b6589849tbc9"]
Oct 30 13:56:03.127526 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:03.127468    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d61c7a-78657kht4h"]
Oct 30 13:56:03.133055 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:03.133032    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d61c7a-78657kht4h"]
Oct 30 13:56:04.968202 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:04.968166    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7df9595-a3d4-4793-a26d-403deda052dd" path="/var/lib/kubelet/pods/d7df9595-a3d4-4793-a26d-403deda052dd/volumes"
Oct 30 13:56:04.968611 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:04.968561    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e345a8d5-e10a-44ee-824b-9966bac27061" path="/var/lib/kubelet/pods/e345a8d5-e10a-44ee-824b-9966bac27061/volumes"
Oct 30 13:56:07.029286 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:07.029253    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-6a914e8f-kserve-8c7fb8749-fw2lj"]
Oct 30 13:56:07.029769 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:07.029681    4364 cpu_manager.go:414] "containerMap: removing container" podUID="d7df9595-a3d4-4793-a26d-403deda052dd" containerName="payload-pre-processing"
Oct 30 13:56:07.029769 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:07.029695    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7df9595-a3d4-4793-a26d-403deda052dd" containerName="payload-pre-processing"
Oct 30 13:56:07.029769 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:07.029715    4364 cpu_manager.go:414] "containerMap: removing container" podUID="03a318c2-ba1f-4c34-a2f4-8b258f71eee3" containerName="istio-proxy"
Oct 30 13:56:07.029769 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:07.029722    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03a318c2-ba1f-4c34-a2f4-8b258f71eee3" containerName="istio-proxy"
Oct 30 13:56:07.029769 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:07.029746    4364 cpu_manager.go:414] "containerMap: removing container" podUID="e345a8d5-e10a-44ee-824b-9966bac27061" containerName="payload-processing"
Oct 30 13:56:07.029769 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:07.029753    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e345a8d5-e10a-44ee-824b-9966bac27061" containerName="payload-processing"
Oct 30 13:56:07.030068 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:07.029817    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="03a318c2-ba1f-4c34-a2f4-8b258f71eee3" containerName="istio-proxy"
Oct 30 13:56:07.030068 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:07.029824    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="e345a8d5-e10a-44ee-824b-9966bac27061" containerName="payload-processing"
Oct 30 13:56:07.030068 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:07.029833    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7df9595-a3d4-4793-a26d-403deda052dd" containerName="payload-pre-processing"
Oct 30 13:56:07.030068 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:07.029844    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="03a318c2-ba1f-4c34-a2f4-8b258f71eee3" containerName="istio-proxy"
Oct 30 13:56:07.034771 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:07.034752    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-6a914e8f-kserve-8c7fb8749-fw2lj"
Oct 30 13:56:07.052546 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:07.052519    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-6a914e8f-kserve-8c7fb8749-fw2lj"]
Oct 30 13:56:07.062098 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:07.062064    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-6a914e8f-kserve-self-signed-certs\""
Oct 30 13:56:07.091023 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:07.090990    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b5764a3-bf33-4184-9641-f3d182eda37d-tls-certs\") pod \"e2e-same-id-b-6a914e8f-kserve-8c7fb8749-fw2lj\" (UID: \"2b5764a3-bf33-4184-9641-f3d182eda37d\") " pod="llm/e2e-same-id-b-6a914e8f-kserve-8c7fb8749-fw2lj"
Oct 30 13:56:07.091178 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:07.091030    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b5764a3-bf33-4184-9641-f3d182eda37d-tmp-dir\") pod \"e2e-same-id-b-6a914e8f-kserve-8c7fb8749-fw2lj\" (UID: \"2b5764a3-bf33-4184-9641-f3d182eda37d\") " pod="llm/e2e-same-id-b-6a914e8f-kserve-8c7fb8749-fw2lj"
Oct 30 13:56:07.091178 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:07.091146    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk54h\" (UniqueName: \"kubernetes.io/projected/2b5764a3-bf33-4184-9641-f3d182eda37d-kube-api-access-hk54h\") pod \"e2e-same-id-b-6a914e8f-kserve-8c7fb8749-fw2lj\" (UID: \"2b5764a3-bf33-4184-9641-f3d182eda37d\") " pod="llm/e2e-same-id-b-6a914e8f-kserve-8c7fb8749-fw2lj"
Oct 30 13:56:07.091262 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:07.091185    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b5764a3-bf33-4184-9641-f3d182eda37d-model-cache\") pod \"e2e-same-id-b-6a914e8f-kserve-8c7fb8749-fw2lj\" (UID: \"2b5764a3-bf33-4184-9641-f3d182eda37d\") " pod="llm/e2e-same-id-b-6a914e8f-kserve-8c7fb8749-fw2lj"
Oct 30 13:56:07.091331 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:07.091313    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b5764a3-bf33-4184-9641-f3d182eda37d-home\") pod \"e2e-same-id-b-6a914e8f-kserve-8c7fb8749-fw2lj\" (UID: \"2b5764a3-bf33-4184-9641-f3d182eda37d\") " pod="llm/e2e-same-id-b-6a914e8f-kserve-8c7fb8749-fw2lj"
Oct 30 13:56:07.091452 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:07.091431    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b5764a3-bf33-4184-9641-f3d182eda37d-dshm\") pod \"e2e-same-id-b-6a914e8f-kserve-8c7fb8749-fw2lj\" (UID: \"2b5764a3-bf33-4184-9641-f3d182eda37d\") " pod="llm/e2e-same-id-b-6a914e8f-kserve-8c7fb8749-fw2lj"
Oct 30 13:56:07.192204 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:07.192173    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b5764a3-bf33-4184-9641-f3d182eda37d-dshm\") pod \"e2e-same-id-b-6a914e8f-kserve-8c7fb8749-fw2lj\" (UID: \"2b5764a3-bf33-4184-9641-f3d182eda37d\") " pod="llm/e2e-same-id-b-6a914e8f-kserve-8c7fb8749-fw2lj"
Oct 30 13:56:07.192391 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:07.192212    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b5764a3-bf33-4184-9641-f3d182eda37d-tls-certs\") pod \"e2e-same-id-b-6a914e8f-kserve-8c7fb8749-fw2lj\" (UID: \"2b5764a3-bf33-4184-9641-f3d182eda37d\") " pod="llm/e2e-same-id-b-6a914e8f-kserve-8c7fb8749-fw2lj"
Oct 30 13:56:07.192391 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:07.192231    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b5764a3-bf33-4184-9641-f3d182eda37d-tmp-dir\") pod \"e2e-same-id-b-6a914e8f-kserve-8c7fb8749-fw2lj\" (UID: \"2b5764a3-bf33-4184-9641-f3d182eda37d\") " pod="llm/e2e-same-id-b-6a914e8f-kserve-8c7fb8749-fw2lj"
Oct 30 13:56:07.192391 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:07.192267    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk54h\" (UniqueName: \"kubernetes.io/projected/2b5764a3-bf33-4184-9641-f3d182eda37d-kube-api-access-hk54h\") pod \"e2e-same-id-b-6a914e8f-kserve-8c7fb8749-fw2lj\" (UID: \"2b5764a3-bf33-4184-9641-f3d182eda37d\") " pod="llm/e2e-same-id-b-6a914e8f-kserve-8c7fb8749-fw2lj"
Oct 30 13:56:07.192391 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:07.192287    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b5764a3-bf33-4184-9641-f3d182eda37d-model-cache\") pod \"e2e-same-id-b-6a914e8f-kserve-8c7fb8749-fw2lj\" (UID: \"2b5764a3-bf33-4184-9641-f3d182eda37d\") " pod="llm/e2e-same-id-b-6a914e8f-kserve-8c7fb8749-fw2lj"
Oct 30 13:56:07.192391 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:07.192338    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b5764a3-bf33-4184-9641-f3d182eda37d-home\") pod \"e2e-same-id-b-6a914e8f-kserve-8c7fb8749-fw2lj\" (UID: \"2b5764a3-bf33-4184-9641-f3d182eda37d\") " pod="llm/e2e-same-id-b-6a914e8f-kserve-8c7fb8749-fw2lj"
Oct 30 13:56:07.192769 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:07.192747    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b5764a3-bf33-4184-9641-f3d182eda37d-tmp-dir\") pod \"e2e-same-id-b-6a914e8f-kserve-8c7fb8749-fw2lj\" (UID: \"2b5764a3-bf33-4184-9641-f3d182eda37d\") " pod="llm/e2e-same-id-b-6a914e8f-kserve-8c7fb8749-fw2lj"
Oct 30 13:56:07.192832 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:07.192790    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b5764a3-bf33-4184-9641-f3d182eda37d-model-cache\") pod \"e2e-same-id-b-6a914e8f-kserve-8c7fb8749-fw2lj\" (UID: \"2b5764a3-bf33-4184-9641-f3d182eda37d\") " pod="llm/e2e-same-id-b-6a914e8f-kserve-8c7fb8749-fw2lj"
Oct 30 13:56:07.192832 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:07.192803    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b5764a3-bf33-4184-9641-f3d182eda37d-home\") pod \"e2e-same-id-b-6a914e8f-kserve-8c7fb8749-fw2lj\" (UID: \"2b5764a3-bf33-4184-9641-f3d182eda37d\") " pod="llm/e2e-same-id-b-6a914e8f-kserve-8c7fb8749-fw2lj"
Oct 30 13:56:07.194449 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:07.194428    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b5764a3-bf33-4184-9641-f3d182eda37d-dshm\") pod \"e2e-same-id-b-6a914e8f-kserve-8c7fb8749-fw2lj\" (UID: \"2b5764a3-bf33-4184-9641-f3d182eda37d\") " pod="llm/e2e-same-id-b-6a914e8f-kserve-8c7fb8749-fw2lj"
Oct 30 13:56:07.194745 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:07.194726    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b5764a3-bf33-4184-9641-f3d182eda37d-tls-certs\") pod \"e2e-same-id-b-6a914e8f-kserve-8c7fb8749-fw2lj\" (UID: \"2b5764a3-bf33-4184-9641-f3d182eda37d\") " pod="llm/e2e-same-id-b-6a914e8f-kserve-8c7fb8749-fw2lj"
Oct 30 13:56:07.215174 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:07.215150    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk54h\" (UniqueName: \"kubernetes.io/projected/2b5764a3-bf33-4184-9641-f3d182eda37d-kube-api-access-hk54h\") pod \"e2e-same-id-b-6a914e8f-kserve-8c7fb8749-fw2lj\" (UID: \"2b5764a3-bf33-4184-9641-f3d182eda37d\") " pod="llm/e2e-same-id-b-6a914e8f-kserve-8c7fb8749-fw2lj"
Oct 30 13:56:07.347815 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:07.347726    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-6a914e8f-kserve-8c7fb8749-fw2lj"
Oct 30 13:56:07.650662 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:07.650579    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-6a914e8f-kserve-8c7fb8749-fw2lj"]
Oct 30 13:56:07.652925 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:56:07.652894    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2b5764a3_bf33_4184_9641_f3d182eda37d.slice/crio-2b40bc5aa0be0ceb7c865c0e6988e5aba4af2fbe1591c8c67f5e2adfcbc99283 WatchSource:0}: Error finding container 2b40bc5aa0be0ceb7c865c0e6988e5aba4af2fbe1591c8c67f5e2adfcbc99283: Status 404 returned error can't find the container with id 2b40bc5aa0be0ceb7c865c0e6988e5aba4af2fbe1591c8c67f5e2adfcbc99283
Oct 30 13:56:08.066910 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:08.066811    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-6a914e8f-kserve-8c7fb8749-fw2lj" event={"ID":"2b5764a3-bf33-4184-9641-f3d182eda37d","Type":"ContainerStarted","Data":"f09465eb5bba482f70d7bce8d98f0fc819467d90b471c945ee58bb33570102d9"}
Oct 30 13:56:08.066910 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:08.066876    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-6a914e8f-kserve-8c7fb8749-fw2lj" event={"ID":"2b5764a3-bf33-4184-9641-f3d182eda37d","Type":"ContainerStarted","Data":"2b40bc5aa0be0ceb7c865c0e6988e5aba4af2fbe1591c8c67f5e2adfcbc99283"}
Oct 30 13:56:08.109146 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:08.109068    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-6a914e8f-kserve-8c7fb8749-fw2lj" podStartSLOduration=1.109054332 podStartE2EDuration="1.109054332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:56:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:56:08.10770481 +0000 UTC m=+3867.274069457" watchObservedRunningTime="2026-10-30 13:56:08.109054332 +0000 UTC m=+3867.275418974"
Oct 30 13:56:08.348722 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:08.348636    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-6a914e8f-kserve-8c7fb8749-fw2lj"
Oct 30 13:56:09.552818 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:09.552781    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:56:11.606388 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:56:11.606307    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:56:17.348577 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:17.348541    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-6a914e8f-kserve-8c7fb8749-fw2lj"
Oct 30 13:56:17.361573 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:17.361544    4364 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-6a914e8f-kserve-8c7fb8749-fw2lj"
Oct 30 13:56:18.180291 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:18.180256    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-6a914e8f-kserve-8c7fb8749-fw2lj"
Oct 30 13:56:30.065556 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:30.065521    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:56:31.096574 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:31.096536    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-fvl2n"]
Oct 30 13:56:31.097909 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:31.097880    4364 cpu_manager.go:414] "containerMap: removing container" podUID="03a318c2-ba1f-4c34-a2f4-8b258f71eee3" containerName="istio-proxy"
Oct 30 13:56:31.098072 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:31.098056    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03a318c2-ba1f-4c34-a2f4-8b258f71eee3" containerName="istio-proxy"
Oct 30 13:56:31.102175 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:31.102152    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-fvl2n"
Oct 30 13:56:31.135492 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:31.135458    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-fvl2n"]
Oct 30 13:56:31.137240 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:31.136696    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\""
Oct 30 13:56:31.137240 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:31.137024    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:56:31.151124 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:31.151097    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:56:31.151490 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:31.151470    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:56:31.195243 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:31.195197    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n56t6\" (UniqueName: \"kubernetes.io/projected/e973b6e6-56ba-416a-9a1f-bbece296188e-kube-api-access-n56t6\") pod \"maas-api-revoke-sub-452b6d7b-fvl2n\" (UID: \"e973b6e6-56ba-416a-9a1f-bbece296188e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-fvl2n"
Oct 30 13:56:31.195470 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:31.195375    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e973b6e6-56ba-416a-9a1f-bbece296188e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-fvl2n\" (UID: \"e973b6e6-56ba-416a-9a1f-bbece296188e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-fvl2n"
Oct 30 13:56:31.195557 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:31.195482    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e973b6e6-56ba-416a-9a1f-bbece296188e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-fvl2n\" (UID: \"e973b6e6-56ba-416a-9a1f-bbece296188e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-fvl2n"
Oct 30 13:56:31.296241 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:31.296192    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e973b6e6-56ba-416a-9a1f-bbece296188e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-fvl2n\" (UID: \"e973b6e6-56ba-416a-9a1f-bbece296188e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-fvl2n"
Oct 30 13:56:31.296498 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:31.296298    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n56t6\" (UniqueName: \"kubernetes.io/projected/e973b6e6-56ba-416a-9a1f-bbece296188e-kube-api-access-n56t6\") pod \"maas-api-revoke-sub-452b6d7b-fvl2n\" (UID: \"e973b6e6-56ba-416a-9a1f-bbece296188e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-fvl2n"
Oct 30 13:56:31.296498 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:31.296403    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e973b6e6-56ba-416a-9a1f-bbece296188e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-fvl2n\" (UID: \"e973b6e6-56ba-416a-9a1f-bbece296188e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-fvl2n"
Oct 30 13:56:31.297166 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:31.297119    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e973b6e6-56ba-416a-9a1f-bbece296188e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-fvl2n\" (UID: \"e973b6e6-56ba-416a-9a1f-bbece296188e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-fvl2n"
Oct 30 13:56:31.299629 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:31.299584    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e973b6e6-56ba-416a-9a1f-bbece296188e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-fvl2n\" (UID: \"e973b6e6-56ba-416a-9a1f-bbece296188e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-fvl2n"
Oct 30 13:56:31.327904 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:31.327865    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n56t6\" (UniqueName: \"kubernetes.io/projected/e973b6e6-56ba-416a-9a1f-bbece296188e-kube-api-access-n56t6\") pod \"maas-api-revoke-sub-452b6d7b-fvl2n\" (UID: \"e973b6e6-56ba-416a-9a1f-bbece296188e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-fvl2n"
Oct 30 13:56:31.437107 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:31.437060    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-fvl2n"
Oct 30 13:56:31.801619 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:31.800350    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-fvl2n"]
Oct 30 13:56:32.258336 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:32.258295    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-fvl2n" event={"ID":"e973b6e6-56ba-416a-9a1f-bbece296188e","Type":"ContainerStarted","Data":"af120ec9c0a801844a5e34cedb0f918fcd3be759a92c4731976722f5dd633387"}
Oct 30 13:56:33.288470 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:33.287971    4364 generic.go:356] "Generic (PLEG): container finished" podID="e973b6e6-56ba-416a-9a1f-bbece296188e" containerID="ce1ef01dc37a66fc06fbc59b7ca4a3600201ec58f8b4b9441112b6cb85d6d3d7" exitCode=0
Oct 30 13:56:33.288470 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:33.288097    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-fvl2n" event={"ID":"e973b6e6-56ba-416a-9a1f-bbece296188e","Type":"ContainerDied","Data":"ce1ef01dc37a66fc06fbc59b7ca4a3600201ec58f8b4b9441112b6cb85d6d3d7"}
Oct 30 13:56:34.437555 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:34.437528    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-fvl2n"
Oct 30 13:56:34.593382 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:34.593293    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e973b6e6-56ba-416a-9a1f-bbece296188e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e973b6e6-56ba-416a-9a1f-bbece296188e-maas-api-openshift-ca\") pod \"e973b6e6-56ba-416a-9a1f-bbece296188e\" (UID: \"e973b6e6-56ba-416a-9a1f-bbece296188e\") "
Oct 30 13:56:34.593382 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:34.593353    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e973b6e6-56ba-416a-9a1f-bbece296188e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e973b6e6-56ba-416a-9a1f-bbece296188e-maas-api-xks-ca\") pod \"e973b6e6-56ba-416a-9a1f-bbece296188e\" (UID: \"e973b6e6-56ba-416a-9a1f-bbece296188e\") "
Oct 30 13:56:34.593663 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:34.593435    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e973b6e6-56ba-416a-9a1f-bbece296188e-kube-api-access-n56t6\" (UniqueName: \"kubernetes.io/projected/e973b6e6-56ba-416a-9a1f-bbece296188e-kube-api-access-n56t6\") pod \"e973b6e6-56ba-416a-9a1f-bbece296188e\" (UID: \"e973b6e6-56ba-416a-9a1f-bbece296188e\") "
Oct 30 13:56:34.593726 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:34.593665    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e973b6e6-56ba-416a-9a1f-bbece296188e-maas-api-openshift-ca" pod "e973b6e6-56ba-416a-9a1f-bbece296188e" (UID: "e973b6e6-56ba-416a-9a1f-bbece296188e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:56:34.593830 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:34.593816    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e973b6e6-56ba-416a-9a1f-bbece296188e-maas-api-openshift-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:56:34.595406 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:34.595385    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e973b6e6-56ba-416a-9a1f-bbece296188e-maas-api-xks-ca" pod "e973b6e6-56ba-416a-9a1f-bbece296188e" (UID: "e973b6e6-56ba-416a-9a1f-bbece296188e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:56:34.595478 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:34.595459    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e973b6e6-56ba-416a-9a1f-bbece296188e-kube-api-access-n56t6" pod "e973b6e6-56ba-416a-9a1f-bbece296188e" (UID: "e973b6e6-56ba-416a-9a1f-bbece296188e"). InnerVolumeSpecName "kube-api-access-n56t6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:56:34.694674 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:34.694645    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e973b6e6-56ba-416a-9a1f-bbece296188e-maas-api-xks-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:56:34.694674 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:34.694672    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n56t6\" (UniqueName: \"kubernetes.io/projected/e973b6e6-56ba-416a-9a1f-bbece296188e-kube-api-access-n56t6\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:56:35.299908 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:35.299826    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-fvl2n"
Oct 30 13:56:35.299908 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:35.299847    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-fvl2n" event={"ID":"e973b6e6-56ba-416a-9a1f-bbece296188e","Type":"ContainerDied","Data":"af120ec9c0a801844a5e34cedb0f918fcd3be759a92c4731976722f5dd633387"}
Oct 30 13:56:35.299908 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:35.299901    4364 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af120ec9c0a801844a5e34cedb0f918fcd3be759a92c4731976722f5dd633387"
Oct 30 13:56:41.504004 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:56:41.503962    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:56:42.379581 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:42.379544    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-6a914e8f-kserve-7489c55769-mzrxr"]
Oct 30 13:56:44.481849 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:44.481808    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:56:44.858657 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:44.858573    4364 scope.go:122] "RemoveContainer" containerID="e79f39a8d0d2dc6b5bfdbe7ebe965f7c58833e47f97d89ec6c481e98cee77158"
Oct 30 13:56:44.867888 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:44.867868    4364 scope.go:122] "RemoveContainer" containerID="abe532b80e4fa58c3f136370102ba5bc8e88164d333f9ef0cb5880ef4e26749e"
Oct 30 13:56:48.195686 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:48.195644    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-6a914e8f-kserve-8c7fb8749-fw2lj"]
Oct 30 13:56:50.763252 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:50.763215    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:56:51.818157 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:51.818109    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-mx6gc"]
Oct 30 13:56:51.818767 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:51.818746    4364 cpu_manager.go:414] "containerMap: removing container" podUID="e973b6e6-56ba-416a-9a1f-bbece296188e" containerName="revoke-keys"
Oct 30 13:56:51.818866 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:51.818769    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e973b6e6-56ba-416a-9a1f-bbece296188e" containerName="revoke-keys"
Oct 30 13:56:51.818941 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:51.818918    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="e973b6e6-56ba-416a-9a1f-bbece296188e" containerName="revoke-keys"
Oct 30 13:56:51.822516 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:51.822491    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-mx6gc"
Oct 30 13:56:51.840833 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:51.840806    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-mx6gc"]
Oct 30 13:56:51.848433 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:51.848407    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:56:51.848736 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:51.848685    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:56:51.848903 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:51.848699    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\""
Oct 30 13:56:51.852592 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:51.851856    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:56:51.925777 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:51.925714    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c97ab5f-97f9-4d5a-b59a-a7abdbff75c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-mx6gc\" (UID: \"4c97ab5f-97f9-4d5a-b59a-a7abdbff75c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-mx6gc"
Oct 30 13:56:51.926004 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:51.925881    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l62xm\" (UniqueName: \"kubernetes.io/projected/4c97ab5f-97f9-4d5a-b59a-a7abdbff75c5-kube-api-access-l62xm\") pod \"maas-api-revoke-sub-3e31a506-mx6gc\" (UID: \"4c97ab5f-97f9-4d5a-b59a-a7abdbff75c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-mx6gc"
Oct 30 13:56:51.926004 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:51.925940    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c97ab5f-97f9-4d5a-b59a-a7abdbff75c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-mx6gc\" (UID: \"4c97ab5f-97f9-4d5a-b59a-a7abdbff75c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-mx6gc"
Oct 30 13:56:52.031335 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:52.031290    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c97ab5f-97f9-4d5a-b59a-a7abdbff75c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-mx6gc\" (UID: \"4c97ab5f-97f9-4d5a-b59a-a7abdbff75c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-mx6gc"
Oct 30 13:56:52.031689 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:52.031664    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l62xm\" (UniqueName: \"kubernetes.io/projected/4c97ab5f-97f9-4d5a-b59a-a7abdbff75c5-kube-api-access-l62xm\") pod \"maas-api-revoke-sub-3e31a506-mx6gc\" (UID: \"4c97ab5f-97f9-4d5a-b59a-a7abdbff75c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-mx6gc"
Oct 30 13:56:52.032618 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:52.032579    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c97ab5f-97f9-4d5a-b59a-a7abdbff75c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-mx6gc\" (UID: \"4c97ab5f-97f9-4d5a-b59a-a7abdbff75c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-mx6gc"
Oct 30 13:56:52.033576 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:52.033478    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c97ab5f-97f9-4d5a-b59a-a7abdbff75c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-mx6gc\" (UID: \"4c97ab5f-97f9-4d5a-b59a-a7abdbff75c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-mx6gc"
Oct 30 13:56:52.052136 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:52.052104    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c97ab5f-97f9-4d5a-b59a-a7abdbff75c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-mx6gc\" (UID: \"4c97ab5f-97f9-4d5a-b59a-a7abdbff75c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-mx6gc"
Oct 30 13:56:52.057339 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:52.057307    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l62xm\" (UniqueName: \"kubernetes.io/projected/4c97ab5f-97f9-4d5a-b59a-a7abdbff75c5-kube-api-access-l62xm\") pod \"maas-api-revoke-sub-3e31a506-mx6gc\" (UID: \"4c97ab5f-97f9-4d5a-b59a-a7abdbff75c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-mx6gc"
Oct 30 13:56:52.150621 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:52.150262    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-mx6gc"
Oct 30 13:56:53.081250 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:53.081207    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-mx6gc"]
Oct 30 13:56:53.142792 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:56:53.142732    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c97ab5f_97f9_4d5a_b59a_a7abdbff75c5.slice/crio-9e8f117bd086e1be1f7960920bad2d98d0a4b609c75b6bf2f1e917bb40574746 WatchSource:0}: Error finding container 9e8f117bd086e1be1f7960920bad2d98d0a4b609c75b6bf2f1e917bb40574746: Status 404 returned error can't find the container with id 9e8f117bd086e1be1f7960920bad2d98d0a4b609c75b6bf2f1e917bb40574746
Oct 30 13:56:53.479734 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:53.479646    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-mx6gc" event={"ID":"4c97ab5f-97f9-4d5a-b59a-a7abdbff75c5","Type":"ContainerStarted","Data":"9e8f117bd086e1be1f7960920bad2d98d0a4b609c75b6bf2f1e917bb40574746"}
Oct 30 13:56:54.486667 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:54.486567    4364 generic.go:356] "Generic (PLEG): container finished" podID="4c97ab5f-97f9-4d5a-b59a-a7abdbff75c5" containerID="90d85e1cd1a6cbf46696d535c2fa9e9f3205945dddb43306fec76e8a9a760428" exitCode=0
Oct 30 13:56:54.486667 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:54.486628    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-mx6gc" event={"ID":"4c97ab5f-97f9-4d5a-b59a-a7abdbff75c5","Type":"ContainerDied","Data":"90d85e1cd1a6cbf46696d535c2fa9e9f3205945dddb43306fec76e8a9a760428"}
Oct 30 13:56:55.629898 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:55.629866    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-mx6gc"
Oct 30 13:56:55.719690 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:55.719658    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c97ab5f-97f9-4d5a-b59a-a7abdbff75c5-kube-api-access-l62xm\" (UniqueName: \"kubernetes.io/projected/4c97ab5f-97f9-4d5a-b59a-a7abdbff75c5-kube-api-access-l62xm\") pod \"4c97ab5f-97f9-4d5a-b59a-a7abdbff75c5\" (UID: \"4c97ab5f-97f9-4d5a-b59a-a7abdbff75c5\") "
Oct 30 13:56:55.719865 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:55.719747    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c97ab5f-97f9-4d5a-b59a-a7abdbff75c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c97ab5f-97f9-4d5a-b59a-a7abdbff75c5-maas-api-openshift-ca\") pod \"4c97ab5f-97f9-4d5a-b59a-a7abdbff75c5\" (UID: \"4c97ab5f-97f9-4d5a-b59a-a7abdbff75c5\") "
Oct 30 13:56:55.719865 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:55.719768    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c97ab5f-97f9-4d5a-b59a-a7abdbff75c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c97ab5f-97f9-4d5a-b59a-a7abdbff75c5-maas-api-xks-ca\") pod \"4c97ab5f-97f9-4d5a-b59a-a7abdbff75c5\" (UID: \"4c97ab5f-97f9-4d5a-b59a-a7abdbff75c5\") "
Oct 30 13:56:55.720107 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:55.720081    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c97ab5f-97f9-4d5a-b59a-a7abdbff75c5-maas-api-openshift-ca" pod "4c97ab5f-97f9-4d5a-b59a-a7abdbff75c5" (UID: "4c97ab5f-97f9-4d5a-b59a-a7abdbff75c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:56:55.721772 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:55.721753    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c97ab5f-97f9-4d5a-b59a-a7abdbff75c5-maas-api-xks-ca" pod "4c97ab5f-97f9-4d5a-b59a-a7abdbff75c5" (UID: "4c97ab5f-97f9-4d5a-b59a-a7abdbff75c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:56:55.721874 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:55.721847    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c97ab5f-97f9-4d5a-b59a-a7abdbff75c5-kube-api-access-l62xm" pod "4c97ab5f-97f9-4d5a-b59a-a7abdbff75c5" (UID: "4c97ab5f-97f9-4d5a-b59a-a7abdbff75c5"). InnerVolumeSpecName "kube-api-access-l62xm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:56:55.821072 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:55.820984    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c97ab5f-97f9-4d5a-b59a-a7abdbff75c5-maas-api-openshift-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:56:55.821072 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:55.821018    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c97ab5f-97f9-4d5a-b59a-a7abdbff75c5-maas-api-xks-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:56:55.821072 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:55.821032    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l62xm\" (UniqueName: \"kubernetes.io/projected/4c97ab5f-97f9-4d5a-b59a-a7abdbff75c5-kube-api-access-l62xm\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:56:56.499122 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:56.499084    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-mx6gc" event={"ID":"4c97ab5f-97f9-4d5a-b59a-a7abdbff75c5","Type":"ContainerDied","Data":"9e8f117bd086e1be1f7960920bad2d98d0a4b609c75b6bf2f1e917bb40574746"}
Oct 30 13:56:56.499122 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:56.499105    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-mx6gc"
Oct 30 13:56:56.499394 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:56.499143    4364 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e8f117bd086e1be1f7960920bad2d98d0a4b609c75b6bf2f1e917bb40574746"
Oct 30 13:56:57.399926 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:57.399881    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-6a914e8f-kserve-7489c55769-mzrxr" podUID="d6e05360-004c-49e4-8359-7d56f2ba6b71" containerName="main" containerID="cri-o://14c65847688573ddb25d4c75f97c4499bda17c917fc1d13e88d5ccc52c76f63a" gracePeriod=45
Oct 30 13:56:57.651198 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:57.651140    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-6a914e8f-kserve-7489c55769-mzrxr"
Oct 30 13:56:57.738369 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:57.738331    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6e05360-004c-49e4-8359-7d56f2ba6b71-tls-certs\" (UniqueName: \"kubernetes.io/secret/d6e05360-004c-49e4-8359-7d56f2ba6b71-tls-certs\") pod \"d6e05360-004c-49e4-8359-7d56f2ba6b71\" (UID: \"d6e05360-004c-49e4-8359-7d56f2ba6b71\") "
Oct 30 13:56:57.738531 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:57.738378    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6e05360-004c-49e4-8359-7d56f2ba6b71-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6e05360-004c-49e4-8359-7d56f2ba6b71-tmp-dir\") pod \"d6e05360-004c-49e4-8359-7d56f2ba6b71\" (UID: \"d6e05360-004c-49e4-8359-7d56f2ba6b71\") "
Oct 30 13:56:57.738531 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:57.738403    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6e05360-004c-49e4-8359-7d56f2ba6b71-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6e05360-004c-49e4-8359-7d56f2ba6b71-dshm\") pod \"d6e05360-004c-49e4-8359-7d56f2ba6b71\" (UID: \"d6e05360-004c-49e4-8359-7d56f2ba6b71\") "
Oct 30 13:56:57.738531 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:57.738456    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6e05360-004c-49e4-8359-7d56f2ba6b71-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6e05360-004c-49e4-8359-7d56f2ba6b71-model-cache\") pod \"d6e05360-004c-49e4-8359-7d56f2ba6b71\" (UID: \"d6e05360-004c-49e4-8359-7d56f2ba6b71\") "
Oct 30 13:56:57.738531 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:57.738507    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6e05360-004c-49e4-8359-7d56f2ba6b71-kube-api-access-wwvq8\" (UniqueName: \"kubernetes.io/projected/d6e05360-004c-49e4-8359-7d56f2ba6b71-kube-api-access-wwvq8\") pod \"d6e05360-004c-49e4-8359-7d56f2ba6b71\" (UID: \"d6e05360-004c-49e4-8359-7d56f2ba6b71\") "
Oct 30 13:56:57.738819 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:57.738533    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6e05360-004c-49e4-8359-7d56f2ba6b71-home\" (UniqueName: \"kubernetes.io/empty-dir/d6e05360-004c-49e4-8359-7d56f2ba6b71-home\") pod \"d6e05360-004c-49e4-8359-7d56f2ba6b71\" (UID: \"d6e05360-004c-49e4-8359-7d56f2ba6b71\") "
Oct 30 13:56:57.738819 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:57.738703    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6e05360-004c-49e4-8359-7d56f2ba6b71-tmp-dir" pod "d6e05360-004c-49e4-8359-7d56f2ba6b71" (UID: "d6e05360-004c-49e4-8359-7d56f2ba6b71"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:56:57.738819 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:57.738802    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6e05360-004c-49e4-8359-7d56f2ba6b71-model-cache" pod "d6e05360-004c-49e4-8359-7d56f2ba6b71" (UID: "d6e05360-004c-49e4-8359-7d56f2ba6b71"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:56:57.738962 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:57.738848    4364 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6e05360-004c-49e4-8359-7d56f2ba6b71-tmp-dir\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:56:57.738962 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:57.738910    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6e05360-004c-49e4-8359-7d56f2ba6b71-home" pod "d6e05360-004c-49e4-8359-7d56f2ba6b71" (UID: "d6e05360-004c-49e4-8359-7d56f2ba6b71"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:56:57.740765 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:57.740743    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6e05360-004c-49e4-8359-7d56f2ba6b71-dshm" pod "d6e05360-004c-49e4-8359-7d56f2ba6b71" (UID: "d6e05360-004c-49e4-8359-7d56f2ba6b71"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:56:57.740846 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:57.740758    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6e05360-004c-49e4-8359-7d56f2ba6b71-tls-certs" pod "d6e05360-004c-49e4-8359-7d56f2ba6b71" (UID: "d6e05360-004c-49e4-8359-7d56f2ba6b71"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:56:57.740846 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:57.740767    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6e05360-004c-49e4-8359-7d56f2ba6b71-kube-api-access-wwvq8" pod "d6e05360-004c-49e4-8359-7d56f2ba6b71" (UID: "d6e05360-004c-49e4-8359-7d56f2ba6b71"). InnerVolumeSpecName "kube-api-access-wwvq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:56:57.839484 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:57.839450    4364 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6e05360-004c-49e4-8359-7d56f2ba6b71-tls-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:56:57.839484 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:57.839481    4364 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6e05360-004c-49e4-8359-7d56f2ba6b71-dshm\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:56:57.839484 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:57.839490    4364 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6e05360-004c-49e4-8359-7d56f2ba6b71-model-cache\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:56:57.839762 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:57.839500    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwvq8\" (UniqueName: \"kubernetes.io/projected/d6e05360-004c-49e4-8359-7d56f2ba6b71-kube-api-access-wwvq8\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:56:57.839762 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:57.839509    4364 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6e05360-004c-49e4-8359-7d56f2ba6b71-home\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:56:58.514122 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:58.514087    4364 generic.go:356] "Generic (PLEG): container finished" podID="d6e05360-004c-49e4-8359-7d56f2ba6b71" containerID="14c65847688573ddb25d4c75f97c4499bda17c917fc1d13e88d5ccc52c76f63a" exitCode=0
Oct 30 13:56:58.514585 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:58.514136    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-6a914e8f-kserve-7489c55769-mzrxr"
Oct 30 13:56:58.514585 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:58.514164    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-6a914e8f-kserve-7489c55769-mzrxr" event={"ID":"d6e05360-004c-49e4-8359-7d56f2ba6b71","Type":"ContainerDied","Data":"14c65847688573ddb25d4c75f97c4499bda17c917fc1d13e88d5ccc52c76f63a"}
Oct 30 13:56:58.514585 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:58.514210    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-6a914e8f-kserve-7489c55769-mzrxr" event={"ID":"d6e05360-004c-49e4-8359-7d56f2ba6b71","Type":"ContainerDied","Data":"379ab5cbdd2eeebc3d9cefbe2a6462b4be1b57dbbf84ae58edcadb994114ae72"}
Oct 30 13:56:58.514585 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:58.514233    4364 scope.go:122] "RemoveContainer" containerID="14c65847688573ddb25d4c75f97c4499bda17c917fc1d13e88d5ccc52c76f63a"
Oct 30 13:56:58.523784 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:58.523767    4364 scope.go:122] "RemoveContainer" containerID="14c65847688573ddb25d4c75f97c4499bda17c917fc1d13e88d5ccc52c76f63a"
Oct 30 13:56:58.524033 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:56:58.524014    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14c65847688573ddb25d4c75f97c4499bda17c917fc1d13e88d5ccc52c76f63a\": container with ID starting with 14c65847688573ddb25d4c75f97c4499bda17c917fc1d13e88d5ccc52c76f63a not found: ID does not exist" containerID="14c65847688573ddb25d4c75f97c4499bda17c917fc1d13e88d5ccc52c76f63a"
Oct 30 13:56:58.524100 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:58.524047    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14c65847688573ddb25d4c75f97c4499bda17c917fc1d13e88d5ccc52c76f63a"} err="failed to get container status \"14c65847688573ddb25d4c75f97c4499bda17c917fc1d13e88d5ccc52c76f63a\": rpc error: code = NotFound desc = could not find container \"14c65847688573ddb25d4c75f97c4499bda17c917fc1d13e88d5ccc52c76f63a\": container with ID starting with 14c65847688573ddb25d4c75f97c4499bda17c917fc1d13e88d5ccc52c76f63a not found: ID does not exist"
Oct 30 13:56:58.567640 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:58.567614    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-6a914e8f-kserve-7489c55769-mzrxr"]
Oct 30 13:56:58.579531 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:58.579510    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-6a914e8f-kserve-7489c55769-mzrxr"]
Oct 30 13:56:58.969169 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:56:58.969131    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6e05360-004c-49e4-8359-7d56f2ba6b71" path="/var/lib/kubelet/pods/d6e05360-004c-49e4-8359-7d56f2ba6b71/volumes"
Oct 30 13:57:02.992622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:02.992566    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:57:03.238061 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:03.238020    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-6a914e8f-kserve-8c7fb8749-fw2lj" podUID="2b5764a3-bf33-4184-9641-f3d182eda37d" containerName="main" containerID="cri-o://f09465eb5bba482f70d7bce8d98f0fc819467d90b471c945ee58bb33570102d9" gracePeriod=45
Oct 30 13:57:03.612834 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:03.612262    4364 generic.go:356] "Generic (PLEG): container finished" podID="2b5764a3-bf33-4184-9641-f3d182eda37d" containerID="f09465eb5bba482f70d7bce8d98f0fc819467d90b471c945ee58bb33570102d9" exitCode=0
Oct 30 13:57:03.612834 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:03.612436    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-6a914e8f-kserve-8c7fb8749-fw2lj" event={"ID":"2b5764a3-bf33-4184-9641-f3d182eda37d","Type":"ContainerDied","Data":"f09465eb5bba482f70d7bce8d98f0fc819467d90b471c945ee58bb33570102d9"}
Oct 30 13:57:03.667623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:03.664820    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-6a914e8f-kserve-8c7fb8749-fw2lj"
Oct 30 13:57:03.810622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:03.809962    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b5764a3-bf33-4184-9641-f3d182eda37d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b5764a3-bf33-4184-9641-f3d182eda37d-dshm\") pod \"2b5764a3-bf33-4184-9641-f3d182eda37d\" (UID: \"2b5764a3-bf33-4184-9641-f3d182eda37d\") "
Oct 30 13:57:03.810622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:03.810038    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b5764a3-bf33-4184-9641-f3d182eda37d-kube-api-access-hk54h\" (UniqueName: \"kubernetes.io/projected/2b5764a3-bf33-4184-9641-f3d182eda37d-kube-api-access-hk54h\") pod \"2b5764a3-bf33-4184-9641-f3d182eda37d\" (UID: \"2b5764a3-bf33-4184-9641-f3d182eda37d\") "
Oct 30 13:57:03.810622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:03.810087    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b5764a3-bf33-4184-9641-f3d182eda37d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b5764a3-bf33-4184-9641-f3d182eda37d-tmp-dir\") pod \"2b5764a3-bf33-4184-9641-f3d182eda37d\" (UID: \"2b5764a3-bf33-4184-9641-f3d182eda37d\") "
Oct 30 13:57:03.810622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:03.810205    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b5764a3-bf33-4184-9641-f3d182eda37d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b5764a3-bf33-4184-9641-f3d182eda37d-model-cache\") pod \"2b5764a3-bf33-4184-9641-f3d182eda37d\" (UID: \"2b5764a3-bf33-4184-9641-f3d182eda37d\") "
Oct 30 13:57:03.810622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:03.810277    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b5764a3-bf33-4184-9641-f3d182eda37d-tls-certs\" (UniqueName: \"kubernetes.io/secret/2b5764a3-bf33-4184-9641-f3d182eda37d-tls-certs\") pod \"2b5764a3-bf33-4184-9641-f3d182eda37d\" (UID: \"2b5764a3-bf33-4184-9641-f3d182eda37d\") "
Oct 30 13:57:03.810622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:03.810324    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b5764a3-bf33-4184-9641-f3d182eda37d-home\" (UniqueName: \"kubernetes.io/empty-dir/2b5764a3-bf33-4184-9641-f3d182eda37d-home\") pod \"2b5764a3-bf33-4184-9641-f3d182eda37d\" (UID: \"2b5764a3-bf33-4184-9641-f3d182eda37d\") "
Oct 30 13:57:03.811148 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:03.810937    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b5764a3-bf33-4184-9641-f3d182eda37d-home" pod "2b5764a3-bf33-4184-9641-f3d182eda37d" (UID: "2b5764a3-bf33-4184-9641-f3d182eda37d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:57:03.829761 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:03.811457    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b5764a3-bf33-4184-9641-f3d182eda37d-model-cache" pod "2b5764a3-bf33-4184-9641-f3d182eda37d" (UID: "2b5764a3-bf33-4184-9641-f3d182eda37d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:57:03.829761 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:03.813174    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b5764a3-bf33-4184-9641-f3d182eda37d-tmp-dir" pod "2b5764a3-bf33-4184-9641-f3d182eda37d" (UID: "2b5764a3-bf33-4184-9641-f3d182eda37d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:57:03.850274 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:03.850237    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b5764a3-bf33-4184-9641-f3d182eda37d-kube-api-access-hk54h" pod "2b5764a3-bf33-4184-9641-f3d182eda37d" (UID: "2b5764a3-bf33-4184-9641-f3d182eda37d"). InnerVolumeSpecName "kube-api-access-hk54h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:57:03.876583 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:03.876543    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b5764a3-bf33-4184-9641-f3d182eda37d-dshm" pod "2b5764a3-bf33-4184-9641-f3d182eda37d" (UID: "2b5764a3-bf33-4184-9641-f3d182eda37d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:57:03.877259 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:03.877234    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b5764a3-bf33-4184-9641-f3d182eda37d-tls-certs" pod "2b5764a3-bf33-4184-9641-f3d182eda37d" (UID: "2b5764a3-bf33-4184-9641-f3d182eda37d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:57:03.922622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:03.919207    4364 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b5764a3-bf33-4184-9641-f3d182eda37d-model-cache\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:57:03.945706 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:03.945673    4364 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b5764a3-bf33-4184-9641-f3d182eda37d-tls-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:57:03.945940 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:03.945923    4364 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b5764a3-bf33-4184-9641-f3d182eda37d-home\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:57:03.946053 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:03.946041    4364 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b5764a3-bf33-4184-9641-f3d182eda37d-dshm\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:57:03.946171 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:03.946157    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hk54h\" (UniqueName: \"kubernetes.io/projected/2b5764a3-bf33-4184-9641-f3d182eda37d-kube-api-access-hk54h\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:57:03.946287 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:03.946272    4364 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b5764a3-bf33-4184-9641-f3d182eda37d-tmp-dir\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:57:04.637978 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:04.637937    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-6a914e8f-kserve-8c7fb8749-fw2lj" event={"ID":"2b5764a3-bf33-4184-9641-f3d182eda37d","Type":"ContainerDied","Data":"2b40bc5aa0be0ceb7c865c0e6988e5aba4af2fbe1591c8c67f5e2adfcbc99283"}
Oct 30 13:57:04.638582 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:04.638560    4364 scope.go:122] "RemoveContainer" containerID="f09465eb5bba482f70d7bce8d98f0fc819467d90b471c945ee58bb33570102d9"
Oct 30 13:57:04.638878 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:04.638859    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-6a914e8f-kserve-8c7fb8749-fw2lj"
Oct 30 13:57:04.810623 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:04.807420    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-6a914e8f-kserve-8c7fb8749-fw2lj"]
Oct 30 13:57:04.838622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:04.837936    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-6a914e8f-kserve-8c7fb8749-fw2lj"]
Oct 30 13:57:04.997673 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:04.996802    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b5764a3-bf33-4184-9641-f3d182eda37d" path="/var/lib/kubelet/pods/2b5764a3-bf33-4184-9641-f3d182eda37d/volumes"
Oct 30 13:57:05.337882 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:05.337777    4364 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-wfkdb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:57:05.338243 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:05.338186    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-wfkdb" podUID="cac946c2-24d3-42b9-9d62-5f45833c81fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:57:07.077820 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:07.077784    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:57:11.504125 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:57:11.504086    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:57:12.182763 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:12.182724    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f929de75-j7xrt"]
Oct 30 13:57:12.183317 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:12.183297    4364 cpu_manager.go:414] "containerMap: removing container" podUID="4c97ab5f-97f9-4d5a-b59a-a7abdbff75c5" containerName="revoke-keys"
Oct 30 13:57:12.183379 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:12.183320    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c97ab5f-97f9-4d5a-b59a-a7abdbff75c5" containerName="revoke-keys"
Oct 30 13:57:12.183379 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:12.183340    4364 cpu_manager.go:414] "containerMap: removing container" podUID="2b5764a3-bf33-4184-9641-f3d182eda37d" containerName="main"
Oct 30 13:57:12.183379 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:12.183345    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b5764a3-bf33-4184-9641-f3d182eda37d" containerName="main"
Oct 30 13:57:12.183379 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:12.183361    4364 cpu_manager.go:414] "containerMap: removing container" podUID="d6e05360-004c-49e4-8359-7d56f2ba6b71" containerName="main"
Oct 30 13:57:12.183379 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:12.183367    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6e05360-004c-49e4-8359-7d56f2ba6b71" containerName="main"
Oct 30 13:57:12.183553 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:12.183421    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b5764a3-bf33-4184-9641-f3d182eda37d" containerName="main"
Oct 30 13:57:12.183553 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:12.183430    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6e05360-004c-49e4-8359-7d56f2ba6b71" containerName="main"
Oct 30 13:57:12.183553 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:12.183440    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c97ab5f-97f9-4d5a-b59a-a7abdbff75c5" containerName="revoke-keys"
Oct 30 13:57:12.188376 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:12.188352    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f929de75-j7xrt"
Oct 30 13:57:12.204812 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:12.204784    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f929de75-j7xrt"]
Oct 30 13:57:12.214075 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:12.214049    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:57:12.214276 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:12.214062    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:57:12.214276 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:12.214058    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:57:12.215655 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:12.215635    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\""
Oct 30 13:57:12.286765 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:12.286728    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b34c6967-3a1c-4e37-afe9-d21af16c5c4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f929de75-j7xrt\" (UID: \"b34c6967-3a1c-4e37-afe9-d21af16c5c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f929de75-j7xrt"
Oct 30 13:57:12.286969 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:12.286788    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b34c6967-3a1c-4e37-afe9-d21af16c5c4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f929de75-j7xrt\" (UID: \"b34c6967-3a1c-4e37-afe9-d21af16c5c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f929de75-j7xrt"
Oct 30 13:57:12.286969 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:12.286902    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74sk2\" (UniqueName: \"kubernetes.io/projected/b34c6967-3a1c-4e37-afe9-d21af16c5c4e-kube-api-access-74sk2\") pod \"maas-api-revoke-sub-f929de75-j7xrt\" (UID: \"b34c6967-3a1c-4e37-afe9-d21af16c5c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f929de75-j7xrt"
Oct 30 13:57:12.388063 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:12.388023    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b34c6967-3a1c-4e37-afe9-d21af16c5c4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f929de75-j7xrt\" (UID: \"b34c6967-3a1c-4e37-afe9-d21af16c5c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f929de75-j7xrt"
Oct 30 13:57:12.388231 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:12.388086    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b34c6967-3a1c-4e37-afe9-d21af16c5c4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f929de75-j7xrt\" (UID: \"b34c6967-3a1c-4e37-afe9-d21af16c5c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f929de75-j7xrt"
Oct 30 13:57:12.388231 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:12.388115    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74sk2\" (UniqueName: \"kubernetes.io/projected/b34c6967-3a1c-4e37-afe9-d21af16c5c4e-kube-api-access-74sk2\") pod \"maas-api-revoke-sub-f929de75-j7xrt\" (UID: \"b34c6967-3a1c-4e37-afe9-d21af16c5c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f929de75-j7xrt"
Oct 30 13:57:12.388738 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:12.388714    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b34c6967-3a1c-4e37-afe9-d21af16c5c4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f929de75-j7xrt\" (UID: \"b34c6967-3a1c-4e37-afe9-d21af16c5c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f929de75-j7xrt"
Oct 30 13:57:12.390466 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:12.390434    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b34c6967-3a1c-4e37-afe9-d21af16c5c4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f929de75-j7xrt\" (UID: \"b34c6967-3a1c-4e37-afe9-d21af16c5c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f929de75-j7xrt"
Oct 30 13:57:12.411429 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:12.411402    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74sk2\" (UniqueName: \"kubernetes.io/projected/b34c6967-3a1c-4e37-afe9-d21af16c5c4e-kube-api-access-74sk2\") pod \"maas-api-revoke-sub-f929de75-j7xrt\" (UID: \"b34c6967-3a1c-4e37-afe9-d21af16c5c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f929de75-j7xrt"
Oct 30 13:57:12.501490 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:12.501407    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f929de75-j7xrt"
Oct 30 13:57:12.802059 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:12.802029    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f929de75-j7xrt"]
Oct 30 13:57:12.804972 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:57:12.804947    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb34c6967_3a1c_4e37_afe9_d21af16c5c4e.slice/crio-ddd5e2bbb1143098be9ed901ae8f62e1b2c6f1b84357c8bd29ac861bdb016d61 WatchSource:0}: Error finding container ddd5e2bbb1143098be9ed901ae8f62e1b2c6f1b84357c8bd29ac861bdb016d61: Status 404 returned error can't find the container with id ddd5e2bbb1143098be9ed901ae8f62e1b2c6f1b84357c8bd29ac861bdb016d61
Oct 30 13:57:13.703001 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:13.702966    4364 generic.go:356] "Generic (PLEG): container finished" podID="b34c6967-3a1c-4e37-afe9-d21af16c5c4e" containerID="0010300c16e47f4b06bc0cdf77f22d52a7827965e8598285617683b93ae2e16b" exitCode=0
Oct 30 13:57:13.703147 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:13.703048    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f929de75-j7xrt" event={"ID":"b34c6967-3a1c-4e37-afe9-d21af16c5c4e","Type":"ContainerDied","Data":"0010300c16e47f4b06bc0cdf77f22d52a7827965e8598285617683b93ae2e16b"}
Oct 30 13:57:13.703147 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:13.703107    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f929de75-j7xrt" event={"ID":"b34c6967-3a1c-4e37-afe9-d21af16c5c4e","Type":"ContainerStarted","Data":"ddd5e2bbb1143098be9ed901ae8f62e1b2c6f1b84357c8bd29ac861bdb016d61"}
Oct 30 13:57:14.843961 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:14.843933    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f929de75-j7xrt"
Oct 30 13:57:14.915035 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:14.914996    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b34c6967-3a1c-4e37-afe9-d21af16c5c4e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b34c6967-3a1c-4e37-afe9-d21af16c5c4e-maas-api-xks-ca\") pod \"b34c6967-3a1c-4e37-afe9-d21af16c5c4e\" (UID: \"b34c6967-3a1c-4e37-afe9-d21af16c5c4e\") "
Oct 30 13:57:14.915035 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:14.915040    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b34c6967-3a1c-4e37-afe9-d21af16c5c4e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b34c6967-3a1c-4e37-afe9-d21af16c5c4e-maas-api-openshift-ca\") pod \"b34c6967-3a1c-4e37-afe9-d21af16c5c4e\" (UID: \"b34c6967-3a1c-4e37-afe9-d21af16c5c4e\") "
Oct 30 13:57:14.915363 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:14.915172    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b34c6967-3a1c-4e37-afe9-d21af16c5c4e-kube-api-access-74sk2\" (UniqueName: \"kubernetes.io/projected/b34c6967-3a1c-4e37-afe9-d21af16c5c4e-kube-api-access-74sk2\") pod \"b34c6967-3a1c-4e37-afe9-d21af16c5c4e\" (UID: \"b34c6967-3a1c-4e37-afe9-d21af16c5c4e\") "
Oct 30 13:57:14.915494 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:14.915470    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b34c6967-3a1c-4e37-afe9-d21af16c5c4e-maas-api-openshift-ca" pod "b34c6967-3a1c-4e37-afe9-d21af16c5c4e" (UID: "b34c6967-3a1c-4e37-afe9-d21af16c5c4e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:57:14.917138 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:14.917111    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b34c6967-3a1c-4e37-afe9-d21af16c5c4e-maas-api-xks-ca" pod "b34c6967-3a1c-4e37-afe9-d21af16c5c4e" (UID: "b34c6967-3a1c-4e37-afe9-d21af16c5c4e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:57:14.917259 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:14.917118    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b34c6967-3a1c-4e37-afe9-d21af16c5c4e-kube-api-access-74sk2" pod "b34c6967-3a1c-4e37-afe9-d21af16c5c4e" (UID: "b34c6967-3a1c-4e37-afe9-d21af16c5c4e"). InnerVolumeSpecName "kube-api-access-74sk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:57:15.016678 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:15.016651    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b34c6967-3a1c-4e37-afe9-d21af16c5c4e-maas-api-xks-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:57:15.016678 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:15.016678    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b34c6967-3a1c-4e37-afe9-d21af16c5c4e-maas-api-openshift-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:57:15.016850 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:15.016689    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74sk2\" (UniqueName: \"kubernetes.io/projected/b34c6967-3a1c-4e37-afe9-d21af16c5c4e-kube-api-access-74sk2\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:57:15.715722 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:15.715691    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f929de75-j7xrt"
Oct 30 13:57:15.715961 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:15.715683    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f929de75-j7xrt" event={"ID":"b34c6967-3a1c-4e37-afe9-d21af16c5c4e","Type":"ContainerDied","Data":"ddd5e2bbb1143098be9ed901ae8f62e1b2c6f1b84357c8bd29ac861bdb016d61"}
Oct 30 13:57:15.715961 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:15.715822    4364 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ddd5e2bbb1143098be9ed901ae8f62e1b2c6f1b84357c8bd29ac861bdb016d61"
Oct 30 13:57:20.046616 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:20.046563    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ps4mr"]
Oct 30 13:57:20.058271 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:20.058237    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ps4mr"]
Oct 30 13:57:20.968185 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:20.968154    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a7ba96d-104e-4933-b40d-13127dc2f8fb" path="/var/lib/kubelet/pods/7a7ba96d-104e-4933-b40d-13127dc2f8fb/volumes"
Oct 30 13:57:23.385328 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:23.385292    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:57:24.600830 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:24.600793    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-spd87"]
Oct 30 13:57:24.601239 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:24.601193    4364 cpu_manager.go:414] "containerMap: removing container" podUID="b34c6967-3a1c-4e37-afe9-d21af16c5c4e" containerName="revoke-keys"
Oct 30 13:57:24.601239 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:24.601205    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b34c6967-3a1c-4e37-afe9-d21af16c5c4e" containerName="revoke-keys"
Oct 30 13:57:24.601345 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:24.601281    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="b34c6967-3a1c-4e37-afe9-d21af16c5c4e" containerName="revoke-keys"
Oct 30 13:57:24.604540 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:24.604523    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-spd87"
Oct 30 13:57:24.621328 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:24.621304    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-4jkwp"]
Oct 30 13:57:24.624704 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:24.624686    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-4jkwp"
Oct 30 13:57:24.632891 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:24.632863    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-spd87"]
Oct 30 13:57:24.633059 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:24.632968    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-zw9vx\""
Oct 30 13:57:24.641164 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:24.641144    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:57:24.641333 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:24.641320    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:57:24.641387 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:24.641332    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 30 13:57:24.643374 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:24.643353    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 13:57:24.647802 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:24.647782    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-4jkwp"]
Oct 30 13:57:24.711548 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:24.711518    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fcd4eb6-58be-46a7-ac2c-70c876aa08ed-openshift-service-ca\") pod \"maas-api-595cc648c-4jkwp\" (UID: \"4fcd4eb6-58be-46a7-ac2c-70c876aa08ed\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-4jkwp"
Oct 30 13:57:24.711757 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:24.711566    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e615be51-c198-4c99-bf87-ba09654ea703-maas-api-tls\") pod \"maas-api-5c48877c75-spd87\" (UID: \"e615be51-c198-4c99-bf87-ba09654ea703\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-spd87"
Oct 30 13:57:24.711757 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:24.711586    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e615be51-c198-4c99-bf87-ba09654ea703-openshift-service-ca\") pod \"maas-api-5c48877c75-spd87\" (UID: \"e615be51-c198-4c99-bf87-ba09654ea703\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-spd87"
Oct 30 13:57:24.711909 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:24.711784    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e615be51-c198-4c99-bf87-ba09654ea703-metrics-certs\") pod \"maas-api-5c48877c75-spd87\" (UID: \"e615be51-c198-4c99-bf87-ba09654ea703\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-spd87"
Oct 30 13:57:24.711909 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:24.711818    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fcd4eb6-58be-46a7-ac2c-70c876aa08ed-metrics-certs\") pod \"maas-api-595cc648c-4jkwp\" (UID: \"4fcd4eb6-58be-46a7-ac2c-70c876aa08ed\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-4jkwp"
Oct 30 13:57:24.711909 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:24.711846    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqpkl\" (UniqueName: \"kubernetes.io/projected/4fcd4eb6-58be-46a7-ac2c-70c876aa08ed-kube-api-access-xqpkl\") pod \"maas-api-595cc648c-4jkwp\" (UID: \"4fcd4eb6-58be-46a7-ac2c-70c876aa08ed\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-4jkwp"
Oct 30 13:57:24.712093 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:24.711927    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fcd4eb6-58be-46a7-ac2c-70c876aa08ed-maas-api-tls\") pod \"maas-api-595cc648c-4jkwp\" (UID: \"4fcd4eb6-58be-46a7-ac2c-70c876aa08ed\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-4jkwp"
Oct 30 13:57:24.712093 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:24.711990    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkncp\" (UniqueName: \"kubernetes.io/projected/e615be51-c198-4c99-bf87-ba09654ea703-kube-api-access-fkncp\") pod \"maas-api-5c48877c75-spd87\" (UID: \"e615be51-c198-4c99-bf87-ba09654ea703\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-spd87"
Oct 30 13:57:24.813101 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:24.813074    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e615be51-c198-4c99-bf87-ba09654ea703-maas-api-tls\") pod \"maas-api-5c48877c75-spd87\" (UID: \"e615be51-c198-4c99-bf87-ba09654ea703\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-spd87"
Oct 30 13:57:24.813267 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:24.813111    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e615be51-c198-4c99-bf87-ba09654ea703-openshift-service-ca\") pod \"maas-api-5c48877c75-spd87\" (UID: \"e615be51-c198-4c99-bf87-ba09654ea703\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-spd87"
Oct 30 13:57:24.813267 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:24.813191    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e615be51-c198-4c99-bf87-ba09654ea703-metrics-certs\") pod \"maas-api-5c48877c75-spd87\" (UID: \"e615be51-c198-4c99-bf87-ba09654ea703\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-spd87"
Oct 30 13:57:24.813362 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:24.813311    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fcd4eb6-58be-46a7-ac2c-70c876aa08ed-metrics-certs\") pod \"maas-api-595cc648c-4jkwp\" (UID: \"4fcd4eb6-58be-46a7-ac2c-70c876aa08ed\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-4jkwp"
Oct 30 13:57:24.813421 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:24.813370    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqpkl\" (UniqueName: \"kubernetes.io/projected/4fcd4eb6-58be-46a7-ac2c-70c876aa08ed-kube-api-access-xqpkl\") pod \"maas-api-595cc648c-4jkwp\" (UID: \"4fcd4eb6-58be-46a7-ac2c-70c876aa08ed\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-4jkwp"
Oct 30 13:57:24.813421 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:24.813403    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fcd4eb6-58be-46a7-ac2c-70c876aa08ed-maas-api-tls\") pod \"maas-api-595cc648c-4jkwp\" (UID: \"4fcd4eb6-58be-46a7-ac2c-70c876aa08ed\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-4jkwp"
Oct 30 13:57:24.813533 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:24.813439    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkncp\" (UniqueName: \"kubernetes.io/projected/e615be51-c198-4c99-bf87-ba09654ea703-kube-api-access-fkncp\") pod \"maas-api-5c48877c75-spd87\" (UID: \"e615be51-c198-4c99-bf87-ba09654ea703\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-spd87"
Oct 30 13:57:24.813533 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:24.813517    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fcd4eb6-58be-46a7-ac2c-70c876aa08ed-openshift-service-ca\") pod \"maas-api-595cc648c-4jkwp\" (UID: \"4fcd4eb6-58be-46a7-ac2c-70c876aa08ed\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-4jkwp"
Oct 30 13:57:24.814026 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:24.814000    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e615be51-c198-4c99-bf87-ba09654ea703-openshift-service-ca\") pod \"maas-api-5c48877c75-spd87\" (UID: \"e615be51-c198-4c99-bf87-ba09654ea703\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-spd87"
Oct 30 13:57:24.814255 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:24.814234    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fcd4eb6-58be-46a7-ac2c-70c876aa08ed-openshift-service-ca\") pod \"maas-api-595cc648c-4jkwp\" (UID: \"4fcd4eb6-58be-46a7-ac2c-70c876aa08ed\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-4jkwp"
Oct 30 13:57:24.816334 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:24.816311    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e615be51-c198-4c99-bf87-ba09654ea703-metrics-certs\") pod \"maas-api-5c48877c75-spd87\" (UID: \"e615be51-c198-4c99-bf87-ba09654ea703\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-spd87"
Oct 30 13:57:24.816409 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:24.816359    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fcd4eb6-58be-46a7-ac2c-70c876aa08ed-maas-api-tls\") pod \"maas-api-595cc648c-4jkwp\" (UID: \"4fcd4eb6-58be-46a7-ac2c-70c876aa08ed\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-4jkwp"
Oct 30 13:57:24.816409 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:24.816359    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fcd4eb6-58be-46a7-ac2c-70c876aa08ed-metrics-certs\") pod \"maas-api-595cc648c-4jkwp\" (UID: \"4fcd4eb6-58be-46a7-ac2c-70c876aa08ed\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-4jkwp"
Oct 30 13:57:24.819114 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:24.819091    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e615be51-c198-4c99-bf87-ba09654ea703-maas-api-tls\") pod \"maas-api-5c48877c75-spd87\" (UID: \"e615be51-c198-4c99-bf87-ba09654ea703\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-spd87"
Oct 30 13:57:24.836892 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:24.836864    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkncp\" (UniqueName: \"kubernetes.io/projected/e615be51-c198-4c99-bf87-ba09654ea703-kube-api-access-fkncp\") pod \"maas-api-5c48877c75-spd87\" (UID: \"e615be51-c198-4c99-bf87-ba09654ea703\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-spd87"
Oct 30 13:57:24.838218 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:24.838195    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqpkl\" (UniqueName: \"kubernetes.io/projected/4fcd4eb6-58be-46a7-ac2c-70c876aa08ed-kube-api-access-xqpkl\") pod \"maas-api-595cc648c-4jkwp\" (UID: \"4fcd4eb6-58be-46a7-ac2c-70c876aa08ed\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-4jkwp"
Oct 30 13:57:24.917442 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:24.917411    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-spd87"
Oct 30 13:57:24.939766 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:24.939736    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-4jkwp"
Oct 30 13:57:25.226916 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:25.226838    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-spd87"]
Oct 30 13:57:25.241625 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:57:25.241586    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode615be51_c198_4c99_bf87_ba09654ea703.slice/crio-bb2d4175a9b8ba9f4c4fdb90f6b622ab579b95c9ec179df24e4821ff93576f54 WatchSource:0}: Error finding container bb2d4175a9b8ba9f4c4fdb90f6b622ab579b95c9ec179df24e4821ff93576f54: Status 404 returned error can't find the container with id bb2d4175a9b8ba9f4c4fdb90f6b622ab579b95c9ec179df24e4821ff93576f54
Oct 30 13:57:25.242901 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:25.242882    4364 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 13:57:25.269749 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:25.269721    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-4jkwp"]
Oct 30 13:57:25.271774 ip-10-0-137-6 kubenswrapper[4364]: W1030 13:57:25.271745    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fcd4eb6_58be_46a7_ac2c_70c876aa08ed.slice/crio-4dc8e140a0647a7c68ed25b600036c1bbfdcab8994393aeb32e3c4dcabb6f6dc WatchSource:0}: Error finding container 4dc8e140a0647a7c68ed25b600036c1bbfdcab8994393aeb32e3c4dcabb6f6dc: Status 404 returned error can't find the container with id 4dc8e140a0647a7c68ed25b600036c1bbfdcab8994393aeb32e3c4dcabb6f6dc
Oct 30 13:57:25.770904 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:25.770861    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-4jkwp" event={"ID":"4fcd4eb6-58be-46a7-ac2c-70c876aa08ed","Type":"ContainerStarted","Data":"af6ff11eb94417b687ed41d5198a0e76b2ed7a75eea5a50cb78251697bd25716"}
Oct 30 13:57:25.771407 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:25.770918    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-4jkwp" event={"ID":"4fcd4eb6-58be-46a7-ac2c-70c876aa08ed","Type":"ContainerStarted","Data":"4dc8e140a0647a7c68ed25b600036c1bbfdcab8994393aeb32e3c4dcabb6f6dc"}
Oct 30 13:57:25.771407 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:25.771056    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-4jkwp"
Oct 30 13:57:25.772659 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:25.772631    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-spd87" event={"ID":"e615be51-c198-4c99-bf87-ba09654ea703","Type":"ContainerStarted","Data":"599cdb77a76afdd89b7943edf25500369b1201fbd86e038fd2bef4760732d9e6"}
Oct 30 13:57:25.772818 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:25.772676    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-spd87" event={"ID":"e615be51-c198-4c99-bf87-ba09654ea703","Type":"ContainerStarted","Data":"bb2d4175a9b8ba9f4c4fdb90f6b622ab579b95c9ec179df24e4821ff93576f54"}
Oct 30 13:57:25.772818 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:25.772702    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-spd87"
Oct 30 13:57:25.825204 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:25.825145    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-595cc648c-4jkwp" podStartSLOduration=1.463389963 podStartE2EDuration="1.825120771s" totalImagesPullingTime="361.730808ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:57:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:57:25.819702639 +0000 UTC m=+3944.986067288" watchObservedRunningTime="2026-10-30 13:57:25.825120771 +0000 UTC m=+3944.991485422"
Oct 30 13:57:25.867742 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:25.867698    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5c48877c75-spd87" podStartSLOduration=1.498510776 podStartE2EDuration="1.867684632s" totalImagesPullingTime="369.173856ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:57:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:57:25.865268237 +0000 UTC m=+3945.031632884" watchObservedRunningTime="2026-10-30 13:57:25.867684632 +0000 UTC m=+3945.034049279"
Oct 30 13:57:31.784348 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:31.784318    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-4jkwp"
Oct 30 13:57:31.784822 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:31.784728    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-spd87"
Oct 30 13:57:31.864167 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:31.864131    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-spd87"]
Oct 30 13:57:31.864420 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:31.864390    4364 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5c48877c75-spd87" podUID="e615be51-c198-4c99-bf87-ba09654ea703" containerName="maas-api" containerID="cri-o://599cdb77a76afdd89b7943edf25500369b1201fbd86e038fd2bef4760732d9e6" gracePeriod=30
Oct 30 13:57:32.109354 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:32.109330    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-spd87"
Oct 30 13:57:32.190835 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:32.190803    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e615be51-c198-4c99-bf87-ba09654ea703-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e615be51-c198-4c99-bf87-ba09654ea703-metrics-certs\") pod \"e615be51-c198-4c99-bf87-ba09654ea703\" (UID: \"e615be51-c198-4c99-bf87-ba09654ea703\") "
Oct 30 13:57:32.191013 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:32.190846    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e615be51-c198-4c99-bf87-ba09654ea703-kube-api-access-fkncp\" (UniqueName: \"kubernetes.io/projected/e615be51-c198-4c99-bf87-ba09654ea703-kube-api-access-fkncp\") pod \"e615be51-c198-4c99-bf87-ba09654ea703\" (UID: \"e615be51-c198-4c99-bf87-ba09654ea703\") "
Oct 30 13:57:32.191013 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:32.190896    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e615be51-c198-4c99-bf87-ba09654ea703-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e615be51-c198-4c99-bf87-ba09654ea703-openshift-service-ca\") pod \"e615be51-c198-4c99-bf87-ba09654ea703\" (UID: \"e615be51-c198-4c99-bf87-ba09654ea703\") "
Oct 30 13:57:32.191013 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:32.190950    4364 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e615be51-c198-4c99-bf87-ba09654ea703-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e615be51-c198-4c99-bf87-ba09654ea703-maas-api-tls\") pod \"e615be51-c198-4c99-bf87-ba09654ea703\" (UID: \"e615be51-c198-4c99-bf87-ba09654ea703\") "
Oct 30 13:57:32.191318 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:32.191289    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e615be51-c198-4c99-bf87-ba09654ea703-openshift-service-ca" pod "e615be51-c198-4c99-bf87-ba09654ea703" (UID: "e615be51-c198-4c99-bf87-ba09654ea703"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:57:32.191472 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:32.191407    4364 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e615be51-c198-4c99-bf87-ba09654ea703-openshift-service-ca\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:57:32.193063 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:32.193034    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e615be51-c198-4c99-bf87-ba09654ea703-kube-api-access-fkncp" pod "e615be51-c198-4c99-bf87-ba09654ea703" (UID: "e615be51-c198-4c99-bf87-ba09654ea703"). InnerVolumeSpecName "kube-api-access-fkncp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:57:32.193182 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:32.193036    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e615be51-c198-4c99-bf87-ba09654ea703-maas-api-tls" pod "e615be51-c198-4c99-bf87-ba09654ea703" (UID: "e615be51-c198-4c99-bf87-ba09654ea703"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:57:32.193182 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:32.193067    4364 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e615be51-c198-4c99-bf87-ba09654ea703-metrics-certs" pod "e615be51-c198-4c99-bf87-ba09654ea703" (UID: "e615be51-c198-4c99-bf87-ba09654ea703"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:57:32.293020 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:32.292938    4364 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e615be51-c198-4c99-bf87-ba09654ea703-maas-api-tls\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:57:32.293020 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:32.292965    4364 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e615be51-c198-4c99-bf87-ba09654ea703-metrics-certs\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:57:32.293020 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:32.292975    4364 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkncp\" (UniqueName: \"kubernetes.io/projected/e615be51-c198-4c99-bf87-ba09654ea703-kube-api-access-fkncp\") on node \"ip-10-0-137-6.ec2.internal\" DevicePath \"\""
Oct 30 13:57:32.812134 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:32.812103    4364 generic.go:356] "Generic (PLEG): container finished" podID="e615be51-c198-4c99-bf87-ba09654ea703" containerID="599cdb77a76afdd89b7943edf25500369b1201fbd86e038fd2bef4760732d9e6" exitCode=0
Oct 30 13:57:32.812629 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:32.812171    4364 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-spd87"
Oct 30 13:57:32.812629 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:32.812194    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-spd87" event={"ID":"e615be51-c198-4c99-bf87-ba09654ea703","Type":"ContainerDied","Data":"599cdb77a76afdd89b7943edf25500369b1201fbd86e038fd2bef4760732d9e6"}
Oct 30 13:57:32.812629 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:32.812254    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-spd87" event={"ID":"e615be51-c198-4c99-bf87-ba09654ea703","Type":"ContainerDied","Data":"bb2d4175a9b8ba9f4c4fdb90f6b622ab579b95c9ec179df24e4821ff93576f54"}
Oct 30 13:57:32.812629 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:32.812282    4364 scope.go:122] "RemoveContainer" containerID="599cdb77a76afdd89b7943edf25500369b1201fbd86e038fd2bef4760732d9e6"
Oct 30 13:57:32.832644 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:32.832621    4364 scope.go:122] "RemoveContainer" containerID="599cdb77a76afdd89b7943edf25500369b1201fbd86e038fd2bef4760732d9e6"
Oct 30 13:57:32.832980 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:57:32.832953    4364 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"599cdb77a76afdd89b7943edf25500369b1201fbd86e038fd2bef4760732d9e6\": container with ID starting with 599cdb77a76afdd89b7943edf25500369b1201fbd86e038fd2bef4760732d9e6 not found: ID does not exist" containerID="599cdb77a76afdd89b7943edf25500369b1201fbd86e038fd2bef4760732d9e6"
Oct 30 13:57:32.833048 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:32.832992    4364 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"599cdb77a76afdd89b7943edf25500369b1201fbd86e038fd2bef4760732d9e6"} err="failed to get container status \"599cdb77a76afdd89b7943edf25500369b1201fbd86e038fd2bef4760732d9e6\": rpc error: code = NotFound desc = could not find container \"599cdb77a76afdd89b7943edf25500369b1201fbd86e038fd2bef4760732d9e6\": container with ID starting with 599cdb77a76afdd89b7943edf25500369b1201fbd86e038fd2bef4760732d9e6 not found: ID does not exist"
Oct 30 13:57:32.869558 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:32.869534    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-spd87"]
Oct 30 13:57:32.871814 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:32.871794    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-spd87"]
Oct 30 13:57:32.968259 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:32.968226    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e615be51-c198-4c99-bf87-ba09654ea703" path="/var/lib/kubelet/pods/e615be51-c198-4c99-bf87-ba09654ea703/volumes"
Oct 30 13:57:33.307148 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:33.307114    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:57:35.761848 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:35.761767    4364 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-2vqwj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:57:35.763069 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:35.762590    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-2vqwj" podUID="fd6ee897-425a-4e17-978d-8afde52b1331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:57:35.836052 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:35.835707    4364 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-vt6z4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.14:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:57:35.836052 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:35.835773    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vt6z4" podUID="adbce7c5-af05-45e2-982c-183972ed458d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.14:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:57:41.504018 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:57:41.503983    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:57:41.618573 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:41.618538    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:57:54.942982 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:54.942948    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:57:59.899622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:59.896693    4364 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-tbw9c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:57:59.899622 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:57:59.896776    4364 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tbw9c" podUID="e40d7a56-5869-4a42-b8a7-33c02604c302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:58:11.504100 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:58:11.504055    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:58:20.785620 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:58:20.785569    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-szbp8"]
Oct 30 13:58:31.907348 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:58:31.907266    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-8vv55_44383748-6649-458a-a7b2-aa6c443d12f6/manager/0.log"
Oct 30 13:58:32.124256 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:58:32.124224    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-5cfb77bf8c-nbfjt_e802a8b0-e3e0-4edd-aed0-c62e45c384e6/manager/0.log"
Oct 30 13:58:32.984705 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:58:32.984671    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-47nwj_9b39dfb8-d390-440c-bad4-9a51d4b00142/server/0.log"
Oct 30 13:58:33.216741 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:58:33.216711    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-f87fcf56c-zwv52_426ba27f-a567-4c9c-a9da-fa6d7d4772da/manager/0.log"
Oct 30 13:58:33.457820 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:58:33.457791    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-observability-584f99c788-w8fvw_a83c7ff7-e667-4c9a-b13b-ac37eb14f3b6/manager/0.log"
Oct 30 13:58:33.964983 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:58:33.964951    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-678d69fff5-rlqml_3afcec39-f8ad-46ba-b002-cf1465de9072/manager/0.log"
Oct 30 13:58:33.997029 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:58:33.996999    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-678d69fff5-rlqml_3afcec39-f8ad-46ba-b002-cf1465de9072/copy-manifests/0.log"
Oct 30 13:58:35.236652 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:58:35.236615    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-4jkwp_4fcd4eb6-58be-46a7-ac2c-70c876aa08ed/maas-api/0.log"
Oct 30 13:58:35.835897 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:58:35.835866    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-mx6gc_4c97ab5f-97f9-4d5a-b59a-a7abdbff75c5/revoke-keys/0.log"
Oct 30 13:58:36.026821 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:58:36.026789    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-fvl2n_e973b6e6-56ba-416a-9a1f-bbece296188e/revoke-keys/0.log"
Oct 30 13:58:36.572637 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:58:36.572588    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d2bc93bc-58w2x_aba25037-7144-4ba0-9a1a-081243f9445c/revoke-keys/0.log"
Oct 30 13:58:36.755727 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:58:36.755698    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d576e68b-k2ft9_17a1d683-7bf3-4c03-8223-2ca23778288c/revoke-keys/0.log"
Oct 30 13:58:37.142546 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:58:37.142520    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f929de75-j7xrt_b34c6967-3a1c-4e37-afe9-d21af16c5c4e/revoke-keys/0.log"
Oct 30 13:58:39.946076 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:58:39.946051    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-szbp8_dddc7c6a-09a5-496c-aa3f-cd8c50ed5a13/limitador/0.log"
Oct 30 13:58:40.915257 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:58:40.915223    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6bc88ffd5-5pkjs_5f540848-0b4f-4398-8a1f-6d1b3d7a379d/istio-proxy/0.log"
Oct 30 13:58:41.358816 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:58:41.358740    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65dc4d65b9-59szd_ac493c83-00bc-43fa-b0d1-ec05f21fcd92/kube-auth-proxy/0.log"
Oct 30 13:58:41.503131 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:58:41.503092    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:58:41.747121 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:58:41.747093    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-64d4cc998c-2vqwj_fd6ee897-425a-4e17-978d-8afde52b1331/istio-proxy/0.log"
Oct 30 13:58:42.973097 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:58:42.973065    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-64d4cc998c-tbw9c_e40d7a56-5869-4a42-b8a7-33c02604c302/istio-proxy/0.log"
Oct 30 13:58:43.409864 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:58:43.409834    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-64d4cc998c-vt6z4_adbce7c5-af05-45e2-982c-183972ed458d/istio-proxy/0.log"
Oct 30 13:58:43.619577 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:58:43.619543    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-64d4cc998c-wfkdb_cac946c2-24d3-42b9-9d62-5f45833c81fc/istio-proxy/0.log"
Oct 30 13:58:44.016016 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:58:44.015986    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-mqpfw_310804ff-17ef-4a28-959c-47995bde35ca/payload-processing/0.log"
Oct 30 13:58:44.932239 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:58:44.932208    4364 scope.go:122] "RemoveContainer" containerID="18f243fcd2277e1f178457e1fe65fb8af437d57cfbf1a1655b91be3893066c25"
Oct 30 13:58:45.004344 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:58:45.004315    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-tvhvc_fda756b1-bfd0-426a-b036-13c90355172a/main/0.log"
Oct 30 13:58:45.209095 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:58:45.209020    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-n7v9l_d3bdb4e6-148f-45c7-9fdd-1f0846e21699/main/0.log"
Oct 30 13:58:45.601193 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:58:45.601118    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-m6h9s_f73c0035-9dd8-47c5-9ea6-e1e3aba4968e/main/0.log"
Oct 30 13:58:56.611941 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:58:56.611908    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-8vv55_44383748-6649-458a-a7b2-aa6c443d12f6/manager/0.log"
Oct 30 13:58:56.840982 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:58:56.840939    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-5cfb77bf8c-nbfjt_e802a8b0-e3e0-4edd-aed0-c62e45c384e6/manager/0.log"
Oct 30 13:58:57.582694 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:58:57.582641    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-47nwj_9b39dfb8-d390-440c-bad4-9a51d4b00142/server/0.log"
Oct 30 13:58:57.777075 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:58:57.777045    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-f87fcf56c-zwv52_426ba27f-a567-4c9c-a9da-fa6d7d4772da/manager/0.log"
Oct 30 13:58:57.998476 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:58:57.998446    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-observability-584f99c788-w8fvw_a83c7ff7-e667-4c9a-b13b-ac37eb14f3b6/manager/0.log"
Oct 30 13:58:58.508256 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:58:58.508226    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-678d69fff5-rlqml_3afcec39-f8ad-46ba-b002-cf1465de9072/copy-manifests/0.log"
Oct 30 13:58:58.547031 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:58:58.547005    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-678d69fff5-rlqml_3afcec39-f8ad-46ba-b002-cf1465de9072/manager/0.log"
Oct 30 13:58:59.598728 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:58:59.598695    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-4jkwp_4fcd4eb6-58be-46a7-ac2c-70c876aa08ed/maas-api/0.log"
Oct 30 13:59:00.209195 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:59:00.209165    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-mx6gc_4c97ab5f-97f9-4d5a-b59a-a7abdbff75c5/revoke-keys/0.log"
Oct 30 13:59:00.386335 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:59:00.386306    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-fvl2n_e973b6e6-56ba-416a-9a1f-bbece296188e/revoke-keys/0.log"
Oct 30 13:59:00.960761 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:59:00.960732    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d2bc93bc-58w2x_aba25037-7144-4ba0-9a1a-081243f9445c/revoke-keys/0.log"
Oct 30 13:59:01.158075 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:59:01.158046    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d576e68b-k2ft9_17a1d683-7bf3-4c03-8223-2ca23778288c/revoke-keys/0.log"
Oct 30 13:59:01.601838 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:59:01.601809    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f929de75-j7xrt_b34c6967-3a1c-4e37-afe9-d21af16c5c4e/revoke-keys/0.log"
Oct 30 13:59:04.135816 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:59:04.135786    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-szbp8_dddc7c6a-09a5-496c-aa3f-cd8c50ed5a13/limitador/0.log"
Oct 30 13:59:04.900856 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:59:04.900823    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6bc88ffd5-5pkjs_5f540848-0b4f-4398-8a1f-6d1b3d7a379d/istio-proxy/0.log"
Oct 30 13:59:05.333349 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:59:05.333261    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65dc4d65b9-59szd_ac493c83-00bc-43fa-b0d1-ec05f21fcd92/kube-auth-proxy/0.log"
Oct 30 13:59:05.731067 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:59:05.731039    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-64d4cc998c-2vqwj_fd6ee897-425a-4e17-978d-8afde52b1331/istio-proxy/0.log"
Oct 30 13:59:06.984636 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:59:06.984589    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-64d4cc998c-tbw9c_e40d7a56-5869-4a42-b8a7-33c02604c302/istio-proxy/0.log"
Oct 30 13:59:07.416247 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:59:07.416214    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-64d4cc998c-vt6z4_adbce7c5-af05-45e2-982c-183972ed458d/istio-proxy/0.log"
Oct 30 13:59:07.625217 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:59:07.625186    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-64d4cc998c-wfkdb_cac946c2-24d3-42b9-9d62-5f45833c81fc/istio-proxy/0.log"
Oct 30 13:59:08.012467 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:59:08.012437    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-mqpfw_310804ff-17ef-4a28-959c-47995bde35ca/payload-processing/0.log"
Oct 30 13:59:08.818526 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:59:08.818492    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-tvhvc_fda756b1-bfd0-426a-b036-13c90355172a/main/0.log"
Oct 30 13:59:09.035251 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:59:09.035193    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-n7v9l_d3bdb4e6-148f-45c7-9fdd-1f0846e21699/main/0.log"
Oct 30 13:59:09.421907 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:59:09.421872    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-m6h9s_f73c0035-9dd8-47c5-9ea6-e1e3aba4968e/main/0.log"
Oct 30 13:59:10.047191 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:59:10.047151    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-k2ft9"]
Oct 30 13:59:10.059249 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:59:10.059222    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-k2ft9"]
Oct 30 13:59:10.975455 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:59:10.975419    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17a1d683-7bf3-4c03-8223-2ca23778288c" path="/var/lib/kubelet/pods/17a1d683-7bf3-4c03-8223-2ca23778288c/volumes"
Oct 30 13:59:11.503747 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:59:11.503710    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:59:26.045252 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:59:26.045213    4364 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-58w2x"]
Oct 30 13:59:26.057200 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:59:26.057172    4364 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-58w2x"]
Oct 30 13:59:26.968592 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:59:26.968556    4364 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aba25037-7144-4ba0-9a1a-081243f9445c" path="/var/lib/kubelet/pods/aba25037-7144-4ba0-9a1a-081243f9445c/volumes"
Oct 30 13:59:41.503361 ip-10-0-137-6 kubenswrapper[4364]: E1030 13:59:41.503322    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 13:59:47.191286 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:59:47.191253    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-mqpfw_310804ff-17ef-4a28-959c-47995bde35ca/payload-processing/0.log"
Oct 30 13:59:47.374908 ip-10-0-137-6 kubenswrapper[4364]: I1030 13:59:47.374879    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-mqpfw_310804ff-17ef-4a28-959c-47995bde35ca/payload-processing/0.log"
Oct 30 14:00:11.503780 ip-10-0-137-6 kubenswrapper[4364]: E1030 14:00:11.503738    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 14:00:33.387657 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:00:33.387626    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-mqpfw_310804ff-17ef-4a28-959c-47995bde35ca/payload-processing/0.log"
Oct 30 14:00:33.572219 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:00:33.572194    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-mqpfw_310804ff-17ef-4a28-959c-47995bde35ca/payload-processing/0.log"
Oct 30 14:00:33.913363 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:00:33.913329    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rkb8w/must-gather-4tz66"]
Oct 30 14:00:33.913789 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:00:33.913775    4364 cpu_manager.go:414] "containerMap: removing container" podUID="e615be51-c198-4c99-bf87-ba09654ea703" containerName="maas-api"
Oct 30 14:00:33.913789 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:00:33.913791    4364 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e615be51-c198-4c99-bf87-ba09654ea703" containerName="maas-api"
Oct 30 14:00:33.913888 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:00:33.913859    4364 memory_manager.go:365] "RemoveStaleState removing state" podUID="e615be51-c198-4c99-bf87-ba09654ea703" containerName="maas-api"
Oct 30 14:00:33.917223 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:00:33.917205    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rkb8w/must-gather-4tz66"
Oct 30 14:00:33.944083 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:00:33.944052    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cf97a905-7834-4980-b3c4-0971b3d89ed4-must-gather-output\") pod \"must-gather-4tz66\" (UID: \"cf97a905-7834-4980-b3c4-0971b3d89ed4\") " pod="openshift-must-gather-rkb8w/must-gather-4tz66"
Oct 30 14:00:33.944230 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:00:33.944103    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2jbs\" (UniqueName: \"kubernetes.io/projected/cf97a905-7834-4980-b3c4-0971b3d89ed4-kube-api-access-g2jbs\") pod \"must-gather-4tz66\" (UID: \"cf97a905-7834-4980-b3c4-0971b3d89ed4\") " pod="openshift-must-gather-rkb8w/must-gather-4tz66"
Oct 30 14:00:33.947273 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:00:33.947247    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rkb8w\"/\"openshift-service-ca.crt\""
Oct 30 14:00:33.947273 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:00:33.947251    4364 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rkb8w\"/\"kube-root-ca.crt\""
Oct 30 14:00:33.947458 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:00:33.947311    4364 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rkb8w\"/\"default-dockercfg-dpzm8\""
Oct 30 14:00:33.949985 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:00:33.949959    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rkb8w/must-gather-4tz66"]
Oct 30 14:00:34.044948 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:00:34.044914    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2jbs\" (UniqueName: \"kubernetes.io/projected/cf97a905-7834-4980-b3c4-0971b3d89ed4-kube-api-access-g2jbs\") pod \"must-gather-4tz66\" (UID: \"cf97a905-7834-4980-b3c4-0971b3d89ed4\") " pod="openshift-must-gather-rkb8w/must-gather-4tz66"
Oct 30 14:00:34.045239 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:00:34.045213    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cf97a905-7834-4980-b3c4-0971b3d89ed4-must-gather-output\") pod \"must-gather-4tz66\" (UID: \"cf97a905-7834-4980-b3c4-0971b3d89ed4\") " pod="openshift-must-gather-rkb8w/must-gather-4tz66"
Oct 30 14:00:34.045639 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:00:34.045591    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cf97a905-7834-4980-b3c4-0971b3d89ed4-must-gather-output\") pod \"must-gather-4tz66\" (UID: \"cf97a905-7834-4980-b3c4-0971b3d89ed4\") " pod="openshift-must-gather-rkb8w/must-gather-4tz66"
Oct 30 14:00:34.073385 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:00:34.073357    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2jbs\" (UniqueName: \"kubernetes.io/projected/cf97a905-7834-4980-b3c4-0971b3d89ed4-kube-api-access-g2jbs\") pod \"must-gather-4tz66\" (UID: \"cf97a905-7834-4980-b3c4-0971b3d89ed4\") " pod="openshift-must-gather-rkb8w/must-gather-4tz66"
Oct 30 14:00:34.231856 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:00:34.231780    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rkb8w/must-gather-4tz66"
Oct 30 14:00:34.537524 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:00:34.537499    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rkb8w/must-gather-4tz66"]
Oct 30 14:00:34.539462 ip-10-0-137-6 kubenswrapper[4364]: W1030 14:00:34.539431    4364 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcf97a905_7834_4980_b3c4_0971b3d89ed4.slice/crio-a564f8e9d7df77a88635fdee33e6d18a4580de1a877381fd8107b37f32ad202a WatchSource:0}: Error finding container a564f8e9d7df77a88635fdee33e6d18a4580de1a877381fd8107b37f32ad202a: Status 404 returned error can't find the container with id a564f8e9d7df77a88635fdee33e6d18a4580de1a877381fd8107b37f32ad202a
Oct 30 14:00:34.895104 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:00:34.895067    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rkb8w/must-gather-4tz66" event={"ID":"cf97a905-7834-4980-b3c4-0971b3d89ed4","Type":"ContainerStarted","Data":"a564f8e9d7df77a88635fdee33e6d18a4580de1a877381fd8107b37f32ad202a"}
Oct 30 14:00:35.910079 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:00:35.910039    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rkb8w/must-gather-4tz66" event={"ID":"cf97a905-7834-4980-b3c4-0971b3d89ed4","Type":"ContainerStarted","Data":"bfa67b0034dc5bf5bad2c3736ed4fc7bb4a1eafb9a8d6926cd4d7ef49cf88406"}
Oct 30 14:00:35.910582 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:00:35.910092    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rkb8w/must-gather-4tz66" event={"ID":"cf97a905-7834-4980-b3c4-0971b3d89ed4","Type":"ContainerStarted","Data":"aa4110b5fc4c896f0b4c146b53d0aadd41ce4ab0ddcb9c3db5607d2b4f30a095"}
Oct 30 14:00:35.952431 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:00:35.952293    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-rkb8w/must-gather-4tz66" podStartSLOduration=2.206941861 podStartE2EDuration="2.952274875s" totalImagesPullingTime="745.333014ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:00:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:00:35.951048439 +0000 UTC m=+4135.117413087" watchObservedRunningTime="2026-10-30 14:00:35.952274875 +0000 UTC m=+4135.118639524"
Oct 30 14:00:38.874182 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:00:38.874150    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-rxg7f_815986e1-07b4-43fb-95f5-745316bd5105/global-pull-secret-syncer/0.log"
Oct 30 14:00:39.729854 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:00:39.729814    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vlfkx_7d4b7525-1c1b-4e65-b263-349ab1a0f32f/konnectivity-agent/0.log"
Oct 30 14:00:40.144930 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:00:40.144895    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-6.ec2.internal_c8cfc7a7881c11ddfae618f888b1edd8/haproxy/0.log"
Oct 30 14:00:41.621178 ip-10-0-137-6 kubenswrapper[4364]: E1030 14:00:41.621101    4364 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zn76j" containerName="extractor"
Oct 30 14:00:44.963769 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:00:44.963736    4364 scope.go:122] "RemoveContainer" containerID="757305002e433f193ff118f678eebe04b0ac0724825d9553324b7006a0a1d4aa"
Oct 30 14:00:44.982084 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:00:44.982061    4364 scope.go:122] "RemoveContainer" containerID="17d3ff853f810588d54bd8dc902300eb3feb0b0f97b59b5abef9a536d9f082c0"
Oct 30 14:00:46.976980 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:00:46.976945    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-szbp8_dddc7c6a-09a5-496c-aa3f-cd8c50ed5a13/limitador/0.log"
Oct 30 14:00:47.792791 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:00:47.792749    4364 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rkb8w/perf-node-gather-daemonset-qprrq"]
Oct 30 14:00:47.800616 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:00:47.798798    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rkb8w/perf-node-gather-daemonset-qprrq"
Oct 30 14:00:47.871888 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:00:47.871846    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rkb8w/perf-node-gather-daemonset-qprrq"]
Oct 30 14:00:47.894427 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:00:47.894384    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6kd7\" (UniqueName: \"kubernetes.io/projected/e60204fd-6f0d-4199-958d-315ecea66d94-kube-api-access-f6kd7\") pod \"perf-node-gather-daemonset-qprrq\" (UID: \"e60204fd-6f0d-4199-958d-315ecea66d94\") " pod="openshift-must-gather-rkb8w/perf-node-gather-daemonset-qprrq"
Oct 30 14:00:47.895099 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:00:47.895072    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e60204fd-6f0d-4199-958d-315ecea66d94-proc\") pod \"perf-node-gather-daemonset-qprrq\" (UID: \"e60204fd-6f0d-4199-958d-315ecea66d94\") " pod="openshift-must-gather-rkb8w/perf-node-gather-daemonset-qprrq"
Oct 30 14:00:47.895293 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:00:47.895274    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e60204fd-6f0d-4199-958d-315ecea66d94-lib-modules\") pod \"perf-node-gather-daemonset-qprrq\" (UID: \"e60204fd-6f0d-4199-958d-315ecea66d94\") " pod="openshift-must-gather-rkb8w/perf-node-gather-daemonset-qprrq"
Oct 30 14:00:47.895488 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:00:47.895465    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e60204fd-6f0d-4199-958d-315ecea66d94-sys\") pod \"perf-node-gather-daemonset-qprrq\" (UID: \"e60204fd-6f0d-4199-958d-315ecea66d94\") " pod="openshift-must-gather-rkb8w/perf-node-gather-daemonset-qprrq"
Oct 30 14:00:47.895585 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:00:47.895535    4364 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e60204fd-6f0d-4199-958d-315ecea66d94-podres\") pod \"perf-node-gather-daemonset-qprrq\" (UID: \"e60204fd-6f0d-4199-958d-315ecea66d94\") " pod="openshift-must-gather-rkb8w/perf-node-gather-daemonset-qprrq"
Oct 30 14:00:47.998390 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:00:47.998154    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e60204fd-6f0d-4199-958d-315ecea66d94-podres\") pod \"perf-node-gather-daemonset-qprrq\" (UID: \"e60204fd-6f0d-4199-958d-315ecea66d94\") " pod="openshift-must-gather-rkb8w/perf-node-gather-daemonset-qprrq"
Oct 30 14:00:47.998390 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:00:47.998299    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6kd7\" (UniqueName: \"kubernetes.io/projected/e60204fd-6f0d-4199-958d-315ecea66d94-kube-api-access-f6kd7\") pod \"perf-node-gather-daemonset-qprrq\" (UID: \"e60204fd-6f0d-4199-958d-315ecea66d94\") " pod="openshift-must-gather-rkb8w/perf-node-gather-daemonset-qprrq"
Oct 30 14:00:47.998390 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:00:47.998357    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e60204fd-6f0d-4199-958d-315ecea66d94-proc\") pod \"perf-node-gather-daemonset-qprrq\" (UID: \"e60204fd-6f0d-4199-958d-315ecea66d94\") " pod="openshift-must-gather-rkb8w/perf-node-gather-daemonset-qprrq"
Oct 30 14:00:47.998390 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:00:47.998391    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e60204fd-6f0d-4199-958d-315ecea66d94-lib-modules\") pod \"perf-node-gather-daemonset-qprrq\" (UID: \"e60204fd-6f0d-4199-958d-315ecea66d94\") " pod="openshift-must-gather-rkb8w/perf-node-gather-daemonset-qprrq"
Oct 30 14:00:47.999202 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:00:47.998432    4364 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e60204fd-6f0d-4199-958d-315ecea66d94-sys\") pod \"perf-node-gather-daemonset-qprrq\" (UID: \"e60204fd-6f0d-4199-958d-315ecea66d94\") " pod="openshift-must-gather-rkb8w/perf-node-gather-daemonset-qprrq"
Oct 30 14:00:47.999202 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:00:47.998470    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e60204fd-6f0d-4199-958d-315ecea66d94-podres\") pod \"perf-node-gather-daemonset-qprrq\" (UID: \"e60204fd-6f0d-4199-958d-315ecea66d94\") " pod="openshift-must-gather-rkb8w/perf-node-gather-daemonset-qprrq"
Oct 30 14:00:47.999202 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:00:47.998527    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e60204fd-6f0d-4199-958d-315ecea66d94-sys\") pod \"perf-node-gather-daemonset-qprrq\" (UID: \"e60204fd-6f0d-4199-958d-315ecea66d94\") " pod="openshift-must-gather-rkb8w/perf-node-gather-daemonset-qprrq"
Oct 30 14:00:47.999202 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:00:47.998552    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e60204fd-6f0d-4199-958d-315ecea66d94-proc\") pod \"perf-node-gather-daemonset-qprrq\" (UID: \"e60204fd-6f0d-4199-958d-315ecea66d94\") " pod="openshift-must-gather-rkb8w/perf-node-gather-daemonset-qprrq"
Oct 30 14:00:47.999202 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:00:47.998584    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e60204fd-6f0d-4199-958d-315ecea66d94-lib-modules\") pod \"perf-node-gather-daemonset-qprrq\" (UID: \"e60204fd-6f0d-4199-958d-315ecea66d94\") " pod="openshift-must-gather-rkb8w/perf-node-gather-daemonset-qprrq"
Oct 30 14:00:48.027117 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:00:48.027079    4364 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6kd7\" (UniqueName: \"kubernetes.io/projected/e60204fd-6f0d-4199-958d-315ecea66d94-kube-api-access-f6kd7\") pod \"perf-node-gather-daemonset-qprrq\" (UID: \"e60204fd-6f0d-4199-958d-315ecea66d94\") " pod="openshift-must-gather-rkb8w/perf-node-gather-daemonset-qprrq"
Oct 30 14:00:48.150639 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:00:48.150579    4364 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rkb8w/perf-node-gather-daemonset-qprrq"
Oct 30 14:00:48.475268 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:00:48.475237    4364 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rkb8w/perf-node-gather-daemonset-qprrq"]
Oct 30 14:00:49.045254 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:00:49.045212    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rkb8w/perf-node-gather-daemonset-qprrq" event={"ID":"e60204fd-6f0d-4199-958d-315ecea66d94","Type":"ContainerStarted","Data":"0c1e35cff579bed6ce34ad84888a98d46ed9b995792aed3941f5a5efd34a64af"}
Oct 30 14:00:49.045733 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:00:49.045274    4364 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rkb8w/perf-node-gather-daemonset-qprrq" event={"ID":"e60204fd-6f0d-4199-958d-315ecea66d94","Type":"ContainerStarted","Data":"b98395a98e536185cdee02621850f674c16aec6c4f48d27b3295b4eca10362f6"}
Oct 30 14:00:49.045733 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:00:49.045488    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-rkb8w/perf-node-gather-daemonset-qprrq"
Oct 30 14:00:49.087177 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:00:49.087124    4364 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-rkb8w/perf-node-gather-daemonset-qprrq" podStartSLOduration=2.087110482 podStartE2EDuration="2.087110482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 14:00:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:00:49.085882339 +0000 UTC m=+4148.252246987" watchObservedRunningTime="2026-10-30 14:00:49.087110482 +0000 UTC m=+4148.253475130"
Oct 30 14:00:50.437826 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:00:50.437794    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-275zk_e10e4c9c-5a7b-4a00-8515-02467437a9d3/node-exporter/0.log"
Oct 30 14:00:50.567116 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:00:50.567089    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-275zk_e10e4c9c-5a7b-4a00-8515-02467437a9d3/kube-rbac-proxy/0.log"
Oct 30 14:00:50.692883 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:00:50.692813    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-275zk_e10e4c9c-5a7b-4a00-8515-02467437a9d3/init-textfile/0.log"
Oct 30 14:00:52.939882 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:00:52.939844    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-cmngs_6bb9bf93-b7c0-4cf8-b1f3-f802c6d91d36/prometheus-operator/0.log"
Oct 30 14:00:53.065008 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:00:53.064981    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-cmngs_6bb9bf93-b7c0-4cf8-b1f3-f802c6d91d36/kube-rbac-proxy/0.log"
Oct 30 14:00:53.195112 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:00:53.195043    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-22gcb_cd7629fa-bfd4-44ab-b51b-39e5eea7032d/prometheus-operator-admission-webhook/0.log"
Oct 30 14:00:53.333065 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:00:53.333030    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5fd6fc79b8-4fh6v_8d63267a-6f3e-4021-a91e-f342dab9e3aa/telemeter-client/0.log"
Oct 30 14:00:53.460042 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:00:53.459963    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5fd6fc79b8-4fh6v_8d63267a-6f3e-4021-a91e-f342dab9e3aa/reload/0.log"
Oct 30 14:00:53.590348 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:00:53.590316    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5fd6fc79b8-4fh6v_8d63267a-6f3e-4021-a91e-f342dab9e3aa/kube-rbac-proxy/0.log"
Oct 30 14:00:55.061124 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:00:55.061092    4364 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-rkb8w/perf-node-gather-daemonset-qprrq"
Oct 30 14:00:58.537309 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:00:58.537273    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-744d9f8b78-x4qbn_4f7d7ad7-a4ec-42bb-b124-8b9ac9e7b1a7/console/0.log"
Oct 30 14:00:58.683492 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:00:58.683458    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-p566k_03052418-91d3-44c9-95f2-f0b014dcbcbe/download-server/0.log"
Oct 30 14:01:01.003788 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:01:01.003739    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6bc88ffd5-5pkjs_5f540848-0b4f-4398-8a1f-6d1b3d7a379d/istio-proxy/0.log"
Oct 30 14:01:02.732788 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:01:02.732751    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ffvhz_62adaa28-5f97-4e1b-865f-d931ffd19d04/dns/0.log"
Oct 30 14:01:02.858360 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:01:02.858330    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ffvhz_62adaa28-5f97-4e1b-865f-d931ffd19d04/kube-rbac-proxy/0.log"
Oct 30 14:01:03.259350 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:01:03.259319    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7mvnz_c83bcf7b-a1d6-457b-9d7c-bb03c2cbd449/dns-node-resolver/0.log"
Oct 30 14:01:04.294789 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:01:04.294758    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65dc4d65b9-59szd_ac493c83-00bc-43fa-b0d1-ec05f21fcd92/kube-auth-proxy/0.log"
Oct 30 14:01:04.417049 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:01:04.417014    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-64d4cc998c-2vqwj_fd6ee897-425a-4e17-978d-8afde52b1331/istio-proxy/0.log"
Oct 30 14:01:04.696015 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:01:04.695986    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-54cf5557c-9c668_6fe75d93-6ca5-4051-91f9-a02766ce12cc/registry/0.log"
Oct 30 14:01:04.800512 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:01:04.800473    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-64d4cc998c-tbw9c_e40d7a56-5869-4a42-b8a7-33c02604c302/istio-proxy/0.log"
Oct 30 14:01:04.930799 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:01:04.930771    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-64d4cc998c-vt6z4_adbce7c5-af05-45e2-982c-183972ed458d/istio-proxy/0.log"
Oct 30 14:01:04.951194 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:01:04.950590    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7pxn6_31d09b91-c973-4cf9-a7d8-d602ff9994f6/node-ca/0.log"
Oct 30 14:01:04.992715 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:01:04.992687    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-64d4cc998c-wfkdb_cac946c2-24d3-42b9-9d62-5f45833c81fc/istio-proxy/0.log"
Oct 30 14:01:05.112943 ip-10-0-137-6 kubenswrapper[4364]: I1030 14:01:05.112912    4364 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-mqpfw_310804ff-17ef-4a28-959c-47995bde35ca/payload-processing/0.log"